Updated Powershell Scripts

For those who are interested, I just uploaded a bunch of changes to the PowerShell Scripts folder on my SkyDrive. Feel free to download them and use them as you need.

• find-to-set-alias – Brad Wilson enhanced this function significantly and broke it out into it’s own script. I had a small issue with his version where the folder search may only return a single value, so you can’t treat it like collection. My version wraps that command in @(…) so that you can always treat it like a collection.
• find-in-path – searches all the folders in your path for a given file name (wildcards supported. Very useful for “where is this app actually installed” kind of debugging.
• get-git-branch – returns the current git branch of a given folder. Got the idea for this originally from Ivan Porto Carrero.
• prompt – my powershell prompt. Pretty basic, but it now shows current git branch.
• elevate-process – create a new PowerShell window or run an app as an administrator. I alias this to su on my machine. I recently reworked the “run an app” part of this script, so it will search the current folder and then the path to run the app you specify.
• _profile – this is my main profile script, which I share across multiple machines via Mesh. I reworked all my alias setting to use the new find-to-set-alias and moved setting the color of the command window to the top of the script.

Update – I just updated elevate-process again, adding a special clause to handle .bat and .cmd files. cmd.exe seems to ignore the working directory setting, so if your batch file relies on being run from the folder it’s in, it’ll break with elevate-process. That’s annoying. So if you elevate a batch file, the script runs cmd.exe directly and executes the specified batch file after first changing to the current directory. Ugly, but it seems to work.

__clrtype__ Metaclasses Demo: Silverlight Databinding

I’ve gotten to the point where I can actually demo something interesting with __clrtype__ metaclasses: Silverlight Databinding. This is a trivial sample, data binding a list of Products (aka the sample class I’ve been using all week) to a list box. But according to Jimmy, this is something he gets asked about on a regular basis and there’s a AgDLR bug open for this. The __clrtype__ feature is specific to IronPython but I bet the IronRuby guys could implement something similar if they wanted to.

When you install IronPython 2.6 (or 2.0.1 for that matter), it comes with the AgDLR bits in the Silverlight subfolder. This includes Silverlight compatible versions of the DLR and IronPython as well as the Silverlight DLR host and the development web server Chiron in the Silverlight\bin directory. There is also a script in the Silverlight\script directory that will generate a dynamic Silverlight application from a template. I ran “sl.bat python sldemo” in order to build the skeleton project.

In the generated app.xaml file, I removed the default text box and replaced it with this XAML code that I stole nearly-verbatim from my blog post on data binding in WPF with IronPython. The only thing I changed was the binding path for the text block (title became name).

Then in the App class, I set the ItemsSource of the ListBox to a hand-built a list of Products.

And that’s pretty much it. I used Chiron’s /z command to create a Silverlight XAP file, uploaded it to Silverlight Streaming and embedded it right here in this post. Code is up on my skydrive as well. Uusing Silverlight Streaming for this app was very easy - basically upload the XAP file to their server and embed some iframe code in this post via the source view and that was it. I’m not sure I would use it for a production app, but it rocked for hosting this demo.

The XAP is a big download for such a trivial app - about 1.3MB. The vast majority of that is the DLR and IronPython assemblies. The XAP would only be 2.9kB if it was just the Python, XAML and manifest files. This kinda stinks, but there’s a new transparent platform extensions feature in Silverlight 3 so we can at least break the DLR and IronPython DLLs out into their own separate XAPs. That way they only get downloaded once and cached in the browser instead of being included in every single IronPython Silverlight application anyone creates.

So that’s one scenario down, one to go. In order to be able to build WCF services in IronPython, I have to add a lot more infrastructure – notably emitting CLR methods that can invoke dynamic methods as well as emitting custom attributes. Invoking dynamic methods means understanding DLR binders, so look for more posts on __clrtype__ next week.

__clrtype__ Metaclasses: Adding CLR Properties

When I was first experimenting with __clrtype__, I got to the point of making CLR fields work and then immediately tried to do some data binding with Silverlight. Didn’t work. Turns out Silverlight can only data bind against properties – fields aren’t supported. So now let’s add basic property support to ClrTypeMetaclass. Python has a rich mechanism for defining properties, but hooking that up requires DLR binders so for now I’m going to generate properties that are simple wrappers around the associated fields.

There’s enough code involved in defining a property to break it out into it’s own method:

You provide define_prop the TypeBuilder for the Type being constructed, the name and type of the property as well as the FieldBuilder that gets returned from the call to DefineField. In the previous installment, I wasn’t bothering to save the FieldBuilder to a variable since I never used it again. Now, I’m stashing it away for the call to define_prop as I’ll show below.

For each field, we define a get method, a set method and a property. The get function first executes ldarg_0 to load the current object reference onto the execution stack, then it executes ldfld to load the specified field from the object onto the stack, then it returns. The set function executes ldarg_0 to load the current object reference and ldarg_1 to load the value passed as the first argument onto the execution stack, then it executes stfld to store the value in the specified field of the object. Once I have the two methods, I call DefineProperty to create the PropertyBuilder and then associate the get and set methods with that property.

As I said before, Reflection.Emit is straightforward though tedious. Honestly, I didn’t go thru the Emit docs to figure out what the methods should look like. Instead, I wrote a basic wrapper property in C# and looked at the generated IL in Reflector.

The only other change here is adding the call to define_prop on our first iteration thru list of _clrfields. Since the rest of __clrtype__ is the same, here’s just that code snippet:

As I said above, I simply save off the result of calling DefineField so I can pass it to define_prop. I also save off the field type in a variable since I use it more than once. Avoids the second dictionary lookup and is clearer to understand what the function does.

Accessing the CLR properties via reflection is pretty straightforward – not very different than reflecting over CLR fields. The only significant difference between them is that CLR properties can be indexable and fields can’t, so you have to pass an index parameter to GetValue and SetValue. These aren’t indexed properties, so I pass in None for the index parameter.

One quick aside about the CLR type I’m generating here. I’m fairly certain this reflected object wouldn’t pass muster with the C# compiler. I’m defining a field and a property with the same name. It clearly works at the IL level, but I’m not sure what the C# compiler would do if you tried to refer to a CLR type like this. I should probably be prepending an underscore or something on the field name, but then I wonder if the field should also be private. There’s a whole API design discussion down that road, but I’m not quite ready to have that yet so I’m just leaving the fields public and having fields and properties with the same name. Luckily, I’m never generating a CLR type on disk so you can’t build a C# project that refers to it anyway.

__clrtype__ Metaclasses: Adding CLR Fields

Now that we have the basic __clrtype__ metaclass infrastructure in place, let’s enhance it to add support for CLR fields. To do this, we’re going to need to add two things to our custom CLR type. First, we need to define the fields themselves. Second, we need to make sure that Python code will read and writes to the statically typed fields for the specified names rather than the storing them in the object dictionary as usual. Here’s the updated version of ClrTypeMetaclass (or you can get it from my skydrive)

All the base type, type name, type builder and constructor code in the first half of the __clrtype__ method is the same as last time, so we’ll focus on the second half. After emitting the constructor(s), next we iterate thru a dictionary named _clrfields (if it exists in the class) that maps field names to types. For each of these dictionary entries, we emit a public field on the CLR type with the specified name and type.

The first time I tried this, I simply added the custom field generation code I just described and left it at that. Didn’t work. Python doesn’t look to store information in fields defined by the static type metadata unless explicitly instructed to. That’s why I need to iterate over the declared list of fields a second time after the type has been created. The first time creates the CLR fields, the second time inserts a ReflectedField instance into the class dictionary. ReflectedField is a Python descriptor that reads and writes the field value by calling GetValue and SetValue on the contained FieldInfo object. Python uses the same name resolution for fields as it does for method (In Python, methods are fields that store callable objects) so when IronPython discovers the ReflectedField descriptor in the class instance, it uses that to get or store the value rather than sticking it in the local dictionary.

Now here’s the new version of the Product class, this time with CLR fields as well as a custom type name:

As you can see, the only thing that’s changed is the addition of the _clrfields dictionary. But now, we can use reflection to get and set the Product fields, like so:

This is great progress, but not enough to get us to our first “real” scenario: data binding in Silverlight. Silverlight only supports data binding against public properties, so I’ll need to wrap all these CLR fields in CLR properties in my next post.

__clrtype__ Metaclasses: Customizing the Type Name

Now that we know a little about how IronPython uses CLR types under the hood, let’s start customizing those types. In a nutshell, __clrtype__ metaclasses are metaclasses that implement a function named __clrtype__ that takes the Python class definition as a parameter and returns a System.Type. IronPython will then use the returned Type  as the underlying CLR type whenever you create an instance of the Python class.

Technically, you could emit whatever custom CLR Type you want to in the __clrtype__, but typically you’ll want to emit a class that both implements whatever static CLR metadata you need as well as the dynamic binding infrastructure that IronPython expects. The easiest way to do this is to ask IronPython emit a type that handles all the dynamic typing and then inherit from that type to add the custom CLR metadata you want.

Let’s start simple and hello-worldly by just customizing the name of the generated CLR type that’s associated with the Python class. There’s a fair amount of boilerplate code that is needed even for this simple scenario, and I can build on that as we add features that actually do stuff. If you want to follow along at home, you’ll need IronPython 2.6 Alpha 1 (or later) and you can get this code from my SkyDrive.

Like all Python metaclasses, ClrTypeMetaclass inherits from the built-in Python type object. If I wanted to customize the Python class as well, I could implement __new__ on ClrTypeMetaclass , but I only care about customizing the CLR type so it only implements __clrtype__. If you want to know more about what you can do with Python metaclasses, check out Michael Foord’s Metaclasses in Five Minutes.

First off, I want to get IronPython to generate the base class that will implement all the typical Pythonic stuff like name resolution and dynamic method dispatch. To do that, I call __clrtype__ on the supertype of ClrTypeMetaclass – aka the built-in type object. That function returns the System.Type that IronPython would have used as the underlying CLR type for the Python class if we weren’t using __clrtype__ metaclasses.

Once I have the base class, next I figure out what the name of the generated CLR type will be. This is pretty simple, I just use the name of the Python class. To make this logic a little more interesting, I added support for a custom namespace. If the Python class has a _clrnamespace field, I append that as the custom namespace for the name. I should probably be using a double underscore – i.e. __clrnamespace – but I didn’t want to wrestle with name mangling in this prototype code.

Now that I have a name and a base class, I can generate the class I’m going to use. I’m using the DefineType method in Microsoft.Scripting.Generation.Snippets DLR class for three reasons. First, there’s a CLR bug that doesn’t let you create a dynamic assembly from a dynamic method. Second, reusing the snippets assembly avoids the overhead of generating a new assembly. Finally, the types in Snippets.Shared get saved to disk if you run with the -X:SaveAssemblies flag, so you can inspect custom CLR type that gets generated. The DefineType function takes four parameters, the type name, the base class, a preserve name flag and a generate debug symbols flag. If you pass false for preserve name, you get a name like foobar$1 instead of just foobar. As for debug symbols, since I don’t have any source code that I’m generating IL from, emitting debug symbols doesn’t make a lot of sense. DefineType returns a TypeGen, but I only need the TypeBuilder.

The last thing I need to do is implement the custom CLR type constructor(s). IronPython CLR types will always have at least one parameter – the PythonType (PythonType == IronPython’s implementation of Python’s built-in type object) that’s used for dynamic name resolution. I don’t want to add any custom functionality in my custom CLR type constructors, so I simply iterate thru the list of constructors on the base class and generate a constructor on the custom CLR type with a matching parameter list and that calls the base class constructor. 

Generating the IL to emit the constructor and the base class is straightforward, if tedious. I define the constructor with the same attributes, calling convention and parameters as the base class constructor. Then I emit IL to load the local instance (i.e. ldarg 0) and all the parameters onto the stack, call the base constructor and finally return. Once all the constructors are defined, I can create the type and return.

Using the ClrTypeMetaclass is very easy - simply specify the __metaclass__ field in a class. If you want to customize the namespace, specify the _clrnamespace field as well. Here’s an example:

You can verify this code has custom CLR metadata by calling GetType on a Product instance and inspecting the result via standard reflection techniques.

Great, so now I have a custom CLR type for my Python class. Unfortunately, at this point it’s pretty useless. Next, I’m going to add instance fields to the CLR type.

__clrtype__ Metaclasses: IronPython Classes Under the Hood

Before we start using __clrtype__ metaclasses, we need to understand a bit about how IronPython maps between CLR types and Python classes. IronPython doesn’t support Reflection based APIs or custom attributes today because IronPython doesn’t emit a custom CLR types for every Python class. Instead, it typically shares a single CLR type across many Python classes. For example, all three of these Python classes share a single underlying CLR type.

Even though cheese_shop inherits from shop and argument_clinic inherits from object, all three classes share the same underlying CLR type. On my machine, running IronPython 2.6 Alpha 1, that type is named “IronPython.NewTypes.System.Object_1$1”.

IronPython can share the CLR type across multiple Python classes because that CLR type has no code specific to a given Python class. CLR types are immutable – once you build a CLR type, you can’t do things like add new methods, remove existing method or change the inheritance hierarchy. But all those things are legal to do in Python. Here, I’m creating an instance of the cheese_shop class, but then changing that instance to be an argument_clinic instance instead.

When you call a method on a Python object, the name is resolved by walking a series of dictionaries. First, the dictionary of the object itself is searched for the method name. Assuming the name isn’t in the object dictionary, Python then looks in the __class__ dictionary. If it’s not there, Python recursively looks through the base classes stored in the __bases__ tuple until it finds the method or the name fails to resolve. If we re-assign __class__ at run time, we change the dictionary Python uses to resolve method names.

There are cases where IronPython generates a new underlying CLR type. For example, if you build a Python class that inherits from a CLR type, then IronPython will have to generate a new underlying CLR type that inherits from the CLR type in order to remain compatible. IronPython automatically overrides all the virtual methods of the base type, implementing the same dynamic method dispatch that I described above. This lets you pass the IronPython class wherever the base CLR type is expected.

The ability to swap Python classes at runtime depends on having the same underlying CLR type. If the underlying CLR type doesn’t match, then assigning a new value to the __class__ field of an object will fail. This applies both to IronPython classes that inherit from CLR types as well as __clrtype__ metaclass types. In the code I’ll be blogging, I always generate a unique CLR type for every Python class, which means that I can’t dynamically retype the object. Given that the point of __clrtype__ metaclasses is to generate static type information, this hardly seems like a limitation. However, it’s something to be aware of as we explore the __clrtypes__ feature.

The Lounge Survey

I just joined The Lounge advertising network so I wanted to pass along an opportunity to win a bunch of great technical books.

The Lounge is asking the readers of the blogs in their network to fill out a survey in order for them to improve how they target their advertising. It’s pretty much what you would expect from an advertising network focused on the .NET development platform: what language(s) do you use, what framework(s), what testing tool(s), etc, etc, etc. Takes like three minutes to fill out at most.

We all know that filling out surveys isn’t what most people consider “exciting” or “fun”. In order to incent you, dear Reader, to take a few minutes of your valuable time to fill out the survey, The Lounge is giving away all forty one of Manning’s “In Action” books, including IronPython In Action. Even if you don’t win, you still get 40% discount off any purchase from Manning.

So it’s up to you, a scant few minutes of your time in exchange for a chance to win enough technical books to keep you busy for months.

Introducing __clrtype__ Metaclasses

Everyone knows Anders announced at PDC08 that C# 4.0 will include new features (aka the dynamic keyword + the DLR) that makes it much easier for C# to call into dynamically typed code. What you probably don’t know is that IronPython 2.6 includes a new feature that makes it easier for IronPython code to be called by statically typed code.

While the vast majority of .NET is available to IronPython, there are certain APIs that just don’t work with dynamic code. In particular, any code that uses Reflection over an object’s CLR type metadata won’t work with IronPython. For example, while WPF supports ICustomTypeDescriptor, Silverlight only supports data binding against reflectable properties. Furthermore, any code that uses custom attributes inherently uses Reflection. For example, Darrel Hawley recently blogged a WCF host he wrote in IronPython, but he wrote the WCF service in C#. You can’t write WCF services in IronPython because WCF expects service classes to be adorned with ServiceContract and OperationContract attributes (among many others). IronPython users want access to use these APIs. Support for custom attributes is one of the most common requests we get - it’s currently the 5th highest vote getter among open issues.

In IronPython 2.6, we’re adding the ability to customize the CLR type of Python classes. This means you can add custom attributes, emit properties, whatever you want. For those of you who’ve been dreaming of implementing WCF services or databinding in Silverlight purely in IronPython, then this is the feature for you.

In a nutshell, IronPython 2.6 extends Python’s metaclass feature that lets you to customize the creation of classes. In the metaclass, you can implement an IronPython-specific method __clrtype__ which returns a custom System.Type of your own creation that IronPython will then use as the underlying CLR type of the Python class. Implementing __clrtype__ gives you the chance to implement whatever reflectable metadata you need: constructors, fields, properties, methods, events, custom attributes, nested classes, whatever.

Over a series of posts, I’ll be demonstrating this new feature and implement some common scenario requests – including Silverlight databinding and WCF services – purely in Python. Quick warning: __clrtype__ uses low level features like Python metaclasses, Reflection.Emit and DLR Binders so these posts will be deeper technically than usual. Don’t worry – this isn’t the API interface we expect everyone to use. Eventually, we want to have an easy to use API that will sit on top of the low-level __clrtype__ hook.

Issue Tracking for my GitHub Projects

FYI, anyone using ipydbg, devhawk_ipy or Pygments for WL Writer can now submit issues, bugs and feature requests other the associated GitHub project page. Head to the associated GitHub project page and click the “Issues” tab at the top. There’s a short video of introducing the GitHub Issues feature you can check out.

BTW, thanks to defunkt @ GitHub for the ipydbg shoutout.

Joining the Lounge Advertising Network

For those of you who read this blog primarily via my RSS feed, I made a change to my homepage over the weekend. Goodbye adSense, hello The Lounge. The Lounge is an ad network run by James Avery’s company InfoZerk. I’ve known James for a while – he included my now-obsolete SccSwitch tool in his book Visual Studio Hacks.

From a financial perspective, I’m not really sure how much of a difference this will make. I guess I’ll see when I get my first check. Given how little I was making with adSense, I’ve got nowhere to go but up. Regardless, I feel much better working with a smaller ad network run by someone I respect and that is focused on the .NET platform.

Thanks for having me in the Lounge ad network, James.

Writing an IronPython Debugger: Breakpoint Management

Setting a breakpoint was the second feature I implemented in ipydbg. While setting a breakpoint on the first line of the Python file being run is convenient, it was obviously necessary to provide the user a mechanism to create their own breakpoints, as well as enable and disable existing breakpoints.

First thing I had to do was to refactor the create_breakpoint method. Originally, I was searching thru the symbol documents looking for the one that matched the filename in OnUpdateModuleSymbols. However, since I wanted to specify by new breakpoints via the same filename/line number combination, it made more sense to move symbol document logic into create_breakpoint:

The new version isn’t much different than the old. It loops thru the symbol documents looking for one that matches the filename argument. Then it creates the breakpoint the same way it did before. Eventually, I’m going to need a better algorithm than “only compare filenames”, but it works for now.

Once I made this change, it was trivial to implement a breakpoint add command. What was harder was deciding on the right user experience for this. I decided that breakpoint management was going to be the first multi-key command in ipydbg. so all the debug commands are prefixed with a “b”. I use the same command routing decorator I used for input commands. As you can see, my breakpoint command looks a lot like my top level input method – read a key from the console then dispatch it via a commands dictionary that gets populated by @inputcmd decorators.

Currently, there are four breakpoint commands: “a” for add, “l” for list, “e” for enable and “d” for disable. List is by far the simplest.

As you can see, I’m keeping a list of breakpoints in my IPyDebugProcess class. Originally, I used AppDomain.Breakpoints list, but that only returns enabled breakpoints so I was forced to store my own list. Note also that I’m using the enumerate function, which returns a tuple of the collection count and item. I do this so I can refer to breakpoints by number when enabling or disabling them:

Since the code was identical, except for the value passed to bp.Activate, I factored the code into a separate _set_bp_status method. After the user presses ‘b’ and then either ‘e’ or ‘d’, they then type the number of the breakpoint provided by the breakpoint list command. _set_bp_status then simply iterates thru the list until it finds the matching breakpoint and calls Activate. Note that since it’s possible to have 10 or more breakpoints, I’m using ReadLine instead of ReadKey, meaning you have to hit return after you type in the breakpoint number.

Finally, I need a way to create new breakpoints. With the refactoring of create_breakpoint, this is pretty straightforward

Most of _bp_add is processing the input arguments, looping through the modules and then storing the breakpoint that gets returned. When I set the initial breakpoint inside OnUpdateModuleSymbols, I have the module with updated symbols as an event argument. However, in the more general case we’ve got no way of knowing which module of the current app domain contains the filename in question. So we loop thru all the modules, calling create_breakpoint on each until one returns a non-null value. Of course, “all the modules” will include the IronPython implementation, but assuming you’re running against released bits the call to create_breakpoint will return right away if debug symbols aren’t available.

As usual, the latest version is up on GitHub. This will be the latest update to ipydbg for a little while. I worked on it quite a bit while I was at PyCon and have been busy with other things since I got home. Don’t worry, I’ll come back to it soon enough. As I mentioned Monday, I want to get function evaluation working so I can have a REPL console running in the target process instead of the one I’ve got currently running in the debugger process.

Pygments for WL Writer v1.0.1

I just replaced the original v1.0.0 Pygments for WL Writer installer with a new and improved v1.0.1. The original URL still works – I archived the old version off with a new name. Updated source is available on on GitHub.

The only change is that I now override OnSelectedContentChanged in the sidebar control. That way, if I have multiple blocks of pygmented code in a given post, the sidebar UI updates with the correct language and color scheme of the currently selected code block.

Writing an IronPython Debugger: REPL Console

While I was banging my head against a wall experimenting with understanding how CorValue extraction worked, I found myself wanting to dink around with the debugger objects in a REPL console. One of IronPython’s core strengths is support for “exploratory programming” via the REPL. It turned out bringing a REPL to ipydbg was quite simple.

Python includes two built-in features that making DIY REPL quite easy: compile and exec (though technically, exec is a statement, not a function). As you might assume from their names, compile converts a string into what Python calls a code object while exec executes a code object in a given scope. Technically, exec can accept a string so I could get by without using compile. However, if you’re compiling a single interactive statement compile can automatically insert a print statement if you’ve passed in a an expression. In other words, if you type in “2+2” on the console it will print “4”, which is the behavior I wanted.

Here’s what my REPL console code look like. I love that it’s only 20 lines of code.

It’s pretty straightforward. I set up a dictionary to act as the local variable scope for the code that gets executed. I’m just reusing the current global scope, but I want the local scope to start with only the reference to the current IPyDebugProcess instance which is passed into _input_repl_cmd as “self”. All the other local variables like cmd and line won’t be available to the REPL code. Then I drop into a loop where I read lines from the console and execute them.

In order to support multi-line statements, I build up the cmd variable over multiple line inputs and I don’t execute it until the user inputs an empty line. In the standard Python console, it can recognize single line statements and execute them immediately. Dino showed me how to use the IronPython parser to do the same thing, but I haven’t implemented that in ipydbg yet. To exit the REPL loop, you type Ctl-Z, which returns None (aka null) from ReadLine instead of the empty string.

Since I never execute the code more than once, I have my exec and compile statements together on a single line. Compile takes the string to be compiled, the name of the file it came from (I’m using <input> for this) and the kind of code. Passing in “single” for the kind of code adds the auto-expression-print functionality I mentioned above. Then I exec the code object that’s returned in specified scope I’m managing for this instance of the REPL loop. If you exit out of the REPL and re-enter it, you get a fresh new copy of the local scope so any functions or variables you define in the last REPL are gone.

Runtime execution of code into a given scope is a hallmark of dynamic languages, but I’m still fairly green when it comes to Python so it took me a while to figure this out. Python code executes in a given scope, a combination of global and local variables. When you’re in the ipy.exe REPL, you’re at top level scope anyway, so global and local scope are the same – if you add something to global scope, it shows up in local scope and vis-versa. Inside a function, you’ll have the same global scope, but the local scope will be different and changes to one won’t be reflected in the other. The ipydbg REPL isn’t a function per-se, but it does provide an explicit local scope that gets disposed when you exit the REPL.

While having a debugger REPL is really convenient for prototyping new ipydbg commands, it’ll really shine once I get function evaluation working. Then I’ll be able to open a REPL console where the commands are executed in the target process instead of the debugger process as they are now. That will be very cool. Until then, the latest code is – as always – up on GitHub.

Writing an IronPython Debugger: Getting Arguments

It’s a small update, but I added support for displaying method arguments along side the local variables. As I mentioned in that post, breaking out the CorValue extraction and display code into a shared function was a good idea – adding support for getting arguments was trivial since I could reuse that code.

Because there’s no hierarchy of scopes to deal with and the names are in the metadata instead of debug symbols, getting arguments is much easier than getting local variables.

You’ll notice that I’m yielding the arguments as a tuple of the name and value, the same as get_locals yields. I did refactor get_locals a bit – there’s no longer an argument to skip hidden variables anymore (though get_locals still skips dynamic call sites caches as it did before). Now, it’s up to the the caller of get_arguments and get_locals to filter hidden variables as they see fit.

Because get_locals and get_arguments yield the same types, I was able to factor the code to print a value and loop through the collection of values into separate local functions.

I really like the local functions feature of Python. In C#, you can define an anonymous delegate using the lambda syntax. But for a scenario like this, I like local functions better. However, I do like C#’s support for statement lambdas – Python only supports expression lambdas. So while I like local functions better in this scenario (because I’m using the method more than once) in something like an event handler, I like the statement lambda syntax better.

As usual, the latest version of ipydbg is up on GitHub.

Pygments for Windows Live Writer

For the past few years, I’ve used the CodeHTMLer plugin for Windows Live Writer for the code snippets in my blog. However, recently I discovered the Pygments Python syntax highlighter package which supports scores more languages than CodeHTMLer does. It also support multiple color schemes and was easily extensible so I could build an HTML formatter that didn’t use <pre> tags (which I’ve found DasBlog has issues with in the RSS feed, though honestly I’m running three minor releases behind the latest DasBlog release). IronPython supports Pygments just fine – at least, the one IPy bug that Pygments exposes has a simple workaround – so I set about building a Windows Live Writer plugin that uses it.

If you’re simply interested in the plugin itself, you can get it from my skydrive. The source is up on GitHub. For now, if you find any bugs, please leave a comment on this post. If there’s enough interest I’ll setup a site somewhere (CodePlex perhaps) where I can track bugs and feature requests.

Pygments for WL Writer is a smart content source. In WL Writer’s terminology, that means when you click inserted text in the editor window, it is treated as an atomic entity which you can then edit by using the Edit Code button in the Pygments for WL Writer sidebar editor. I I often found that I would edit my code multiple times – usually to shorten lines so they’d fit on my blog without wrapping. CodeHTMLer for WL Writer is a standard content source, so it just spews the formatted code as HTML onto the page.

From an IronPython perspective, there’s some interesting stuff there. I decided to compile the pygments library into a DLL for easier distribution. If you look in the source, there’s a folder for the Pygments source as well as the parts of the standard Python library that Pygments depends on and my custom HTML formatter. Those all get compiled via a custom script which can be called by the build.bat file in the project root.

Some features I’m thinking about adding:

• An extensibility model so that you can add new languages by dropping new Pygments lexers into the same folder the plugin is installed to. Pygments supports lots of languages, but not all of them – notably it’s missing Powershell and F#.
• Support for new HTML formatters and color schemes using the same extensibility mechanism described above.
• Support for selecting an HTML formatter.
• Improving the code editor window. Currently, I’m using a standard WinForms multi-line TextBox, but that leaves a lot to be desired. With the Python work I do, I often need to be able to select a bunch of text and change it’s indenting via tab and shift-tab. If anyone has a suggestion for a good WinForms text editing control, let me know.
• Being able to specify the font and size of the Pygmented code.
• Storing user preferences – remembering the most recent syntax and color scheme the user used.

Feedback, as always is appreciated. I’ll probably write a few posts about the project when I get a chance, so let me know if there’s anything you’re dying to hear about.

Writing an IronPython Debugger: Command Routing

At this point, ipydbg support seven commands: Continue, Quit, Show Stack Trace, Show Locals, Step Over, Step In, and Step Out. All these commands are invoked by a single keystroke. I’m using Console.ReadKey in an attempt to cut down on the number of keystrokes needed for interacting with the debugger. If I only type ‘s’ instead of ‘s <enter>’ to step, I figure I’ll be twice as productive! :)

If I was writing ipydbg in C#, I could use switch statement to dispatch commands in the _input method based on user keystrokes. However, Python doesn’t have a switch statement so I’ve been using a cascading set of if/elif/else statements instead. When you get up to seven if/elif clauses plus an else clause, the code smell is pretty overwhelming.

Python might not have a switch statement, but it does have first-order functions so you can get the effects of a switch by using a dictionary.

I like this approach much better. Individual if/elif blocks are now broken out into separate functions, which smells better than embedding them in one big function. Also, I like that my pseduo-switch statement is completely separate from the how the _switch dictionary is initialized. However, this approach also separates the pseudo-case statement functions from the _switch dictionary as well. That’s not a good thing. You can easily imagine screwing up by adding a new function but forgetting to manually update the _switch dictionary.

What I need is a way to declaratively associate the switch function with the dictionary lookup key that’s associated with it. Luckily, Python Decorators provides a very clean way to do this.

I’ve blogged about decorators before when I wanted to automatically invoke operations on the right thread in my WPF photo viewing app. The @inputcmd decorator is a bit more complicated than the @BGThread and @UIThread decorators since @inputcmd decorator accepts arguments. Each of the @input command decorators in the code above is the equivalent to this code:

As you can see, the inputcmd function returns the decorator that wraps do_a, rather than being the decorator itself. This function that returns a function that returns a function is kinda confusing at first. But this approach allows you to configure the decorator for a specific purpose via the arguments – in this case, specifying which dictionary and which console key this function is associated with.

Also unlike @BGThread and @UIThread, I don’t actually want to modify the behavior of the methods decorated with @inputcmd. I only want to store a reference to them in the passed in dictionary. So implementing this decorator is very easy:

The decorator simply inserts the function into the passed-in dictionary using the passed in key. It then returns the function as is, so it’s not really rebinding the symbol to a new method (technically, it’s rebinding the symbol to the same function it’s currently bound to). If I wanted also wrap the passed in function to provide additional functionality, I could do that with a second locally defined function inside the deco function.

The latest version of ipydbg as been refactored to use @inputcmd instead of set of a cascading if/elif statement blocks. Now that that’s done, I can start working on multi-key commands.