IronPython 2.0.3 and 2.6 RC2

It has been a very busy week for the IronPython team. Just under a week ago, we shipped a CTP for .NET Framework 4.0. Since then, we’ve shipped two – yes, two! – more versions of IronPython. Three releases in one week! If we could keep up that pace, we’d be shipping like 27 more releases of IronPython by the end of the year!

FYI, we’re not going to keep up the pace of shipping three releases a week for the next two months. We may be a little crazy on the IronPython team, but we’re not THAT crazy!

Actually, all three of these releases represent fairly small changes in the IronPython source tree. The .NET 4.0 beta was a CTP, so it’s basically whatever we had in our main trunk when they forked .NET Framework for the beta.

IronPython 2.0.3 is a minor point release in the 2.0 branch (duh). In addition to backporting some fixes from 2.6, we had to fix an CLR breaking change in partial trust on Windows 7. If you’re using IronPython 2.0.x in partial trust on Windows 7 you *MUST* upgrade to 2.0.3 (or 2.6 when it’s released). Sorry about that – but it was out of our hands.

IronPython 2.6 RC2 is – as you would expect - a minor update over the first release candidate. There was a memory leak discovered in the hosting APIs which forced us to do a second release candidate. Since we had to fix that, we took in a few of other fixes including some standard library changes (we left out json by accident in RC1 and Michael Foord got logging updated to work better with IronPython so we took the latest version of it). As per the release notes, we expect this to be the final RC and will re-ship it as RTM in about a month. Please start using this latest release and let us know if you find anything.

Joining Windows

Big news in DevHawk land: I’ll be taking on a new role in the Windows division after I come back from TechEd Europe (plus a week vacation visiting my bro-in-law in Germany after TechEd).

You guys have heard of Windows, right? They just released a new version recently you might be aware of…Actually, I hear it was your idea!

Seriously, I am stoked to be joining the Windows team. I can’t say much about the new job beyond a) it’s in Windows Client (as opposed to Windows Server) and b) I’m working on a team that’s focused on the Windows developer experience. The Windows team is deep in what you might call “building on teh awesome that is Windows 7” but that they simply call “planning”, so sorry if specifics are kinda sparse. I’ll be back working for Mahesh Prakriya, who originally hired me into my current role on the IronPython team. Someday I might tell you the Mahesh PyCon Lego Story, but for now I’ll just say I was great working for Mahesh the first time and I think this time is going to be even better.

Working on Windows…focused on developer experience…for Mahesh - It’s like the perfect storm of work geekdom for me.

Of course, starting a new job means my time on the IronPython team is coming to an end. As excited as I am about this new opportunity in the Windows division, I’m a little sad to be leaving Developer Division and the IronPython team. I’ve joked with audiences that I care about Python because Microsoft pays me to care about Python, but that’s not completely true. Python is a fantastic language and IronPython’s combination of Python + .NET is hard to beat in my opinion. IronPython has made significant progress while I’ve been here the last eighteen months – two major releases (well, 2.6 is almost done), redisting the Python standard library, Visual Studio 2010 compat - but there’s still much for IronPython to accomplish. And of course, leaving behind such great teammates like Dino, Dave, Jim and Jimmy is ALWAYS hard.

The Windows team is somewhat tighter lipped than the totally transparent approach we use in IronPython. Not to worry, my evangelism skills were part of the reason I got the job so you’ll be hearing plenty from me soon enough. However, my posting here will be kinda sparse until I get my bearings over there. Until then, I’m sure that you will be absolutely fascinated by non-work-related-but-still-sometimes-geeky minutia I post on Twitter.

IronPython and IronRuby CTPs for .NET 4.0 Beta 2

In case you’ve been hiding under a rock (or maybe just aren’t tracking developments in the .NET community outside of IronPython), Microsoft released Visual Studio 2010 beta 2 this week. Of course for me personally, the most important feature in Visual Studio 2010 is C# 4.0 new dynamic type (also available in Visual Basic, but since VB already supported some level of late binding it’s not exactly “new” to VB).

For those of you who want to experiment with this cool new feature, may I present IronPython 2.6 CTP for .NET 4.0 Beta 2. If you can’t think of any cool things to try with this new feature, the VB team blog has some scenarios to get your started.

Also available: IronRuby CTP for .NET 4.0 Beta 2 if you’re more into gemstones than snakes.

These are preview releases, which means they’ve gone thru basic testing. If you find any bugs, PLEASE report them via the usual channel. I wrote in my Post 2.6 Roadmap post, “we are committed to shipping the RTM of our .NET 4.0 version the day that Visual Studio 2010 is publicly available” but that means shaking out the bugs between now and then. We need your help so we’re ready to go by Visual Studio 2010 launch - March 22, 2010 as per Soma’s blog.

BTW, Alcides Fonseca suggested we call this release “IronPython 2.6 N4” since it’s designed to run on .NET Framework 4.0. I like that. What do you think?

Hybrid App Debugging – The Debug Window

In my last installment, I added support for a separate debug window on a separate thread from the main window thread. That way, I can pause the execution of the main window while the debug window stays responsive to user input. Now, let’s add some functionality to the debug window. I’m going to start by showing the source code of the python file being executed.

The TraceBackFrame instance has a property f_code that represents the FunctionCode object being executed in this frame. We have to explicitly cast to FunctionCode type because currently we’re exposing all properties that hang off TraceBackFrame as object type. Since Python is a dynamic language, we’re going to use reflection against the instance itself anyway so it doesn’t really matter what the return type is. However, I’ve asked Dino to change the TraceBackFrame type to use explicit types in order to make it easier to use SetTrace from statically typed languages like C#. Look for that in RC2.

After we cast the code object so it can be used from C#, we check to see if the currently loaded file matches the file currently loaded into the UI. I’ve ranted recently about the limitations of WPF’s TextBox but I didn’t want to get hung up syntax highlighting for this sample so I just went ahead and used the RichTextBox. In the DebugWindow Loaded event handler, I create _source as a WPF Paragraph and then wrap it in a FlowDocument and use it as the RichTextBox’s Document. I set the FlowDocument to be extremely wide, so as to avoid word wrapping. Then when I need to load a new source file, I clear _source of it’s current contents and add a single line run for every line of code in the file. This convention becomes useful later when I go to highlight the current line of code.

Once I load the current file, I save the current frame, code, result and payload in instance fields and then switch on result to determine what to do next. Currently, I’m just highlighting the relevant line of code and setting a TextBlock control in the menu bar.

In Visual Studio, we typically highlight the current line of code in yellow. However, that doesn’t work as great in a language like Python that delineates code blocks with whitespace. In ipydbg, I indicated function return with three carets. But I didn’t want to be modifying the text in the RichTextBox here so instead I used different colors for the different traceback event types: light green for call, light pink for return and yellow for line. The frame object contains the current line number, which I use for call and line, while the code object has the first line of the current code object, which I use for return. HighlightLine highlights the line in question with the colors provided and also scrolls that line into view if it isn’t already visible.

So now when a traceback is handled, it shows the text for the file being executed and highlights the appropriate line, based on the type of traceback event that happened. Now all we need is to have some way be able to continue execution. In the code, you’ll see I’ve defined a series of RoutedUICommands for common debugger commands. I’ve got the StepIn command wired up in the DebugWindow XAML to a menu item and the “S” keystroke. All that remains is to define StepInExecuted.

This function does three basic things: changes the dbgStatus text, resets all the text in the RichTextBox back to the default coloring, and sets the _dbgContinue AutoResetEvent which signals the main window thread that’s been blocked in OnTracebackReceived to continue.

With this post, I’m about even with the code that’s up on GitHub. That code has a few other capabilities – notably it will stop tracing if you close the debug window and it supports StepOut command which disables traceback for the current scope by returning null in OnTracebackReceived. But I haven’t implemented things like:

• Set Next Statement
• Viewing and changing variables
• Debugger REPL
• Breakpoint Management

Any suggestions on which of those would you like to see next?

Hybrid App Debugging – Threading

I added traceback to my GetThings app in just two lines of code, but so far it doesn’t actually do anything that you would expect a debugger to do. But before we get to that, we need understand a little about how threading works for traceback debugging.

As I mentioned last time, the traceback debugger works by calling into the registered traceback handler at various times (entering/exiting a function, before executing a line of code and on exceptions). Execution of the Python code continues when the traceback function exits. That means that you have to block the execution thread while you let the user poke around with the debugger UI. For a console based app, that’s easy. For a GUI app, not so much.

At a minimum, you need to run your debugger UI on a separate thread from your main app window. If you want your main app window to be responsive while you debug, you’ll need to pump messages at a minimum (DoEvents in Windows Forms, similar approaches are available for WPF) or preferably run your python scripts on a background thread separate from either the main window UI thread or the debugger UI thread. To keep things simple, I’m going to simply block the main window thread while the debugger is active.

Since I’m going to have to setup a new thread for the debugger window, I decided to use a static constructor to centralize creating the thread, creating the window and registering the traceback handler all in one place.

As you can see, InitDebugWindow spins up a new thread and creates the debug window on that thread. Since it’s not the main WPF application thread, you have to explicitly call Dispatcher.Run to get the event queue pumping. You also have to explicitly set the apartment state to be single threaded for any threads creating WPF objects. Finally, I wait for the window to signal that it’s ready (it set’s the _debugWindowReady AutoResetEvent in the Window Loaded event) and then call SetTrace, passing in the debug window’s OnTracebackReceived event, on the thread that called InitDebugWindow.

It’s critical that you call SetTrace – and thus InitDebugWindow – on the thread that’s going to execute the Python code. Debugging in Python is per thread. Even if you execute the same code in the same ScriptScope with the same ScriptEngine but on a different thread, the traceback handler calls won’t fire. The way DebugWindow is written, it will only support debugging a single thread, but it would be pretty straightforward to support multiple threads by changing the way OnTracebackReceived gets signaled to continue.

Speaking of OnTracebackReceived, this was my initial basic implementation of it:

As we saw, the DebugWindow is running on a different thread than the traceback handler call will come in on. So OnTracebackReceived needs to invoke a new call on the correct thread by using Dispatcher.BeginInvoke. Even though OnTracebackReceived is always called on the main window thread, it still has access to the properties of the debug window thread like its Dispatcher. I used BeginInvoke to invoke OnTraceback asynchronously – OnTraceback isn’t going to return anything interesting and we’re going to wait on an AutoResetEvent before continuing anyway so I didn’t see any reason to use a synchronous call.

We’ll discuss OnTraceback more next post, but basically it will configure the UI for the traceback event that happened. Then DebugWindow will wait for user input. When the user indicates they want to resume execution, the command handler in question will set _dbgContinue and the original traceback will return so execution can continue.

Hybrid App Debugging Aside - The DLR Hosting API

In my series on Hybrid App Debugging, I showed the following code for executing a Python file in a hybrid C#/IronPython app.

The DLR Hosting API has three distinct levels of functionality. As simple as this is, technically it’s level 2 since it’s using a ScriptEngine directly. If you wanted to use the simplest level 1 hosting API, you could use runtimes instead of engines and save a line of code.

The ScriptRuntime version of ExecuteFile doesn’t include an overload that takes a ScriptScope like ScriptEngine does, so instead you add the items variable to the globals scope. However, this doesn’t automatically add the items object to every child scope – you have to explicitly import items into the local scope if you want to use it. So for Python, that means you need to add “import items” to the top of the GetThings.py script. Nothing else changes.

Personally, I find DLR Hosting API Level 2 to be straightforward and easy enough to understand, so I tend to code to that level by default. I actually had to go read the doc to discover the ScriptRuntime.Globals property and talk to Dino about importing those variables into a local scope. However, I wanted to point out that nothing in my Hybrid App Debugging sample so far is really dependent on the level 2 API. If you just want to execute some Python files in the context of your C# application, you can stick with the simpler level 1 API if you want. You can even use lightweight debugging with the level 1 API – there’s an overload of the SetTrace extension method for ScriptRuntimes just as there is for ScriptEngines. Just something to keep in mind.

Hybrid App Debugging – TracebackDelegate and SetTrace

Now that I’ve introduced my simple hybrid GetThings app, we need to set about adding support for debugging just the IronPython part of the app via the new lightweight debugging functionality we’re introducing in 2.6. Note, the code is up on github, but isn’t going to exactly match what I show on the blog. Also, I have a post RC1 daily build of IronPython in the Externals folder since I discovered a few issues while building this sample that Dino had to fix after RC1. Those assemblies will be updated as needed as the sample progresses.

We saw last time how how easy it is to execute a Python script to configure a C# app – only four lines of code. If we want to support debugging, we need to add a fifth:

You’ll notice the one new line – the call to engine.SetTrace. This is actually an extension method – ScriptEngine is a DLR hosting API class and but SetTrace is IronPython specific functionality [1]. If you look at the source of Python.SetTrace, you’ll see that it’s just a wrapper around SysModule.settrace, but it avoids needing to get the engine’s shared PythonContext yourself.

SetTrace takes a TracebackDelegate as a parameter. That delegate gets registered as the global traceback handler for the Python engine (on that thread, but we’ll ignore threading for now). Whenever that engine enters a new scope (i.e. a new function), the IronPython runtime calls into the global traceback handler. While the traceback handler runs, execution of the python code in that engine is paused. When the traceback handler returns, the engine resumes executing python code.

In addition to the global traceback handler, each scope has a local traceback handler as well. The TracebackDelegate type returns a TracebackDelegate which is used as the local traceback handler for the next traceback event within that scope. Traceback handlers can return themselves, some other TracebackDelegate, or null if they don’t want any more traceback events for the current scope. It’s kinda confusing, so here’s a picture:

You’ll notice three different traceback event types in the picture above: call, line and return. Call indicates the start of a scope, and is always invoked on the global traceback handler (i.e. the traceback passed to SetTrace). Line indicates the Python engine is about to execute a line of code and return indicates the end of a scopes execution. As you can see, the runtime uses the return value of the traceback for the next tracing call until the end of the scope. The return value from the “return” event handler is ignored.

So now that we know the basics of traceback handlers, here’s a simple TracebackDelegate that simply returns itself. The “Hello, world!” of traceback debugging if you will.

If you run this code, there will be no functional difference from the code before you added the SetTrace call. That’s because we’re not doing anything in the traceback handler. But if you run this in the debugger with a breakpoint on this function, you’ll see that it gets called a bunch of times. In the python code from the last post, there are three scopes – module scope, download_stuff function scope and the get_nodes function scope. Each of those function scopes will have a call and return event, plus a bunch of line events in between.

The parameters for TracebackDelegate are described in the Python docs. The frame parameter is the current stack frame – it has information about the local and global variables, the code object currently executing, the line number being executed and a pointer to the previous stack frame if there is one. More information on code and frame objects is available in the python data model (look for “internal types”). Result is the reason why the traceback function is being called (in Python docs, it’s called “event” but that’s a keyword in C#). IronPython supports four traceback results: “call”, “line” and “return” as described above plus “exception” when an exception is thrown. Finally, the payload value’s meaning depends on the traceback result. For call and line, payload is null. For return, payload is the value being returned from the function. For exception, the payload is information about the exception and where it was thrown.

As I mentioned above, python code execution is paused while the traceback handler executes and then continues when the traceback handler returns. That means you need to block in that function if you want to let the user interact with the debugger. For a console app like PDB, you can do that with a single thread of execution easily enough. For a GUI app like GetThings, that means running the debugger and debugee windows on separate threads. And as I alluded to, tracing for Python script engines is per thread. So next time, we’ll look deeper into how to use multiple threads for lightweight debugging a hybrid app.

[1] Eventually, I’d like to see IronRuby support lightweight debugging as well. However, there’s no built in mechanism for Ruby debugging the way there is for Python, so it’s less clear how we should expose debugging to the Ruby developer. We’d also want to build a language neutral DLR Hosting API mechanism for lightweight debugging as well at that point. But honestly, we have higher priorities at this point.

Lightweight Debugging for Hybrid C#/IronPython Apps

One of the IronPython scenarios that I’m hearing more and more about recently is for polyglot programs. In these scenarios, part of the application is built in IronPython other parts are build in compiled, statically typed languages like C# or Visual Basic. Sometimes, programs are written this way to allow the C# app to access a Python library, like my Pygments for WL Writer plugin. Other programs want to be customizable by the end user, like Intellipad. Whatever the reason, I think that the number of these hybrid polyglot programs is going up, which partially explains why the C# team added the new dynamic type to C# 4.0.

(FYI: the You had me at “dynamic” shirt above is available for sale in my Zazzle store along with my Architecture Help 5¢ shirt)

The thing is that if you’re going to build polyglot apps, you’re probably going to want the ability to debug polyglot apps as well. I’ve written extensively about building a debugger for IronPython. However, ipydbg uses the CLR debugger under the hood which means you have to have the debugger and the code it’s debugging in separate processes. That’s a huge design burden for building a debuggable polyglot application. Luckily, as of IronPython 2.6, we support Python’s built-in trace debugging capability (aka sys.settrace). While you can use this in pure Python apps (like PDB), you can also use it polyglot C# (or VB)/IronPython apps as well. If only someone were to take the time to build a sample and document what he did along the way…

Hey, that sounds like PM work!

Seriously, let me introduce you to the worlds simplest Twitter application: GetThings. The app downloads a list of my tweets via the Twitter API and displays them in a list box. The UI is written in C# while the tweet download code is written in Python. Clearly, this is a pretty brain dead app – but the point isn’t to build a great Twitter app but rather to show how to use the settrace API from C#.

I’ve stuck the code up on GitHub. If you want to see the basic app in action sans debugging, start with the initial checkin. As you can see here, basic C# / IronPython integration is pretty trivial. I’m simply creating an engine and a scope, adding the list boxes’ Items property to the scope, and executing the getthings.py file from the disk.

Since GetThings.py is just a text file, the user can modify it to get a list of anything they want – some other user’s timeline, the public timeline, or even – gasp! – something not from Twitter! In fact, as you see below, I’ve actually modified it to pull the tweets from a file on disk so I can avoid hitting the network on every run.

OK, so that’s the basics of the world’s simplest hybrid C#/IronPython Twitter application. Next up, I’ll add the settrace basics.

IronPython Post 2.6 Roadmap

It’s not quite out the door yet, but things have gotten quieter around here since we shipped the Release Candidate of 2.6. But there’s no rest for the dynamic, so we’ve already started thinking about what we do next.

Since we shipped 2.0 last December, we’ve shipped two service releases: 2.0.1 two months later in February and 2.0.2 four months after that in June. We weren’t planning on doing a 2.0.3 release, but then we discovered the CLR folks made a breaking change to partial trust in Windows 7. David recently emailed the IronPython mailing list looking for feedback on other must-fix bugs we can get to for 2.0.3. If you’ve got an opinion on 2.0.3 must-fix bugs, please respond to that thread.

For 2.6 service releases, we’re looking to tighten up the timeframe a bit from last time. We’re planning to ship service releases of 2.6 in December and February. However, since we don’t have a major release ship vehicle in the 2.x branch until Python 2.7 next summer, these service releases may contain new functionality in addition to bug fixes. In particular, we will look to include any missing modules work that I discussed in my RC announcement post in these service releases.

In addition to the IronPython 2.6 service releases, we’ll also continue to track the Visual Studio 2010 beta cycle as it heads to RTM. I can’t comment on VS2010 dates, but I can say that we will ship a CTP of IronPython 2.6 for .NET Framework 4.0 for Visual Studio 2010 Beta 2 and that we are committed to shipping the RTM of our .NET 4.0 version the day that Visual Studio 2010 is publicly available. Given that dynamic the big new feature of C# 4.0, we want to make sure IronPython is ready to go as soon as C# 4.0 is available.

BTW, we are well aware that “IronPython 2.6 for .NET Framework 4.0” is a long ugly name. Any suggestions on a different one? We’ve been thinking “IronPython 2.6 R2” since the “R2” nomenclature is big around here. But “R2” doesn’t really capture the essence of the .NET 4.0 compatible release. Besides, when has the dynamic languages team at Microsoft ever done ANYTHING because it was popular with the Microsoft marketing folks? :)

At this point, we’ve got the next few months mapped out, but not much more beyond that. Specifically, we have two gaping holes in the roadmap:

• Visual Studio Integration
• IronPython 3.x

For now, I’m going to leave these holes unfilled. Currently, the rest of my VS Languages teammates (along with the rest of DevDiv) are heads down driving towards beta 2 of Visual Studio 2010. Once they reach that milestone, planning on Visual Studio v.next will begin. Those plans have the potential for impacting how the IronPython team proceeds going forward.

Frankly, several members of the dynamic languages team have been pitching in with the “drive to beta 2” effort so we’re a little more short handed than usual. If we get those people back, then we’re in a much better position to execute on both VS Integration and IronPython 3.x. If we don’t, then we have to make some hard choices about how to proceed. I’m guessing most would agree that VS integration is more important IronPython 3.x support. However, those aren’t equal efforts by any stretch of the imagination. How valuable is VS integration if, for example, we don’t have the bandwidth to build decent intellisense? As I said, hard choices.

If you want to make your voice heard on Microsoft’s level of commitment to IronPython, make sure you go rate the IronPython integration issue on Microsoft Connect. (while you’re there, rate the IronRuby integration issue too.) The IronPython integration is currently the top rated open issue on the VS Connect site and is the second highest vote getter out of all the Connect issues (active or otherwise) logged since VS 2010 shipped it’s first CTP! [1]

[1] Seriously, the next most recent Connect issue with more votes than IronPython integration is HttpRuntime.ProcessRequest() does not work in IIS7 Integrated Pipeline Mode from November 2007. The next one before that was Create Service Pack for Visual Studio 2005 from November 2005.

IronPython 2.6 Release Candidate

We released the IronPython 2.6 release candidate on CodePlex yesterday. If all goes well, this will be the ONLY RC and we’ll republish it as the RTW build in about a month. So if you’ve been holding off on experimenting with the 2.6 release, now’s the time to jump in with both feet.

As I’ve written before on this blog, this is a HUGE release for us:

• Python 2.6 features such as with statement, class decorators and byte literals
• Adaptive Compilation
• __clrtype__ Metaclasses
• ctypes and Frames support
• Lightweight Debugging
• 417 bugs fixed!

Anyway, with 2.6 winding down, the IronPython team finds ourselves in a unique position that we’ve never been in before: caught up. As far as I can tell, most of the Python community hasn’t made the move to Python 3.1 and Python 2.7 is looking like it will be released next summer. So IronPython is caught up with the latest version of Python most of the Python community appears to be using.

So that begs the question: what do we do now?

Of course, we want to hear from you regarding our next steps, but some things we are looking at include:

• .NET Framework 4.0
  We’ve shipped CTP releases of IronPython for each public beta of Visual Studio 2010 and  .NET Framework 4.0. You can expect that to continue as Visual Studio 2010 winds down and ships.
• Fixing Bugs
  417 bugs fixed is good, but there are still 839 active work items in our issue tracker. In previous releases, we’ve done minor bug fix releases every few months so we’ll probably keep up that cadence. Make sure you go vote for bugs you think are important.
• App Compatibility
  One thing we can do is take some of the more popular Python apps such as Django and Mercurial and make sure they run well on IronPython. In some cases, there may need to be changes to the Python apps to get them to run on IronPython (for example, see Jeff Hardy’s patch for running Django on IronPython) which assuredly means more time with lawyers for me.
• Missing Modules
  While IronPython 2.6 is our implementation of Python 2.6, there are binary modules we haven’t implemented yet like _csv, _ast and pyexpat. You could consider this App Compatibility work, but we have a different internal process for bug fixing and implementing new features so I broke this out separately.
• Documentation
  Invariably an area for improvement in all software, our doc story today is pretty much “go look at docs.python.org”. Of course, that doesn’t cover any IronPython specific functionality. Of course, what would be great would be to combine existing Python docs with IronPython specific docs in a single reference, which also assuredly means more time with lawyers for me.
• Tutorials
  If you haven’t seen Michael Foord’s Try Python, it’s awesome. However, it was recently pointed out to me that it doesn’t include any IronPython specific behavior (importing and interoperating with .NET types for example). The IronRuby Tutorial has specific IronRuby features and it would be awesome to do the same for IronPython. Of course, if we wrote new tutorials in reStructured Text, then I’m guessing it would be easy for Michael to include it in Try Python via his rst2xaml tool.
• “Just Text” in Silverlight
  If you haven’t seen MIX Online Labs Gestalt project, browse to one of the samples and View Source. Python as “just text” in the browser. Cool, eh? Jimmy is working on implementing the “just text” model for the Silverlight version of IronPython (and IronRuby).
• Visual Studio
  There’s no work item for VS Integration in our issue tracker, but there have been 112 votes for IronPython integration (as well as 79 votes for IronRuby integration) in the VS 2010 connect bug database. No promises here, but we are acutely aware of how popular this suggestion is.
• CodePlex Foundation
  The CodePlex Foundation is a new non-profit foundation sponsored by Microsoft with the explicit mission “to enable the exchange of code and understanding among software companies and open source communities.” As one of the oldest open source projects at Microsoft, we are very interested in the CodePlex Foundation as you might imagine. However, CodePlex Foundation is VERY new - everything is tagged “interim” for the first 100 days. Once the Foundation elects non-interim board members and establishes things like a charter and by-laws, you can be sure we’ll be investigating it more thoroughly. In other words, definitely more time with lawyers for me.

In case I haven’t said it lately, it’s great working on the IronPython team with Dino Viehland, Dave Fugate and David DiCato (The IronRuby guys aren’t bad either!). Also, I may whine about the amount of time I spend with lawyers, but honestly Yong, Kathryn and Kevin – our main LCA contacts – do a great job helping us figure out how and where to push the envelope so thanks to them as well.

Danish University Tour Trip Report

I’ve been back from Denmark for a week – it took me that long to get back on Pacific time zone and dig out from the mountain of email that collected while I was gone. But I got word from my esteemed host Martin Esmann that the video of my TechTalk had been posted to Channel 9, so it seems as good a time as any for my trip report.

At each of these universities, I did two talks. The first was Pumping Iron: The State of Dynamic Languages on the .NET Framework. That’s the one in the Channel 9 video. The other talk was Developing with the DLR, which I’ve posted to my Skydrive. That talk was more focused on the CLR and DLR as a platform for language development. If there’s interest (leave a comment), I’ll record audio for that presentation and post it up on SlideShare or something like that.

Aalborg University

First stop of the University Tour was Aalborg University – about a four hour drive from Copenhagen (or as the Danish spell it København). As you can see on the map, we started with the furthest university away then worked our way back to Copenhagen. Martin picked me up at the airport and we hit the road. I was horribly jet lagged, but we spent most of the drive geeking out about programming languages (when I wasn’t napping).

Each of the universities had a different focus that I was interested in. At Aalborg, it was in teaching programming. Given the popularity of Python in education, we had lots to talk about. My host at Aalborg was Bent Thomsen who has done significant research on this topic, as has his wife Lone. One of the areas we discussing in particular was about teaching classes vs. objects first. Apparently, these days there’s significant momentum behind teaching class first, but the folks at Aalborg – as I understand it – have seen that approach has been ineffective. That squared with a talk I saw on teaching OOP with Python that I saw last year at PyCon by Dr. Goldwasser.

The other area we discussed was teaching compiler design. Clearly, this is an area I thought the DLR could be a big help in. Bent primarily uses Programming Language Processors in Java to teach this course, but he finds that it’s a little out of date + he’s much more interested in the compiler aspect than the interpreter aspect of that book. The idea of generating DLR Expression Trees which can be compiled into IL with a simple call to LambdaExpression.Compile and can then run directly on the CLR seemed to sound appealing to him.

Aarhus University

Next up was Aarhus University. I didn’t get to spend as much time with the folks from Aarhus, though our host Olivier Danvy did help me find a “uniquely Danish” gift for my wife (though I also brought her Danish bread & chocolate). We spent some time talking about F# and compiler design, and Olivier made this great comment over lunch that “OCaml is a domain specific language for compiler writers”. That is very true.

Olivier had to leave for a previous engagement, so Jan Midtgaard - who teaches Aarhus’ Compilation class – hosted me for the actual talks. In that class, they implement a subset of Java known as Joos in AspectJ. One of the cool things about this class is how they grade the compilers – you upload your complier code to a web site and they run it against an existing test framework. They couldn’t give the students the test framework directly because it would be too easy to reverse engineer the tests from the compiled Java code (I assume the Java world has the equivalent of Reflector?) Like Aalborg, I spent significant time discussing the idea of targeting the DLR in their compiler class.

University of Southern Denmark

Of the four universities I visited, this was probably the poorest fit for the content I was delivering. On the other hand, it was awesome – USD has a strong focus on Robotics. They even have a RoboDays Robot Festival in Odense. Unfortunately, the festival was the weekend after I visited, so many of the robots were out in preparation for the festival. However, our host Ulrik Schultz did bring us to to visit the Modular Robotics Research Lab so I got to see these guys in action:

Real transforming robots! Not quite Optimus Prime mind you, but you gotta start somewhere. These robots are call ATRON and they’re pretty impressive to see in action. However, they are kinda weak computationally – they only have 128k of flash memory + 4k of ROM. 4K! However, the next generation of these bots is supposed to have like 64MB of memory and a FPGA – more than enough power to use the the Micro Framework.

University of Copenhagen

The final stop on my university tour was University of Copenhagen, where I was hosted by Fritz Henglein of the Algorithms and Programming Languages Group. He’s done some research on dynamic languages, but these days he and his team are doing some fascinating research on the intersection of business and programming languages. He leads a project called 3gERP in partnership with Microsoft Development Center Copenhagen to “develop a standardized, yet highly configurable and flexible, global ERP system for SME’s based on fundamentally new software architecture.” That topic isn’t as interesting to me in my current role in the VS Languages team, but was very relevant to my background in services architecture and MSIT.

Copenhagen .NET Users Group

While I was in Denmark, I got to speak to the local .NET users group about IronPython. These are always fun since they are less formal. Also, this was a much more .NET knowledgeable crowd than I had had all week so I got some deeper questions about the .NET stack. Among the members of that user group is Mads Kristensen, founder of the BlogEngine.NET project. So I put together a special demo “integrating” IronPython into BlogEngine.NET. It was kinda silly – I wrote an extension that would execute any blog post as python that was in the RunPython category. But it was also mind-numbingly simple:

I’d love to see REAL IronPython integration into BlogEngine.NET, but I needed something I could do in a couple of hours late at night in a code base I wasn’t familiar with.

Pumping Iron TechTalk

Finally, on Friday I delivered my final talk of the week, which is available to watch and download from Channel 9. It was a packed house and I was a little nervous about having the talk recorded, but I think it went well. Certainly, it seemed well received by the audience.

I ended the work part of my trip to Denmark with an interview with a Danish IT newspaper (which I’ll post a link to when it gets published) and a sit down with the Dyanmics:NAV team. My wife always tells me that she’d love it if I got a 6-12 month assignment to work overseas, so I tried to convince the NAV team that they REALLY want to integrate IronPython like the Dynamics AX team is doing. If you see a post here about me moving to Copenhagen, you’ll know I was successful! :)

Final Thoughts

Other than the jet lag, which seemed tougher on this trip than when I’ve gone westward to New Zealand, Australia or Malaysia, I had a blast. It was a real treat seeing so much of Denmark and getting to talk to so many interesting people. I even got to do some sight seeing in Copenhagen and Odense. Hans Christian Andersen was born in Odense and so I got to see the house he was born in as well as get my picture taken with this statue of him right outside our hotel. Major thanks to Martin Esmann for inviting me, driving me around – we drove a megameter, aka 1000km, over the course of the trip – and being an all around amazing host.

DevHawk World Tour FY2010

As I’ve done the past two years, here’s a list of all the places I’m going in the next fiscal year. Traditionally, I’ve done this post by calendar year, but all MSFT planning is done by FY and so invariably I miss events early in the calendar year but late in the fiscal (like PyCon last year). I'll be updating this post periodically as I get tapped for more presentations. There are several other conferences I'm considering, submitting sessions for, in discussions with, but these are the ones that are confirmed.

Danish University Tour, Sept 7-11

My FY10 travels first take me to Copenhagen, where I was invited by the local subsidiary to present at four different universities in a single week. Don’t know how much sightseeing I’ll get done, but I’ll sure be talking a lot. My host Martin Esmann writes Stud.blog for Danish ComputerWorld and has a post (in Danish) about my visit. Personally, I am just excited about being featured in something called “Stud.blog”! :) Actually, Stud here means “Student” not “slender, upright members of wood” or any other definition of the term “stud”.

I’ll be visiting Aalborg University, Aarhus University, University of Southern Denmark and University of Copehhagen as well as delivering a TechTalk at the Microsoft Development Center Copenhagen, which is Microsoft’s biggest development center in Europe. I’ll primarily be delivering my Iron Languages introductory talk “Pumping Iron”, but there’s also some interest in language development on the DLR so I’ll be talking on that topic as well.

patterns & practices Summit Redmond 2009, Oct 12-16

This will be my third p&p Summit in a row and fourth in five years. This year, I’m doing a talk called “Not Everything is a new Nail() : How Languages Influence Design”. I was supposed to deliver this talk last year, but got side track with my day job and ended up talking about IronPython instead. Keith has made it VERY clear he doesn’t want another last minute substitution again this year.

Turing award winner Alan Perlis is credited with saying 'A language that doesn't affect the way you think about programming is not worth knowing.' Yet, most programmers rarely venture outside of the comfort zone of statically-typed object-oriented languages. Our heavy use of object-oriented languages influences our thinking to the point that we can?t see alternative approaches at all. This isn?t to say the object-oriented languages are bad, but as is typical in most things, there is no one 'best' way for all situations. In this talk, VS Languages PM Harry Pierson will look at a given software development scenario from both the object-oriented and functional perspectives, in order to see how much on an influence language really has on our engineering efforts.

Tech·Ed Europe 2009, Nov 9-13

I knew I was going to be updating this post over time, but I didn’t expect to have to update it so soon! Literally the day after I posted this, I got the speaker invite for Tech·Ed Europe 2009. My session hasn’t been posted yet, but this is the abstract we submitted:

Dynamic Languages on the Microsoft .NET Framework
The Dynamic Language Runtime (DLR) adds a shared dynamic type system, a standard hosting model, and support for generating fast dynamic code to the CLR. IronPython and IronRuby are Microsoft’s dynamic language implementations on .NET. In this talk, we’ll show you how to interactively create great .NET applications using dynamic languages. You’ll walk away knowing why dynamic languages deserve a spot in your toolbox!

It’s kind of generic, but given that most of the audience probably hasn’t seen IronPython or IronRuby, having broad latitude in my presentation topic is a good thing. I’ll probably deliver a variant of my standard “Pumping Iron” talk like I’m doing in Denmark. I delivered it recently at an internal event with Jimmy, so there’s lots more IronRuby content than there used to be.

The only bummer about doing Tech·Ed Europe is that I’m only doing one measly talk. I’m asking around – I’d love to do a .NET user group or university talk while I’m in town. Any takers?

Microsoft Professional Developers Conference 2009, Nov 17-19

Update: Tech·Ed Europe and PDC are on back-to-back weeks this year so we’ll be sending a teammate-to-be-determined to PDC in my stead. My family is very pleased I won’t be gone for two weeks straight.

Last year, I was on the content team for PDC. This year, that PITA responsibility belongs to someone else so I might actually get real work done in the four weeks leading up to PDC. My team will tell you, last year PDC sucked up 100% of my time for a month as we were driving towards our 2.0 release.

Technically, I haven’t had a talk for PDC accepted yet. But I submitted three and two are looking good (though I assume only one will make it to the actual show) so I thought I’d just go ahead and include it on this post. If/when my talks get accepted, I’ll post links and abstracts. Also, if one of my PDC talks is accepted, I’ll probably submit a talk for SoCal Code Camp as well.

PyCon 2010, Feb 19-21

This will also be my third PyCon in a row, though PyCon last year was a bit of a whirlwind since I had literally just joined the IronPython team. I finally feel like I might have something interesting to present at PyCon this year. Last year Dino and Jim handled the presentation duties from our team (with Michael Foord and Jonathan Hartley delivering a tutorial and Sarah Sutkiewicz speaking on FePy). We already have one announcement that I think is pretty significant lined up and might have a second depending on how hard I can push LCA and management between now and then. Talk proposals are due October 1st, so any suggestions would be appreciated!

CodePlex Editor Role

Ask Sara, I have been bugging her for a LONG time for this CodePlex feature. Actually, my team has been bugging her team for longer than either of us have been in these jobs.

Last week’s CodePlex release includes a feature known as “Editor Role”. If you look at the Project Role Matrix, you’ll notice two primary differences from what the standard logged-in user can do: they can create/edit wiki pages and they can’t rate releases. Developers and Coordinators can’t rate releases either – I guess the idea is that they don’t want members of the team rating their own releases (5 Stars! Again! Wow, we’re awesome!).

Until now, the only way to give members of the community the ability to edit the wiki also gave permission to edit work items, check in source code and make releases. We’re still working on getting Microsoft at large to understand the benefits of community collaboration aspect in open source, but in the meantime we just can’t give those permissions to people off the team. However, we would love to have contributions to our documentation wiki. [1] With the new Editor Role, we’ll be able to grant wiki editor access without any of the other permissions.

Of course, the whole idea of “wiki permissions” kinda flies in the face of the basic wiki design principles. So we’re going to be pretty liberal about handing out editor permissions. If you’re interested in editing the wiki, drop me a line and I’ll get you hooked up. 

Big mega-thanks to the CodePlex team for making this feature happen. I guess I’ll have to find something new to bug Sara about!

[1] You can tell we’re a real open source project because we’re begging for documentation help!

Invoking Python Functions from C# (Without Dynamic)

So I’ve compiled the Pygments package into a CLR assembly and loaded an embedded Python script, so now all that remains is calling into the functions in that embedded Python script. Turns out, this is the easiest step so far.

We’ll start with get_all_lexers and get_all_styles, since they’re nearly identical. Both functions are called once on initialization, take zero arguments and return a PythonGenerator (for you C# devs, a PythonGenerator is kind of like the IEnumerable that gets created when you yield return from a function). In fact, the only difference between them is that get_all_styles returns a generator of simple strings, while get_all_lexers returns a PythonTuple of the long name, a tuple of aliases, a tuple of filename patterns and a tuple of mime types. Here’s the implementation of Languages property:

If you recall from my last post, I initialized the _scope on a background thread, so I first have to wait for the thread to complete. If I was using C# 4.0, I’d simply be able to run _scope.get_all_lexers, but since I’m not I have to manually reach into the _scope and retrieve the get_all_lexers function via the GetVariable method. I can’t invoke the PythonFunction directly from C#, instead I have to use the Invoke method that hangs off _engine.Operations. I cast the return value from Invoke to a PythonGenerator and iterate over it to populate the array of languages.

If you’re working with dynamic languages from C#, the ObjectOperations instance than hangs off the ScriptEngine instance is amazingly useful. Dynamic objects can participate in a powerful but somewhat complex protocol for binding a wide variety of dynamic operation types. The DynamicMetaObject class supports twelve different Bind operations. But the DynamicMetaObject binder methods are designed to be used by language implementors. The ObjectOperations class lets you invoke them fairly easily from a higher level of abstraction.

The last Python function I call from C# is generate_html. Unlike get_all_lexers, generate_html takes three parameters and can be called multiple times. The Invoke method has a params argument so it can accept any number of additional parameters, but when I tried to call it I got a NotImplemented exception. It turns out that Invoke currently throws NotImplemented if it receives more than 2 parameters. Yes, we realize that’s kinda broken and we are looking to fix it. However, it turns out there’s another way that’s also more efficient for a function like generate_html that we are likely to call more than once. Here’s my implementation of GenerateHtml in C#.

Instead of calling Invoke, I convert the PythonFunction instance into a delegate using Operations.ConvertTo which I then cache and call like any other delegate from C#. Not only does Invoke fail for more than two parameters, it creates a new dynamic call site every time it’s called. Since get_all_lexers and get_all_styles are each only called once, it’s no big deal. But you typically call generate_html multiple times for a block of source code. Using ConvertTo generates a dynamic call site as part of the delegate, so that’s more efficient than creating one on every call.

The rest of the C# code is fairly pedestrian and has nothing to do with IronPython, as all access to Python code is hidden behind GenerateHtml as well as the Languages and Styles property.

So as I’ve shown in the last few posts, embedding IronPython inside a C# application – even before we get the new dynamic functionality of C# 4.0 – isn’t really all that hard. Of course, we’re always interested in ways to make it easier. If you’ve got any questions or suggestions, please feel free to leave a comment or drop me a line.

Embedding Python Scripts in C# Applications

Now that I’ve got Pygments and its dependencies packaged up in an easy-to-distribute assembly, I need to be able to call it from C#. However, if you pop open pygments.dll in Reflector, you’ll notice it’s not exactly intuitive to access. Lots of compiler generated names like pygments$12 and StringIO$64 in a type named DLRCachedCode. Clearly, this code isn’t intended to be used by anything except the IronPython runtime.

So we better create one of those IronPython runtime thingies.

As you can see in the layer diagram to the left, PygmentsCodeSource is split into two parts – a C# part and a Python part. The Python part is very simple – just importing a couple of Pygments functions into the global namespace and a simple helper function to generate syntax highlighted HTML from a given block of code in a given language and style. The code itself is pretty simple. Note the reference to the pygments assembly I described last post. Here’s the entire file:

Instead of including this in the Pygments assembly, I embedded this file as a resource in my C# assembly. This way, I could use the standard DLR hosting APIs to create a script source and execute this code. I did have to build a concrete StreamContentProvider class to wrap the resource stream in, but otherwise, it’s pretty straight forward.

Once I got the engine and script source set up, all that remains is setup a script scope to execute the script source in. For this specific application, it’s probably overkill to have a scope per instance – I think the syntax highlighting process is stateless so a single scope should be easily shared across multiple PygmentsCodeSource instances. But I didn’t take any chances, I created a script scope per instance to execute the source in.

You’ll notice that I’m executing the source in the scope on a background thread. That’s because it takes a while to execute, especially the first time. However, I don’t actually use the Python code until after the user types or copies a block of code into the UI and presses OK. In my experience, executing the Python code is typically finished by the time I get code into the box and press OK. I just need to make sure I add an _init_thread.Join guard anywhere I’m going to access the _scope to be sure the initialization is complete before I try to use it.

In the next, and last, post in this small series we’ll see how to invoke Python functions in the _scope I initialized above from C#.

Compiling Python Packages into Assemblies

In looking at my hybrid IronPython / C# Windows Live Writer plugin, we’re going to start at the bottom with the Pygments package. Typically Python packages are a physical on-disk folder that contain a collection of Python files (aka modules). And during early development of Pygments for WLWriter, that’s exactly how I used it. However, when it can time for deployment, I figured it would be much easier if I packaged up the Pygments package, my custom HTML formatter and the standard library modules that Pygments depends on into a single assembly.

IronPython ships with a script named pyc for compiling Python files into .NET assemblies. However, pyc is pretty much just a wrapper around the clr module CompileModules function. I wrote my own custom script to build the Pygments assembly from the files in a the pygments and pygments_dependencies folders.

Most of this code is a custom implementation of walk. I have all the IronPython and DLR dlls including ipy.exe checked into my source tree, but I don’t have the standard library checked in. Other than that, the code is pretty straight forward – collect a bunch of files in a list and call CompileModules.

The problem with this approach is that IronPython isn’t doing any kind of dependency checking when we compile the assembly. If you pass just the contents of the Pygments package into CompileModules, it will emit an assembly but that assembly will still depend on some modules in the standard library. If those aren’t available, the Pygments assembly won’t load. I’d love to have an automatic tool to determine module dependencies, but since I didn’t have such a tool I used a brute-force, by-hand solution. I wrote a small script to exercise the Pygments assembly. If there were any missing dependencies, test_compiled_pygments would throw an exception indicating the missing module. For each missing dependency, I copied over the missing dependency, recompiled to project and tried again. Lather, rinse, repeat. Not fun, but Pygments only depended on seven standard library modules so it didn’t end up taking that long.

So having gone down this path of compiling Python files into an assembly, would I do it again? For an application with an installer like this one, yes no question. I added the Pygments assembly as a reference to my C# library and it got added to the installer automatically. That was much easier than managing all of the Pygments files and its dependencies in the installer project manually. Plus, I still would have had to manually figure out the dependencies unless I chose to include the entire standard library.

I will point out that the compiled Pygments assembly is the largest single file in my deployed solution. It clocks in at 2.25MB. That’s about twice the size of the Python files that I compiled it from. So clearly, I’m paying for the convenience of deploying a single file in space and maybe load time. [1] I’m also paying in space for a private copy of IronPython and the DLR – the two IronPython and five DLR assemblies clock in around 3.16MB. In comparison, the actual Writer plugin assembly itself is only about 25KB! But for an installed desktop app like a WLWriter plugin, 5MB of assorted infrastructure isn’t worth worrying about compared to the hassle of ensuring a shared copy of IronPython is installed. I mean, even if you don’t know IronPython exists, you can still install and use Pygments for WLWriter. Simplifying the install process is easily worth 5MB in storage space on the user’s computer in my opinion.

Next up, we’ll look at the Python half of the PygmentsCodeSource component, which calls into this compiled Pygments library.

[1] I haven’t done it, but it would be interesting to compare the load time for the single larger pygments assembly vs. loading and parsing the Python files individually. If I had to guess, I’m thinking the single assembly would load faster even though it’s bigger since there’s less overhead (only loading one big file vs. lots of small ones) and you skip the parsing step. But that’s pure guesswork on my part.

Building a Hybrid C# / IronPython App Without Dynamic Type

Arguably, the biggest feature of C# 4.0 is the new dynamic type. And it’ll be great…when it ships. In the meantime, some of us what to build hybrid C# and IronPython applications today, such as my Pygments for Windows Live Writer plugin.

Pygments is a syntax highlighter, written in Python, with support for over one hundred languages. With the exception of a couple of bugs in our importer (discussed here) it works great with IronPython. It’s also extensible, so I was able to easily build a custom formatter to output exactly the HTML I want inserted in my blog posts. So it made perfect sense to use Pygments as the basis of a Windows Live Writer plugin.

As great a tool as Windows Live Writer is, it’s developers haven’t exactly seen the light when it comes to dynamic languages. If you want to create a custom Content Source for Windows Live Writer, you have to generate a compiled on-disk assembly with a static type and custom attributes. Not exactly IronPython’s forte, if you know what I mean. I did try and build a pure IronPython solution, but eventually gave up. So I ended up building a hybrid solution. The front end of the plugin as well as the UI elements are written in C# while the syntax highlighter engine is written in IronPython. And since this is running on the current .NET framework, I didn’t have the new fangled C# 4.0 dynamic type to help me.

Over the next couple of blog posts, I want to highlight a few aspects how I built this plugin, including compiling Python packages into assemblies and invoking Python code from C# 3.0 and earlier. If you want to look for your self, the source is up on GitHub.

Pygments for Windows Live Writer v1.0.2

I just uploaded a new version of my Pygments for WL Writer plugin to my skydrive. Nothing major here – some minor UI cleanup + an upgrade to IronPython 2.6 beta 2. Installing over the old version worked on my machine, but that’s as far as my testing has gone. I also pushed the latest source out to GitHub. 

I’m still waiting on a fix for what Dino has taken to calling “Harry’s Pygments Import Bug” – which actually turned out to be three importer bugs. The Pygments lexers package is customized so as to abstract away the specific modules the individual lexers are defined in. I don’t use that functionality – I’m using get_all_lexers and get_lexer_by_name instead – but the bugs caused importing the package to fail so in the mean time I commented out the lines that don’t work under IronPython. I think Dino’s got the fixes for this checked in, but I probably won’t update Pygments for WL Writer again until IronPython 2.6 RC.

I Hate Global.asax

One of the things I’ve always loved about ASP.NET is how easily extensible it is. Back in 2000, I had a customer that wanted to “skin” their website using XML and XSLT – an approach Martin Fowler later called Transform View. We were working with classic ASP at the time, so the solution we ended up with was kind of ugly. But I was able to implement this approach in ASP.NET in a few hundred lines of code, which I wrote up in an MSDN article published back in 2003. In the conclusion of that article, I wrote the following:

Using ASP.NET is kind of like having your mind read. If you ever look at a site and think "I need something different," you'll most likely find that the ASP.NET architects have considered that need and provided a mechanism for you to hook in your custom functionality. In this case, I've bypassed the built-in Web Forms and Web Services support to build an entire engine that services Web requests in a unique way.

Nearly ten years later, I finally ran into a situation where ASP.NET failed to read my mind and doesn’t provide a mechanism to hook in custom functionality: Global.asax.

I always thought of global.asax as an obsolete construct primarily intended to ease migration from classic ASP. After all, ASP.NET has first class support for customizing request handling at various points throughout the execution pipeline via IHttpModule. Handling those events in global.asax always felt vaguely hacky to me.

However, what I didn’t realize is that there are some events that can only be handled via global.asax (or its code behind). In particular, Application_Start/End and Session_Start/End can only be handled in global.asax. Worse, these aren’t true events. For reasons I’m sure made sense at the time but that I don’t understand, the HttpApplicationFactory discovers these methods via reflection rather than by an interface or other more typical mechanism. You can check it out for yourself with Reflector or the Reference Source – look for the method with the wonderful name ReflectOnMethodInfoIfItLooksLikeEventHandler. No, I’m not making that up.

The reason I suddenly care about global.asax is because Application_Start is where ASP.NET MVC apps configure their route table. But if you want to access the Application_Start method in a dynamic language like IronPython, you’re pretty much out of luck. The only way to receive the Application_Start pseudo-event is via a custom HttpApplication class. But you can’t implement your custom HttpApplication in a dynamically typed language like IronPython since it finds the Application_Start method via Reflection. Ugh.

If someone can explain to me why ASP.NET uses reflection to fire the Application_Start event, I’d love to understand why it works this way. Even better - I’d love to see this fixed in some future version of ASP.NET. You come the only way to configure a custom HttpApplication class is to specify it via global.asax? Wouldn’t it make sense to specify it in web.config instead?

In order to support Application_Start for dynamic languages you basically have two choices:

1. Build a custom HttpApplication class in C# and reference it in global.asax. This is kind of the approach used by Jimmy’s ironrubymvc project. He’s got a RubyMvcApplication which he inherits his GlobalApplication from. Given that GlobalApplication is empty, I think he could remove his global.asax.cs file and just reference RubyMvcApplication from global.asax directly.
2. Build custom Application_Start/End-like events out of IHttpModule Init and Dispose. You can have multiple IHttpModule instances in a given web app, so you’d need to make sure you ran fired Start and End only once. This is the approach taken by the ASP.NET Dynamic Language Support. [1]

So here’s the question Iron Language Fans: Which of these approaches is better? I lean towards Option #1, since it traps exactly the correct event though it does require a global.asax file to be hanging around (kind of like how the ASP.NET MVC template has a blank default.aspx file “to ensure that ASP.NET MVC is activated by IIS when a user makes a "/" request”). But I’m curious what the Iron Language Community at large thinks. Feel free to leave me a comment or drop me an email with your thoughts.

[1] FYI, I’m working on getting the code for ASP.NET Dynamic Language Support released. In the meantime, you can verify what I’m saying via Reflector.

Microsoft.Scripting.Debugging

If you’ve compiled IronPython from source recently, you may have noticed a new DLL: Microsoft.Scripting.Debugging. This DLL contains a lightweight, non-blocking debugger for DLR based languages that is going to enable both new scenarios as well as better compatibility with CPython. Needless to say, we’re very excited about it.

When I was actively working on my ipydbg series, I got several emails asking about using it in an embedded scripting scenario. Unfortunately, the ipydbg approach doesn’t work very well in the embedded scripting scenario. ipydbg uses ICorDebug and friends, which completely blocks the application being debugged. This means, your debugger has to run in a separate process. So either you run your debugger in your host app process and your scripts in a separate process or you run your debugger in a separate process debugging both the scripts and the host app. Neither option is very appealing.

Now with the DLR Debugger, you can run all three components in the same process. I think of the DLR debugger as a “cooperative” debugger in much the same way that Windows 3.x supported cooperative multitasking. It’s also known as trace or traceback debugging. Code being debugged yields to the debugger at set points during its execution. The debugger then does whatever it wants, including showing UI and/or letting the developer inspect or modify program state. When the debugger returns, execution of the original code continues until the next set point wherein the process repeats itself.

The primary point of entry for the DLR Debugger is the DebugContext class. Notable there is the TransformLambda method, which takes a normal DLR LambdaExpression and transforms it into a cooperatively debugged LambdaExpression. LambdaExpressions can contain DebugInfoExpressions – typically we insert them at the start of every Python code line as well as one at the end of the function. When we run IronPython in debug mode (i.e. –D), those get turned into sequence points as we saw back when I was working on ipydbg. When using the DLR Debugger, those DebugInfoExpressions are transformed into calls out to IDebugCallback.OnDebugEvent. The DLR Debugger implements the IDebugCallback interface on the TracePipeline class which also implements ITracePipeline. In OnDebugEvent, TracePipeline calls out to an ITraceCallback instance you provide. The extra layer of indirection means you can change your traceback handler without having to regenerate the debuggable version of your functions.

Of course, we hide all this DLR Debugger goo from you in IronPython. Python already has a mechanism for doing traceback debugging – sys.settrace. Our ITraceCallback, PythonTracebackListener, wrapps the DLR Debugger API to expose the sys.settrace API. That makes this feature a twofer – new capability for IronPython + better compatibility with CPython. Instead of needing a custom tool (i.e. ipydbg) you can now use PDB from the standard Python library (modulo bugs in our implementation). I haven’t been working on ipydbg recently since you’ll be able to use PDB soon enough.

For those hosting IronPython, we also have a couple of static extension methods in our hosting API (look for the SetTrace functions in IronPython\Hosting\Python.cs). These are simply wrappers around sys.settrace, so it has the same API regardless if you access it from inside IronPython or from the hosting API. But if you’re hosting IronPython in a C# application, those extension methods are very convenient to use.

This debugger will be in our regular releases of IronPython as of 2.6 beta 2 which is scheduled to drop at the end of this month. For those who just can’t wait, it’s available as source code starting with yesterday’s changeset. Please let us know what you think!

__clrtype__ Metaclasses: Named Attribute Parameters

In my last post, I added support for custom attribute positional parameters . To finish things off, I need to add support for named parameters as well. Custom attributes support named parameters for public fields and settable properties. It works kind of like C# 3.0’s object initalizers. However, unlike object initalizers, the specific fields and properties to be set on a custom attribute as well as their values are passed to the CustomAttributeBuilder constructor. With six arguments – five of which are arrays – it’s kind of an ugly constructor. But luckily, we can hide it away in the make_cab function by using Python’s keyword arguments feature.

You’ll notice that make_cab now takes a third parameter: the attribute type and the tuple of positional arguments we saw last post. This third parameter “**kwds” is a dictionary of named parameters. Python supports both positional and named parameter passing, like VB has for a while and C# will in 4.0. However, this **kwds parameter contains all the extra or leftover named parameters that were passed in but didn’t match any existing function arguments. Think of it like the params of named parameters.

As I wrote earlier, custom attributes support setting named values of both fields and properties. We don’t want the developer to have to know if given named parameter is a field or property, so make_cab iterates over all the named parameters, checking first to see if it’s a property then if it’s a field. It keeps a list of all the field / property infos as well as their associated values. Assuming all the named parameters are found, those lists are converted to tuples and passed into the CustomAttributeBuilder constructor.

In addition to the change to make_cab, I also updated cab_builder slightly in order to pass the **kwds parameter on thru to the make_cab function. No big deal. So now, I can add an attribute with named parameters to my IronPython class and it still looks a lot like a C# attribute specification.

As usual, sample code is up on my skydrive.

Now that I can support custom attributes on classes, it would be fairly straightforward to add them to methods, properties, etc as well. The hardest part at this point is coming up with a well designed API that works within the Python syntax. If you’ve got any opinions on that, feel free to share them in the comments, via email, or on the IronPython mailing list.

__clrtype__ Metaclasses: Positional Attribute Parameters

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

In make_cab, I build a tuple of CLR types from the list of positional arguments that was passed in. If you haven’t seed the *args syntax before, it works like C#’s params keyword – any extra arguments are passed into the function as a tuple names args. I use Python’s built in map function (FP FTW!) to build a tuple of CLR types of the provided arguments, which I then pass to GetConstructor. Previously, I passed an empty tuple to GetConstructor because I wanted the default constructor. If you don’t pass any positional arguments, you still get the default constructor. Once I’ve found the right constructor, I pass it and the original tuple of arguments to the CustomAttributeBuilder constructor.

One major benefit of this approach is that it simplifies the metaclass code. Since _clrclassattribs is now a list of CustomAttributeBuilders, now I just need to iterate over that list and call SetCustomAttribute for each.

The only problem with this approach is that specifying the list of custom attributes is now extremely verbose. Not only am I specifying the full attribute class name as well as the positional arguments, I’m also having to insert a call to make_cab. Previously, it kinda looked like a C# custom attribute, albeit in the wrong place. Not anymore. So I decided to write a function called cab_builder to generates less verbose calls to make_cab:

The cab_builder function returns an anonymous lambda function that closes over the attrib_type variable. Python lambdas are just like C# lambdas, except that they only support expressions [1]. The results of calling the lambda returned from cab_builder is exactly the same as calling make_cab directly, but less verbose. And since I named the function returned from cab_builder Obsolete, now my list of class custom attributes looks exactly like it does in C# (though still in a different place). As usual, the code is up on my skydrive.

If you’re only using the attribute once like this, it is kind of annoying to first declare the cab_builder function. If you wanted to you could iterate over the types in a given assembly, looking for ones that inherit from Attribute and generate the cab_builder call dynamically. However, I’m not sure how performant that would be. Another possibility would be to iterate over the types in a given assembly and generate a Python module on disk with the calls to cab_builder. Then, you’d just have to import this module of common attributes but still be able to include additional calls to cab_builder as needed.

[1] The lack of statement lambdas in Python is one of my few issues with the language.

__clrtype__ Metaclasses: Simple Custom Attributes

I know it’s been a while since my last __clrtype__ post, but I was blocked on some bug fixes that shipped as part of IronPython 2.6 Beta 1. So now let’s start looking at one of the most requested IronPython features – custom attributes!

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

Python list comprehensions use the same square bracket syntax as C# properties, so it kinda looks right to someone with a C# eye – though having the attribute specifications inside the class, rather than above it, is totally different. I wish I could use Python’s class decorators for custom class attributes, but class decorators run after metaclasses so unfortunately that doesn’t work. Also, I can’t leave off the “Attribute” suffix like you can in C#. If I really wanted to, I could provide a new type name in the import statement (“from System import ObsoleteAttribute as Obsolete”) but I thought spelling it out was clearer for this post.

Now that I have specified the class attributes, I can update the metaclass __clrtype__ method to set the attribute on the generated CLR class:

I’m simply iterating over the list of _clrclassattribs (if it exists), getting the default parameterless constructor for each attribute type, creating a CustomAttributeBuilder instance from that constructor and then calling SetCustomAttribute. Of course, this is very simple because we’re not supporting any custom arguments or setting of named properties. We’ll get to that in the next post. In the mean time, you can get the full code for this post from my skydrive.

There is one significant issue with this custom attribute code. Attributes are typically marked with the AttributeUsage attribute that specifies a set of constraints, such as the kind of targets a given attribute can be attached to and if it can be specified multiple times. For example, the MTAThread attribute can’t be specified multiple times and it can only be attached to methods. However, those attribute constraints are validated by the compiler, not the runtime. I haven’t written any code yet to validate those constraints, so you can specify invalid combinations like multiple MTAThread attributes on a class. For now, I’m just going to leave it to the developer not to specify invalid attribute combinations. Custom attributes are passive anyway so I’m figure no one will come looking for a MTAThread attribute on a class or other such scenarios.

However, I’m interested in your opinion: When we get to actually productizing a higher-level API for __clrtype__, what kinds of attribute validation should we do, if any?

IronPython 2.6 Beta 1

In addition to the IronPython CTP for .NET Framework 4.0 Beta 1 I blogged about earlier, we also released the first beta of IronPython 2.6 today. How about that – two IronPython releases in one day! This is our second preview release as we work towards our 2.6 RTM in September. 2.6 Alpha 1 was released back in March.

There are two big new features in this release. The first is our implementation of the ctypes module. The ctypes module is like P/Invoke for Python. It allows Python code to call into unmanaged DLL functions. Here, for example, I’m calling into the standard wprintf function from msvcrt.dll

Between ctypes and Ironclad, I think we’ll eventually be able to load most native Python extensions in IronPython. Woot!

The other big new feature in this release is a real implementation of sys._getframe. _getframe lets you write code that inspects the Python callstack. Previously, we supported _getframe only with a depth of zero which is to say you could inspect the current frame, but no others. Now, by default we don’t implement _getframe at all unless you pass in –X:Frames or –X:FullFrames on the command line. Removing the version of _getframe that only worked for depth zero fixes an issue with collections.py that broke much of the 2.6 standard library in IronPython 2.6 Alpha 1.

The difference between Frames and FullFrames is in what is returned by frame.f_locals member. If you’re running with FullFrames, we hoist all local variables into the heap so they can be accessed by our frame walker. If you’re running with Frames, our ability to access locals up the stack is limited. Sometimes they are available - If you called locals() in a frame up the stack for example, then f_locals will be available – but usually not. There’s a performance difference between the default (i.e. no Frames), –X:Frames and –X:FullFrames, hence why we provide the user fine grained control over the Frame support.

Our performance has gotten better relative to 2.6 Alpha 1. Our PyStone numbers have improved 80% from Alpha 1, similar to where we were in IronPython 2.0.1. We’ve also been able to cut our startup time about 25% from 2.0.1. We’re still an order of magnitude slower than CPython on startup, but we’re getting better. We’re significantly worse on PyBench than we were in 2.6 Alpha 1, but that’s primarily because there’s now a second exception test. As I described back in March, we get killed on the exceptions benchmarks – the two combine to consume nearly 62% of our total run time. Ouch!

Finally, there are bug fixes. Of particular relevance to readers of this blog are a series of fixes that allow me to continue on with my __clrtype__ series. Watch for that soon.

As I said back when we released Alpha 1, the release cycle on 2.6 will be much shorter than it was for 2.0. 2.0 had eight alphas, five betas and two release candidates over the course of around twenty months. We expect 2.6 to have one alpha, two betas and a release candidate over eight months. So please start trying using the beta as soon as you can so you can give us your feedback and we can fix your bugs!

IronPython 2.6 CTP for .NET 4.0 Beta 1

The .NET Framework 4.0 and Visual Studio 2010 Beta 1 is now generally available for download. Jason Zander has a very thorough rundown on some of the new features in this release. Of course, my favorite new features in VS2010 is the new dynamic language support in C# and Visual Basic, which let’s you easily call out to IronPython code from those languages.

For anyone who wants to experiment with interoperating C# or VB with IronPython, we released IronPython 2.6 CTP for .NET 4.0 Beta 1 today. There’s also a walkthru showing how you can use the standard Python library module random from both C# and VB. Note, there’s currently a URL bug in that walkthru – it links to IronPython 2.6 Alpha 1 rather than the .NET 4.0 Beta 1 IronPython CTP. Make sure you pick up the right version of IronPython if you want to try out the walkthru. Looks like they fixed the redirect in the walkthru.

FYI, this is a CTP quality release with about the same functionality as IronPython 2.6 Alpha 1.  Essentially, this is the version of IronPython that was in the source tree when the VS team branched for Beta 1.

If you’ve got any feedback, please drop us a line on the mailing list.

Checkin Comments for IronPython Source

We’ve been slowly but surely increasing the frequency of IronPython source drops. When I joined the team last April, we we only pushing the source about twice a month (sometimes only once a month). By last July, we were pushing source about once a week. Since mid-January, we’ve pushed out the latest source 131 times, which comes to about once a day on average since the start of the year. Big kudos to Dave Fugate, who’s primarily responsible for improving the frequency of our source code drops.

However, while we’ve been good about source code drop frequency, we haven’t been good about transparency. All those source drops have the same less-than-useful checkin comment “Latest IP sources migrated to CodePlex TFS”. If you wanted to know what was changed in a given changeset, you had to do the diff yourself.

But all that opaque code changes is a thing of the past now. Dave upgraded out source push script so that it emails a list of changes as well as the checkin comments whenever we update the source on CodePlex. For example, check out the source push announcement for our latest source drop.  Now we publish added, deleted and modified sources as well as the comments for any checkins included in the source drop.

As Dave said on the mailing list, please let us know if you have any feedback on these source update emails. I think they’re awesome (though I did have one small suggestion) but we want to know what you think.

__clrtype__ Metaclasses Demo: Silverlight Databinding

I’ve gotten to the point where I can actually demo something interesting with __clrtype__ metaclasses: Silverlight Databinding. This is a trivial sample, data binding a list of Products (aka the sample class I’ve been using all week) to a list box. But according to Jimmy, this is something he gets asked about on a regular basis and there’s a AgDLR bug open for this. The __clrtype__ feature is specific to IronPython but I bet the IronRuby guys could implement something similar if they wanted to.

When you install IronPython 2.6 (or 2.0.1 for that matter), it comes with the AgDLR bits in the Silverlight subfolder. This includes Silverlight compatible versions of the DLR and IronPython as well as the Silverlight DLR host and the development web server Chiron in the Silverlight\bin directory. There is also a script in the Silverlight\script directory that will generate a dynamic Silverlight application from a template. I ran “sl.bat python sldemo” in order to build the skeleton project.

In the generated app.xaml file, I removed the default text box and replaced it with this XAML code that I stole nearly-verbatim from my blog post on data binding in WPF with IronPython. The only thing I changed was the binding path for the text block (title became name).

Then in the App class, I set the ItemsSource of the ListBox to a hand-built a list of Products.

And that’s pretty much it. I used Chiron’s /z command to create a Silverlight XAP file, uploaded it to Silverlight Streaming and embedded it right here in this post. Code is up on my skydrive as well. Uusing Silverlight Streaming for this app was very easy - basically upload the XAP file to their server and embed some iframe code in this post via the source view and that was it. I’m not sure I would use it for a production app, but it rocked for hosting this demo.

The XAP is a big download for such a trivial app - about 1.3MB. The vast majority of that is the DLR and IronPython assemblies. The XAP would only be 2.9kB if it was just the Python, XAML and manifest files. This kinda stinks, but there’s a new transparent platform extensions feature in Silverlight 3 so we can at least break the DLR and IronPython DLLs out into their own separate XAPs. That way they only get downloaded once and cached in the browser instead of being included in every single IronPython Silverlight application anyone creates.

So that’s one scenario down, one to go. In order to be able to build WCF services in IronPython, I have to add a lot more infrastructure – notably emitting CLR methods that can invoke dynamic methods as well as emitting custom attributes. Invoking dynamic methods means understanding DLR binders, so look for more posts on __clrtype__ next week.

__clrtype__ Metaclasses: Adding CLR Properties

When I was first experimenting with __clrtype__, I got to the point of making CLR fields work and then immediately tried to do some data binding with Silverlight. Didn’t work. Turns out Silverlight can only data bind against properties – fields aren’t supported. So now let’s add basic property support to ClrTypeMetaclass. Python has a rich mechanism for defining properties, but hooking that up requires DLR binders so for now I’m going to generate properties that are simple wrappers around the associated fields.

There’s enough code involved in defining a property to break it out into it’s own method:

You provide define_prop the TypeBuilder for the Type being constructed, the name and type of the property as well as the FieldBuilder that gets returned from the call to DefineField. In the previous installment, I wasn’t bothering to save the FieldBuilder to a variable since I never used it again. Now, I’m stashing it away for the call to define_prop as I’ll show below.

For each field, we define a get method, a set method and a property. The get function first executes ldarg_0 to load the current object reference onto the execution stack, then it executes ldfld to load the specified field from the object onto the stack, then it returns. The set function executes ldarg_0 to load the current object reference and ldarg_1 to load the value passed as the first argument onto the execution stack, then it executes stfld to store the value in the specified field of the object. Once I have the two methods, I call DefineProperty to create the PropertyBuilder and then associate the get and set methods with that property.

As I said before, Reflection.Emit is straightforward though tedious. Honestly, I didn’t go thru the Emit docs to figure out what the methods should look like. Instead, I wrote a basic wrapper property in C# and looked at the generated IL in Reflector.

The only other change here is adding the call to define_prop on our first iteration thru list of _clrfields. Since the rest of __clrtype__ is the same, here’s just that code snippet:

As I said above, I simply save off the result of calling DefineField so I can pass it to define_prop. I also save off the field type in a variable since I use it more than once. Avoids the second dictionary lookup and is clearer to understand what the function does.

Accessing the CLR properties via reflection is pretty straightforward – not very different than reflecting over CLR fields. The only significant difference between them is that CLR properties can be indexable and fields can’t, so you have to pass an index parameter to GetValue and SetValue. These aren’t indexed properties, so I pass in None for the index parameter.

One quick aside about the CLR type I’m generating here. I’m fairly certain this reflected object wouldn’t pass muster with the C# compiler. I’m defining a field and a property with the same name. It clearly works at the IL level, but I’m not sure what the C# compiler would do if you tried to refer to a CLR type like this. I should probably be prepending an underscore or something on the field name, but then I wonder if the field should also be private. There’s a whole API design discussion down that road, but I’m not quite ready to have that yet so I’m just leaving the fields public and having fields and properties with the same name. Luckily, I’m never generating a CLR type on disk so you can’t build a C# project that refers to it anyway.

__clrtype__ Metaclasses: Adding CLR Fields

Now that we have the basic __clrtype__ metaclass infrastructure in place, let’s enhance it to add support for CLR fields. To do this, we’re going to need to add two things to our custom CLR type. First, we need to define the fields themselves. Second, we need to make sure that Python code will read and writes to the statically typed fields for the specified names rather than the storing them in the object dictionary as usual. Here’s the updated version of ClrTypeMetaclass (or you can get it from my skydrive)

All the base type, type name, type builder and constructor code in the first half of the __clrtype__ method is the same as last time, so we’ll focus on the second half. After emitting the constructor(s), next we iterate thru a dictionary named _clrfields (if it exists in the class) that maps field names to types. For each of these dictionary entries, we emit a public field on the CLR type with the specified name and type.

The first time I tried this, I simply added the custom field generation code I just described and left it at that. Didn’t work. Python doesn’t look to store information in fields defined by the static type metadata unless explicitly instructed to. That’s why I need to iterate over the declared list of fields a second time after the type has been created. The first time creates the CLR fields, the second time inserts a ReflectedField instance into the class dictionary. ReflectedField is a Python descriptor that reads and writes the field value by calling GetValue and SetValue on the contained FieldInfo object. Python uses the same name resolution for fields as it does for method (In Python, methods are fields that store callable objects) so when IronPython discovers the ReflectedField descriptor in the class instance, it uses that to get or store the value rather than sticking it in the local dictionary.

Now here’s the new version of the Product class, this time with CLR fields as well as a custom type name:

As you can see, the only thing that’s changed is the addition of the _clrfields dictionary. But now, we can use reflection to get and set the Product fields, like so:

This is great progress, but not enough to get us to our first “real” scenario: data binding in Silverlight. Silverlight only supports data binding against public properties, so I’ll need to wrap all these CLR fields in CLR properties in my next post.

__clrtype__ Metaclasses: Customizing the Type Name

Now that we know a little about how IronPython uses CLR types under the hood, let’s start customizing those types. In a nutshell, __clrtype__ metaclasses are metaclasses that implement a function named __clrtype__ that takes the Python class definition as a parameter and returns a System.Type. IronPython will then use the returned Type  as the underlying CLR type whenever you create an instance of the Python class.

Technically, you could emit whatever custom CLR Type you want to in the __clrtype__, but typically you’ll want to emit a class that both implements whatever static CLR metadata you need as well as the dynamic binding infrastructure that IronPython expects. The easiest way to do this is to ask IronPython emit a type that handles all the dynamic typing and then inherit from that type to add the custom CLR metadata you want.

Let’s start simple and hello-worldly by just customizing the name of the generated CLR type that’s associated with the Python class. There’s a fair amount of boilerplate code that is needed even for this simple scenario, and I can build on that as we add features that actually do stuff. If you want to follow along at home, you’ll need IronPython 2.6 Alpha 1 (or later) and you can get this code from my SkyDrive.

Like all Python metaclasses, ClrTypeMetaclass inherits from the built-in Python type object. If I wanted to customize the Python class as well, I could implement __new__ on ClrTypeMetaclass , but I only care about customizing the CLR type so it only implements __clrtype__. If you want to know more about what you can do with Python metaclasses, check out Michael Foord’s Metaclasses in Five Minutes.

First off, I want to get IronPython to generate the base class that will implement all the typical Pythonic stuff like name resolution and dynamic method dispatch. To do that, I call __clrtype__ on the supertype of ClrTypeMetaclass – aka the built-in type object. That function returns the System.Type that IronPython would have used as the underlying CLR type for the Python class if we weren’t using __clrtype__ metaclasses.

Once I have the base class, next I figure out what the name of the generated CLR type will be. This is pretty simple, I just use the name of the Python class. To make this logic a little more interesting, I added support for a custom namespace. If the Python class has a _clrnamespace field, I append that as the custom namespace for the name. I should probably be using a double underscore – i.e. __clrnamespace – but I didn’t want to wrestle with name mangling in this prototype code.

Now that I have a name and a base class, I can generate the class I’m going to use. I’m using the DefineType method in Microsoft.Scripting.Generation.Snippets DLR class for three reasons. First, there’s a CLR bug that doesn’t let you create a dynamic assembly from a dynamic method. Second, reusing the snippets assembly avoids the overhead of generating a new assembly. Finally, the types in Snippets.Shared get saved to disk if you run with the -X:SaveAssemblies flag, so you can inspect custom CLR type that gets generated. The DefineType function takes four parameters, the type name, the base class, a preserve name flag and a generate debug symbols flag. If you pass false for preserve name, you get a name like foobar$1 instead of just foobar. As for debug symbols, since I don’t have any source code that I’m generating IL from, emitting debug symbols doesn’t make a lot of sense. DefineType returns a TypeGen, but I only need the TypeBuilder.

The last thing I need to do is implement the custom CLR type constructor(s). IronPython CLR types will always have at least one parameter – the PythonType (PythonType == IronPython’s implementation of Python’s built-in type object) that’s used for dynamic name resolution. I don’t want to add any custom functionality in my custom CLR type constructors, so I simply iterate thru the list of constructors on the base class and generate a constructor on the custom CLR type with a matching parameter list and that calls the base class constructor. 

Generating the IL to emit the constructor and the base class is straightforward, if tedious. I define the constructor with the same attributes, calling convention and parameters as the base class constructor. Then I emit IL to load the local instance (i.e. ldarg 0) and all the parameters onto the stack, call the base constructor and finally return. Once all the constructors are defined, I can create the type and return.

Using the ClrTypeMetaclass is very easy - simply specify the __metaclass__ field in a class. If you want to customize the namespace, specify the _clrnamespace field as well. Here’s an example:

You can verify this code has custom CLR metadata by calling GetType on a Product instance and inspecting the result via standard reflection techniques.

Great, so now I have a custom CLR type for my Python class. Unfortunately, at this point it’s pretty useless. Next, I’m going to add instance fields to the CLR type.

__clrtype__ Metaclasses: IronPython Classes Under the Hood

Before we start using __clrtype__ metaclasses, we need to understand a bit about how IronPython maps between CLR types and Python classes. IronPython doesn’t support Reflection based APIs or custom attributes today because IronPython doesn’t emit a custom CLR types for every Python class. Instead, it typically shares a single CLR type across many Python classes. For example, all three of these Python classes share a single underlying CLR type.

Even though cheese_shop inherits from shop and argument_clinic inherits from object, all three classes share the same underlying CLR type. On my machine, running IronPython 2.6 Alpha 1, that type is named “IronPython.NewTypes.System.Object_1$1”.

IronPython can share the CLR type across multiple Python classes because that CLR type has no code specific to a given Python class. CLR types are immutable – once you build a CLR type, you can’t do things like add new methods, remove existing method or change the inheritance hierarchy. But all those things are legal to do in Python. Here, I’m creating an instance of the cheese_shop class, but then changing that instance to be an argument_clinic instance instead.

When you call a method on a Python object, the name is resolved by walking a series of dictionaries. First, the dictionary of the object itself is searched for the method name. Assuming the name isn’t in the object dictionary, Python then looks in the __class__ dictionary. If it’s not there, Python recursively looks through the base classes stored in the __bases__ tuple until it finds the method or the name fails to resolve. If we re-assign __class__ at run time, we change the dictionary Python uses to resolve method names.

There are cases where IronPython generates a new underlying CLR type. For example, if you build a Python class that inherits from a CLR type, then IronPython will have to generate a new underlying CLR type that inherits from the CLR type in order to remain compatible. IronPython automatically overrides all the virtual methods of the base type, implementing the same dynamic method dispatch that I described above. This lets you pass the IronPython class wherever the base CLR type is expected.

The ability to swap Python classes at runtime depends on having the same underlying CLR type. If the underlying CLR type doesn’t match, then assigning a new value to the __class__ field of an object will fail. This applies both to IronPython classes that inherit from CLR types as well as __clrtype__ metaclass types. In the code I’ll be blogging, I always generate a unique CLR type for every Python class, which means that I can’t dynamically retype the object. Given that the point of __clrtype__ metaclasses is to generate static type information, this hardly seems like a limitation. However, it’s something to be aware of as we explore the __clrtypes__ feature.

Introducing __clrtype__ Metaclasses

Everyone knows Anders announced at PDC08 that C# 4.0 will include new features (aka the dynamic keyword + the DLR) that makes it much easier for C# to call into dynamically typed code. What you probably don’t know is that IronPython 2.6 includes a new feature that makes it easier for IronPython code to be called by statically typed code.

While the vast majority of .NET is available to IronPython, there are certain APIs that just don’t work with dynamic code. In particular, any code that uses Reflection over an object’s CLR type metadata won’t work with IronPython. For example, while WPF supports ICustomTypeDescriptor, Silverlight only supports data binding against reflectable properties. Furthermore, any code that uses custom attributes inherently uses Reflection. For example, Darrel Hawley recently blogged a WCF host he wrote in IronPython, but he wrote the WCF service in C#. You can’t write WCF services in IronPython because WCF expects service classes to be adorned with ServiceContract and OperationContract attributes (among many others). IronPython users want access to use these APIs. Support for custom attributes is one of the most common requests we get - it’s currently the 5th highest vote getter among open issues.

In IronPython 2.6, we’re adding the ability to customize the CLR type of Python classes. This means you can add custom attributes, emit properties, whatever you want. For those of you who’ve been dreaming of implementing WCF services or databinding in Silverlight purely in IronPython, then this is the feature for you.

In a nutshell, IronPython 2.6 extends Python’s metaclass feature that lets you to customize the creation of classes. In the metaclass, you can implement an IronPython-specific method __clrtype__ which returns a custom System.Type of your own creation that IronPython will then use as the underlying CLR type of the Python class. Implementing __clrtype__ gives you the chance to implement whatever reflectable metadata you need: constructors, fields, properties, methods, events, custom attributes, nested classes, whatever.

Over a series of posts, I’ll be demonstrating this new feature and implement some common scenario requests – including Silverlight databinding and WCF services – purely in Python. Quick warning: __clrtype__ uses low level features like Python metaclasses, Reflection.Emit and DLR Binders so these posts will be deeper technically than usual. Don’t worry – this isn’t the API interface we expect everyone to use. Eventually, we want to have an easy to use API that will sit on top of the low-level __clrtype__ hook.

Writing an IronPython Debugger: Breakpoint Management

Setting a breakpoint was the second feature I implemented in ipydbg. While setting a breakpoint on the first line of the Python file being run is convenient, it was obviously necessary to provide the user a mechanism to create their own breakpoints, as well as enable and disable existing breakpoints.

First thing I had to do was to refactor the create_breakpoint method. Originally, I was searching thru the symbol documents looking for the one that matched the filename in OnUpdateModuleSymbols. However, since I wanted to specify by new breakpoints via the same filename/line number combination, it made more sense to move symbol document logic into create_breakpoint:

The new version isn’t much different than the old. It loops thru the symbol documents looking for one that matches the filename argument. Then it creates the breakpoint the same way it did before. Eventually, I’m going to need a better algorithm than “only compare filenames”, but it works for now.

Once I made this change, it was trivial to implement a breakpoint add command. What was harder was deciding on the right user experience for this. I decided that breakpoint management was going to be the first multi-key command in ipydbg. so all the debug commands are prefixed with a “b”. I use the same command routing decorator I used for input commands. As you can see, my breakpoint command looks a lot like my top level input method – read a key from the console then dispatch it via a commands dictionary that gets populated by @inputcmd decorators.

Currently, there are four breakpoint commands: “a” for add, “l” for list, “e” for enable and “d” for disable. List is by far the simplest.

As you can see, I’m keeping a list of breakpoints in my IPyDebugProcess class. Originally, I used AppDomain.Breakpoints list, but that only returns enabled breakpoints so I was forced to store my own list. Note also that I’m using the enumerate function, which returns a tuple of the collection count and item. I do this so I can refer to breakpoints by number when enabling or disabling them:

Since the code was identical, except for the value passed to bp.Activate, I factored the code into a separate _set_bp_status method. After the user presses ‘b’ and then either ‘e’ or ‘d’, they then type the number of the breakpoint provided by the breakpoint list command. _set_bp_status then simply iterates thru the list until it finds the matching breakpoint and calls Activate. Note that since it’s possible to have 10 or more breakpoints, I’m using ReadLine instead of ReadKey, meaning you have to hit return after you type in the breakpoint number.

Finally, I need a way to create new breakpoints. With the refactoring of create_breakpoint, this is pretty straightforward

Most of _bp_add is processing the input arguments, looping through the modules and then storing the breakpoint that gets returned. When I set the initial breakpoint inside OnUpdateModuleSymbols, I have the module with updated symbols as an event argument. However, in the more general case we’ve got no way of knowing which module of the current app domain contains the filename in question. So we loop thru all the modules, calling create_breakpoint on each until one returns a non-null value. Of course, “all the modules” will include the IronPython implementation, but assuming you’re running against released bits the call to create_breakpoint will return right away if debug symbols aren’t available.

As usual, the latest version is up on GitHub. This will be the latest update to ipydbg for a little while. I worked on it quite a bit while I was at PyCon and have been busy with other things since I got home. Don’t worry, I’ll come back to it soon enough. As I mentioned Monday, I want to get function evaluation working so I can have a REPL console running in the target process instead of the one I’ve got currently running in the debugger process.

Pygments for WL Writer v1.0.1

I just replaced the original v1.0.0 Pygments for WL Writer installer with a new and improved v1.0.1. The original URL still works – I archived the old version off with a new name. Updated source is available on on GitHub.

The only change is that I now override OnSelectedContentChanged in the sidebar control. That way, if I have multiple blocks of pygmented code in a given post, the sidebar UI updates with the correct language and color scheme of the currently selected code block.

Writing an IronPython Debugger: REPL Console

While I was banging my head against a wall experimenting with understanding how CorValue extraction worked, I found myself wanting to dink around with the debugger objects in a REPL console. One of IronPython’s core strengths is support for “exploratory programming” via the REPL. It turned out bringing a REPL to ipydbg was quite simple.

Python includes two built-in features that making DIY REPL quite easy: compile and exec (though technically, exec is a statement, not a function). As you might assume from their names, compile converts a string into what Python calls a code object while exec executes a code object in a given scope. Technically, exec can accept a string so I could get by without using compile. However, if you’re compiling a single interactive statement compile can automatically insert a print statement if you’ve passed in a an expression. In other words, if you type in “2+2” on the console it will print “4”, which is the behavior I wanted.

Here’s what my REPL console code look like. I love that it’s only 20 lines of code.

It’s pretty straightforward. I set up a dictionary to act as the local variable scope for the code that gets executed. I’m just reusing the current global scope, but I want the local scope to start with only the reference to the current IPyDebugProcess instance which is passed into _input_repl_cmd as “self”. All the other local variables like cmd and line won’t be available to the REPL code. Then I drop into a loop where I read lines from the console and execute them.

In order to support multi-line statements, I build up the cmd variable over multiple line inputs and I don’t execute it until the user inputs an empty line. In the standard Python console, it can recognize single line statements and execute them immediately. Dino showed me how to use the IronPython parser to do the same thing, but I haven’t implemented that in ipydbg yet. To exit the REPL loop, you type Ctl-Z, which returns None (aka null) from ReadLine instead of the empty string.

Since I never execute the code more than once, I have my exec and compile statements together on a single line. Compile takes the string to be compiled, the name of the file it came from (I’m using <input> for this) and the kind of code. Passing in “single” for the kind of code adds the auto-expression-print functionality I mentioned above. Then I exec the code object that’s returned in specified scope I’m managing for this instance of the REPL loop. If you exit out of the REPL and re-enter it, you get a fresh new copy of the local scope so any functions or variables you define in the last REPL are gone.

Runtime execution of code into a given scope is a hallmark of dynamic languages, but I’m still fairly green when it comes to Python so it took me a while to figure this out. Python code executes in a given scope, a combination of global and local variables. When you’re in the ipy.exe REPL, you’re at top level scope anyway, so global and local scope are the same – if you add something to global scope, it shows up in local scope and vis-versa. Inside a function, you’ll have the same global scope, but the local scope will be different and changes to one won’t be reflected in the other. The ipydbg REPL isn’t a function per-se, but it does provide an explicit local scope that gets disposed when you exit the REPL.

While having a debugger REPL is really convenient for prototyping new ipydbg commands, it’ll really shine once I get function evaluation working. Then I’ll be able to open a REPL console where the commands are executed in the target process instead of the debugger process as they are now. That will be very cool. Until then, the latest code is – as always – up on GitHub.

Writing an IronPython Debugger: Getting Arguments

It’s a small update, but I added support for displaying method arguments along side the local variables. As I mentioned in that post, breaking out the CorValue extraction and display code into a shared function was a good idea – adding support for getting arguments was trivial since I could reuse that code.

Because there’s no hierarchy of scopes to deal with and the names are in the metadata instead of debug symbols, getting arguments is much easier than getting local variables.

You’ll notice that I’m yielding the arguments as a tuple of the name and value, the same as get_locals yields. I did refactor get_locals a bit – there’s no longer an argument to skip hidden variables anymore (though get_locals still skips dynamic call sites caches as it did before). Now, it’s up to the the caller of get_arguments and get_locals to filter hidden variables as they see fit.

Because get_locals and get_arguments yield the same types, I was able to factor the code to print a value and loop through the collection of values into separate local functions.

I really like the local functions feature of Python. In C#, you can define an anonymous delegate using the lambda syntax. But for a scenario like this, I like local functions better. However, I do like C#’s support for statement lambdas – Python only supports expression lambdas. So while I like local functions better in this scenario (because I’m using the method more than once) in something like an event handler, I like the statement lambda syntax better.

As usual, the latest version of ipydbg is up on GitHub.

Pygments for Windows Live Writer

For the past few years, I’ve used the CodeHTMLer plugin for Windows Live Writer for the code snippets in my blog. However, recently I discovered the Pygments Python syntax highlighter package which supports scores more languages than CodeHTMLer does. It also support multiple color schemes and was easily extensible so I could build an HTML formatter that didn’t use <pre> tags (which I’ve found DasBlog has issues with in the RSS feed, though honestly I’m running three minor releases behind the latest DasBlog release). IronPython supports Pygments just fine – at least, the one IPy bug that Pygments exposes has a simple workaround – so I set about building a Windows Live Writer plugin that uses it.

If you’re simply interested in the plugin itself, you can get it from my skydrive. The source is up on GitHub. For now, if you find any bugs, please leave a comment on this post. If there’s enough interest I’ll setup a site somewhere (CodePlex perhaps) where I can track bugs and feature requests.

Pygments for WL Writer is a smart content source. In WL Writer’s terminology, that means when you click inserted text in the editor window, it is treated as an atomic entity which you can then edit by using the Edit Code button in the Pygments for WL Writer sidebar editor. I I often found that I would edit my code multiple times – usually to shorten lines so they’d fit on my blog without wrapping. CodeHTMLer for WL Writer is a standard content source, so it just spews the formatted code as HTML onto the page.

From an IronPython perspective, there’s some interesting stuff there. I decided to compile the pygments library into a DLL for easier distribution. If you look in the source, there’s a folder for the Pygments source as well as the parts of the standard Python library that Pygments depends on and my custom HTML formatter. Those all get compiled via a custom script which can be called by the build.bat file in the project root.

Some features I’m thinking about adding:

• An extensibility model so that you can add new languages by dropping new Pygments lexers into the same folder the plugin is installed to. Pygments supports lots of languages, but not all of them – notably it’s missing Powershell and F#.
• Support for new HTML formatters and color schemes using the same extensibility mechanism described above.
• Support for selecting an HTML formatter.
• Improving the code editor window. Currently, I’m using a standard WinForms multi-line TextBox, but that leaves a lot to be desired. With the Python work I do, I often need to be able to select a bunch of text and change it’s indenting via tab and shift-tab. If anyone has a suggestion for a good WinForms text editing control, let me know.
• Being able to specify the font and size of the Pygmented code.
• Storing user preferences – remembering the most recent syntax and color scheme the user used.

Feedback, as always is appreciated. I’ll probably write a few posts about the project when I get a chance, so let me know if there’s anything you’re dying to hear about.

Writing an IronPython Debugger: Command Routing

At this point, ipydbg support seven commands: Continue, Quit, Show Stack Trace, Show Locals, Step Over, Step In, and Step Out. All these commands are invoked by a single keystroke. I’m using Console.ReadKey in an attempt to cut down on the number of keystrokes needed for interacting with the debugger. If I only type ‘s’ instead of ‘s <enter>’ to step, I figure I’ll be twice as productive! :)

If I was writing ipydbg in C#, I could use switch statement to dispatch commands in the _input method based on user keystrokes. However, Python doesn’t have a switch statement so I’ve been using a cascading set of if/elif/else statements instead. When you get up to seven if/elif clauses plus an else clause, the code smell is pretty overwhelming.

Python might not have a switch statement, but it does have first-order functions so you can get the effects of a switch by using a dictionary.

I like this approach much better. Individual if/elif blocks are now broken out into separate functions, which smells better than embedding them in one big function. Also, I like that my pseduo-switch statement is completely separate from the how the _switch dictionary is initialized. However, this approach also separates the pseudo-case statement functions from the _switch dictionary as well. That’s not a good thing. You can easily imagine screwing up by adding a new function but forgetting to manually update the _switch dictionary.

What I need is a way to declaratively associate the switch function with the dictionary lookup key that’s associated with it. Luckily, Python Decorators provides a very clean way to do this.

I’ve blogged about decorators before when I wanted to automatically invoke operations on the right thread in my WPF photo viewing app. The @inputcmd decorator is a bit more complicated than the @BGThread and @UIThread decorators since @inputcmd decorator accepts arguments. Each of the @input command decorators in the code above is the equivalent to this code:

As you can see, the inputcmd function returns the decorator that wraps do_a, rather than being the decorator itself. This function that returns a function that returns a function is kinda confusing at first. But this approach allows you to configure the decorator for a specific purpose via the arguments – in this case, specifying which dictionary and which console key this function is associated with.

Also unlike @BGThread and @UIThread, I don’t actually want to modify the behavior of the methods decorated with @inputcmd. I only want to store a reference to them in the passed in dictionary. So implementing this decorator is very easy:

The decorator simply inserts the function into the passed-in dictionary using the passed in key. It then returns the function as is, so it’s not really rebinding the symbol to a new method (technically, it’s rebinding the symbol to the same function it’s currently bound to). If I wanted also wrap the passed in function to provide additional functionality, I could do that with a second locally defined function inside the deco function.

The latest version of ipydbg as been refactored to use @inputcmd instead of set of a cascading if/elif statement blocks. Now that that’s done, I can start working on multi-key commands.

DevHawk on CodeCast

Ken Levy used to work around the corner from my office, back in his days on the VSX team. These days, he’s hosting the CodeCast (among other things) and he dropped my my office a while back to chat about IronPython for his podcast.

Check it out.

Writing an IronPython Debugger: Displaying Values

Now that I can get the local variables for a given frame, I need to display them in the console. Eventually, I’d like to provide the ability to update the local variables as well, but you gotta crawl before you can run. Luckily, the debugger API is consistent about using same COM interfaces – wrapped by the managed CorValue class – to represent all data values, including local variables, function arguments and object fields. So the work I do now to display CorValues in the console will be reusable in other contexts down the road.

While the debugger API is consistent about how it represents values in the target process, the API it uses is very complicated. The primary COM interface for accessing values is ICorDebugValue, but it has eight siblings: ICorDebugReferenceValue, ICorDebugHandleValue, ICorDebugStringValue, ICorDebugObjectValue, ICorDebugGenericValue, ICorDebugBoxValue, ICorDebugArrayValue, ICorDebugHeapValue. All those COM interfaces are represented in managed code by CorValue and it’s subclasses.

Furthermore, confusingly ICorDebugValues have both a Type and an ExactType. ExactType is what .NET developers typically think of as the type, aka the CLR type. Well, the debugger API’s representation of the CLR type at any rate. You can retrieve the value’s metadata as a System.Type compatible object via value.ExactType.Class.GetTypeInfo().CorValue’s Type property, on the other hand, represents the object’s primitive or element type. For example, instances of .NET classes have an element Type of ELEMENT_TYPE_CLASS. There are a collection of primitive types (boolean, char, ints of various signage and size, floats of various size) as well as types you wouldn’t call primitive but that the runtime has specific knowledge of (string, array and value types - aka structs in C# terminology).

If you’re confused by all that, don’t worry so am I. Honestly, I’ve re-written this code several times, each time understanding the API just a bit better. Whatever the *right* way to use the interfaces, I’m sure I don’t know it. For my first cut at this, I essentially ported MDbg’s high level CorValue API – aka MDbgValue::InternalGetValue if you’re looking at the MDbg source code – over to Python. Along the way, I’ve improved on that code as I’ll describe below.

A given CorValue may be a primitive value like an int or it may be a reference to or a boxed version of some other CorValue object. So in order to print the CorValue, you have to go thru a series of attempts to dereference and unbox until you get to the “real” underlying CorValue object. From there, converting the value to a string I can print depends on the value’s element type. For primitive types like ints and floats, you can call CastToGenericValue to get a CorGenericValue “view” of the same CorValue object [1]. A CorGenericValue can read and write the raw bytes from memory in the target process of the value. The GetValue method reads the data from target process then does an unsafe cast to appropriate managed type. For example, an ELEMENT_TYPE_R4 CorValue gets cast into a System.Single. For CorValue strings, I call CastToStringValue and then access the String property. For classes, value types and objects, there’s no simple or standard approach to retrieving the data, so for now I return the result of calling CastToObjectValue. Eventually, I’ll want to provide a mechanism to read the specific fields of a class or value type.

Unfortunately, the mechanism above to read primitive types doesn’t work with IronPython. GetValue needs to know the correct element type in order to do the unsafe cast. For value types (aka any struct other than the basic primitives), GetValue will return a data as a byte array. The problem is that when you box a primitive, the original element types gets overwritten by ELEMENT_TYPE_VALUETYPE. You can’t get the original element type back, even after unboxing. So for boxed primitives, you can only retrieve the data as a raw byte array or as a CorObjectValue, neither of which is very useful.

Luckily, I was able to work around this. Under the hood, GetValue calls UnsafeGetValueAsType to do the actual work of reading the data from the target process and casting it to the right managed type. UnsafeGetValueAsType It accepts the an element type value as a method parameter. If your know the right element type value, you could call UnsafeGetValueAsType directly if instead of going thru GetValue. While boxing overwrites the original element type value, an unboxed CorValue still has the CLR type metadata available. So I was able to map CLR Types to element types (e.g. System.Single –> ELEMENT_TYPE_R4) in order to retrieve the underlying value of boxed primitive types.

It’s kinda ugly code and I’m thinking that at least some of really belongs in the CorValue C# classes rather than in ipydbg. However, I’m not that interested in doing the significant refactoring it would take to make the CorValue API developer-friendly, so I did it here.

One thing to note that I didn’t cover earlier is the NullCorValue object. For reference values, there’s a IsNull property that may be set. If it is set, I need a mechanism to indicate the null value, but also includes the type information. So I created a custom type that can store the type name to represent null. Again, something that should be a part of the CorValue API.

Once I have my extracted value, I need to display it in the console. This is much simpler than the extracting the value. As I wrote above, I’m not making any attempt to print a real representation for CorObjectValues. I could look at making a call ToString call to get something useful, but that requires invoking a function in the target process and I haven’t gotten that far with ipydbg yet. So I just print “<…>” if it isn’t a string, primitive or null value.

Now all I need is to iterate thru the list of local variables and call extract_value and display_value on each in turn and print the results. I won’t reproduce that code here, but you can see it in the ipydbg project source on GitHub.

I’m happy with what I’ve gotten working (it took several days of banging my head against the proverbial wall to get it this far) but there’s still room for improvement. First, I’d like to be able to call ToString to get a class-specific generic representation as I described above. Second, I need a way to display the fields of a CorObejctValue object. It’s just a combination of metadata reading and CorObjectValue::GetFieldValue, but that code won’t write itself. Finally, there are other Python primitives - like list, dictionary and tuple – that ipydbg should have specific knowledge of and be able to display without requiring the user to drill into the member variables and the like.

[1] While the CorValue API does certain things very well, I wish it did a better job abstracting away the existence of the various ICorDebugValue interfaces. Hence the need for all the calls to CastToWhatever().

IronPython 2.6 Alpha 1

Just in type for PyCon, we just shipped the first alpha of IronPython 2.6. As you can guess from the version number, the main feature of this version of IronPython will be the new features introduced in Python 2.6. As you can see, we’ve synced version numbers between IronPython and Python. No more explaining which version of IPy goes with which version of Python.

In addition to the start of 2.6 support, the other big feature of IronPython 2.6 is something called Adaptive Compilation. IronPython’s performance is pretty good compared to CPython. We’re about 28% faster than CPython (IPy 2.0.1 vs. CPy 2.5) on PyStone and about 10% faster on PyBench if you exclude the TryRaiseExcept test. [1] However, our startup time is not very good. These two facts are related: it takes a long time on startup to compile to Python code to IL (and then JITted from IL to native code), but once that’s done the code runs really fast. However, if you’re only going to execute a function a few times, it typically isn’t worth the overhead to compile the function to IL. The Adaptive Compilation feature is an interpreter for DLR trees. The first few times you run a given Python function, it gets interpreted. At some point, after you’ve called the function enough times, IronPython 2.6 decides to take the hit and compile the function. If you want to go back to the old “always compile to IL” model, you can pass –O on the command line.

This is our first alpha of 2.6, and some things are kinda broken. In particular, there was a change to collections.py that breaks much of the Python Standard Library under IronPython. Dave has the details and the workaround. Rest assured, this will get fixed before we release. Dino is hard at work making _getframe work for depths greater than zero. Because it will have some perf impact, it won’t be enabled by default – you’ll have to pass a command-line parameter to enable it. But if you have to opt-in to _getframe support for depth > 0, it makes sense to opt-into _getframe support entirely and do away with the current _getframe(0) only support. What’s nice about this approach is that it will work with collections.py regardless if you opt-in to _getframe or not.

As stated in the release notes, the release cycle on 2.6 will be much shorter than 2.0. There was only seven months between 1.0 and 1.1, and we’re shooting for a slightly longer timeframe for 2.6. Certainly not like the twenty months that passed between 1.1 and 2.0. So please start trying it out as soon as you can and give us your feedback.

[1] IPy is over 4000% slower than CPy on TryRaiseExcept, 58,234 ms vs. 1,286ms. This one test represents 44% of our overall test run time and causes IPy to run PyBench 57% slower than CPy instead of 10% faster. Python has a different philosophy on exceptions than CLR does. Several Python exceptions like GeneratorExit and StopIteration are explicitly documented as “not considered an error”. This is a very different approach to CLR’s approach. At some point, we’re going to have to look at improving exception performance, but it’s not really a priority for the 2.6 release.

Writing an IronPython Debugger: Getting Local Variables

I just pushed out a new drop of ipydbg that includes the first cut of support for showing local variables. Getting the value for a local variable is actually pretty simple. The CorFrame object (which hangs off active_thread) includes a method to get a local variable by index as well getting a count of all local variables. The problem with these functions is that they don’t provide the name of the variable. For that, you’ve got to look in debug symbols.

From a CorFrame, you can retrieve the associated CorFunction. Since I added symbol reader support to CorModule, I added support for directly retrieving the ISymbolMethod for a CorFunction. From the method symbols, I can get the root lexical scope of the method. And from the symbol scope, I can get the locals. Scopes can be nested, so to get all the locals for a given function, you need to iterate thru all the child scopes as well.

So here’s my get_locals function:

The function is designed to automatically retrieve the scope and offset, if they’re available. That way, I can simply call get_locals with the frame argument and it does the right thing. For example, if you don’t pass in a symbol scope explicitly get_locals will attempt to retrieve the debug symbols. If debug symbols aren’t available, iterates over the locals in the frame and yields each with a fake name (local_0, local_1, etc). If the debug symbols are available, then it iterates over the locals in the scope, then calls itself for each of the child scopes (skipping child scopes who’s offset range doesn’t overlap with the current offset).

The other feature of get_locals is deciding which locals to include. As you might expect, IronPython emits some local variables that are for internal runtime use. These variables get prefixed with a dollar sign. The dollar sign is not a legal identifier character in C# or Python, but IL has no problem with it. If you pass in False for show_hidden (or use the default value), then get_locals skips over any local variables who’s name starts with the dollar sign.

Even if you pass in True for show_hidden, get_locals still skips over any variable named “$site”. $site variables are dynamic call site caches, a DLR feature that are used to efficiently dispatch dynamic calls by caching the results of previous invocations. Martin Maly’s blog has more details on these caches. As they are part of method dispatch, I never want to show them to the ipydbg user, so they get skipped regardless of the value of show_hidden.

Now that I can get the local variables for a given frame, we need to convert those variables to something you can print on the screen. That turns out to be more complicated that you might expect, so it’ll have to wait for the next post (which may be a while, given that PyCon is this weekend). In the meantime, you can get the latest version of ipydbg from GitHub.

Writing an IronPython Debugger: A Little Hack…err…Cleanup

Yesterday, I pushed out two commits to ipydbg. The first was simple, I removed all of the embedded ConsoleColorMgr code in favor of the separate consolecolor.py module I blogged about Thursday. The second commit…well, let’s just say it’s not quite so simple.

Last weekend, I was experimenting with breakpoints when I discovered that the MoveNext method of BreakpointEnumerator was throwing a NotImplementedException. Up to that point, I hadn’t modified any of the MDbg C# source code except to merge the corapi and raw assemblies into a single assembly. But since I had to fix BreakpointEnumerator, I figured I should make some improvements to the C# code as well. For example, I added helper functions to easily retrieve the metadata for a class or function.

In my latest commit, I’ve added a SymbolReader property to CorModule. Previously, I managed the mapping from CorModules to SymbolReaders in my IPyDebugProcess class via the symbol_readers field. However, since mapping CorModules to SymbolReaders is something pretty much any debugger app would have to do, it made more sense to have that be a part of CorModule directly. So now, you can set and retrieve the SymbolReader directly on the module. Furthermore, I moved the logic to retrieve a SymbolReader from the IStream provided in the OnUpdateModuleSymbols event into the CorModule class as well.

I wouldn’t have bothered to blog this change at all, except that if you look at how the SymbolReader property is implemented under the hood, it’s not what you would expect. Instead of having SymbolReader as an instance variable on CorModule – as you might expect -CorModule has a static dictionary mapping CorModules to SymbolReaders. The instance SymbolReader property simply then access to the underlying static dictionary.

Now obviously, this the way you typically implement properties. However, the problem is that there isn’t a 1-to-1 mapping between the underlying debugger COM object instances and the managed objects instances that wrap them. For example, if you look at the CorClass:Module property, it constructs a new managed wrapper for the COM interface it gets back from ICorDebugClass.GetModule. That means that I can’t store the symbol reader as an instance field in the managed wrapper since I probably will never see a given managed wrapper module instance ever again.

All of the debugger API wrapper classes including CorModule inherit from a class named WrapperBase which overrides Equals and GetHashCode. The overridden implementations defer to the wrapped COM interface, which means that two separate managed wrapper instances of the same COM interface will have the same hash code and will evaluate as equal. The upshot is that object uniqueness is determined by the wrapped COM object rather that the managed object instance itself.

Using a static dictionary to store a module instance property provides the necessary “it doesn’t matter what managed object instance you use as long as they all wrap the same COM object underneath” semantics. If I create multiple instances CorModule that all wrap the same underlying COM interface pointer, they’ll all share the same SymbolReader instance from the dictionary.

Yeah, it’s feels kinda hacky, but it works.

IronPython ConsoleColorMgr

I really liked the ConsoleColorMgr class from my last ipydbg post so I took a few minutes to yank it out into its own seperate module. I also took the opportunity to make a few improvements.

First off, I added support for background colors as well as foreground colors. Furthermore, both colors default to “None” which ConsoleColorMgr takes to mean leave that color unchanged.

The other change I made was to build a set of default ConsoleColorMgr instances in the consolecolor module, one for each of the values in ConsoleColor.

Note that for this set of default ConsoleColorMgr instances, I’m only setting the foreground color. If you want to set the background color, you have to create your own ConsoleColorMgr instances. This allows me to write the following:

If you want it, I’ve put consolecolor.py up on my skydrive or it’s available as part of my devhawk_ipy project on GitHub.

Update - Christopher Bermingham pointed out that my sample snippet at the end doesn’t work unless you add “from __future__ import with_statement” to the top of your python file. I updated my code snippet to include this. Thanks Christopher!

Writing an IronPython Debugger: Colorful Console

Now that I’ve added the current source code line to the console output, I wanted to start using color in order to make it clearer to understand the various pieces of data that gets output. Now, the various event handler messages get output in dark grey while the current line of source is in yellow. Here’s what it looks like on my machine (note, the top line with the green [11] is PowerShell and ipy2 is a PowerShell alias to ipy.exe v2.0.1)

Writing color to the windows console is a hassle because of the stateful API it uses. The problem is that I always want to return to the default color after I’ve written out a line of colored text. I wish there was an overload of Console.Write and WriteLine that took the foreground and background colors as arguments. 

Of course, I could easily implement my own write and writeline methods that took color parameters. However, I was loath to do that as Python’s print statement is so convenient. So instead, I build a console color context manager. I got the idea from Luis Fallas’ XmlWriter context manager.

Python’s with statement is similar to C#’s using statement. However, unlike IDisposable object, Python context managers support both an enter and exit method. This means I don’t have to construct an object in order to get a context (in this case, the console colors) managed. So far, I’ve got three console color context managers defined – Grey, DarkGrey and Yellow. I’m thinking that ConsoleColorMgr is a candidate for my assorted module collection at some point.

Now that I can print in color, I wanted to modify my line printer to use color. Usually, the current sequence point corresponds to an entire line of python source. But as we see below, sometimes only part of a given line of source text is associated with a given sequence point.

The other issue I ran into is that there’s a always a sequence point at the very end of a function. Unlike the break at the start of the function I wrote about in my last post, this one I didn’t want to automatically step over. This is the last breakpoint for a given scope, so I should give the user one last chance to inspect the scope (once I add the ability to do that, at any rate) before we step out of it. However, I wanted a way of showing that we’re about to step out in the source code line view. I decided on writing a series of carets ^^^ to indicate that we’re at the end of a function.

As you can see in the dark grey line in the screenshot above, the current sequence point starts and ends at line 4 column 23. Column 23 is beyond the end of line 4, so that’s what I look for in order to draw the three carets. Here’s the final version of _print_source_line:

So colorizing the current line of source code turned out to be a little harder than I had expected. But hey, I got a start of a reusable module out of it. That’s pretty cool. Anyway, the latest bits are, as always, up on GitHub.

Writing an IronPython Debugger: Showing Source Code

It’s been almost a week since my last ipydbg post. I’m not done, I just needed to catch my breath for a few days and get some other work done. Contrary to popular believe, my day job revolves around more than just ipydbg! :)

Actually, I’ve made ten commit since my last post, but it’s been a mostly minor changes. For example, I was hacking around with breakpoints and restored a bunch of commented out code in BreakpointEnumerator. Since I was changing the original C# CorDebug wrapper source, I decided to add a few helper functions to return metadata for functions and classes as well as cleaning up some C# filenames. On the Python side, I added an active_appdomain field to IPyDebugProcess to go along with active_thread.

Today, I added what started as a fairly minor feature – showing the current line of source code at the start of the input loop. The initial code for this was cake, simply getting the sequence point for the current location and mapping that to a source file. In order to avoid hitting the file system over and over, I cache source files the first time they are accessed.

However, when I did this, I discovered a slight issue. When you step into a Python function, the CLR debugger breaks at the very beginning of the function being stepped into. In C#, the function start is mapped to the opening curly brace of the function. IronPython, on the other hand, doesn’t map the start of the function to anything since there’s a bunch of infrastructure code at the start of every function that has no correlation to the python source. This means _get_location return a null sequence point when I first step into a function and thus I wouldn’t be able to show any source code.

I could make the argument that start of the function should be mapped to the colon that starts the function block. However, I’m not in a position to make changes to how the shipping version of IronPython emits debug symbols. So instead, I decided to insert an automatic step whenever I step into a function by modifying OnStepComplete:

I have this nagging feeling that a simple step won’t suffice and I’ll need to add logic to ensure that I’m only auto-stepping when the start of the function doesn’t have a matching sequence point. But I have tested this with a few different python scripts and it appears to work fine. If I need something more sophisticated, I can always add it later. BTW, notice I modified the signature of _do_step so that it takes the thread as an argument rather than picking it up as an IPyDebugProcess field.

As usual, latest ipydbg (including new compiled version of CorDebug.dll) is available at GitHub.

Writing an IronPython Debugger: Debugging Just My Code

As I wrote last time, in order to make debug stepping actually useful in ipydbg I need to avoid stepping into frames that are part of the IronPython infrastructure. I did something similar when I hide infrastructure frames in the stack trace. Originally, I had planned to automatically stepping again if we ended up on a frame that didn’t correspond to a python file. However, Mike Stall showed me a much cleaner and better performing solution: Just My Code. As I mentioned at the start of this series, support for JMC is one of the main reasons I wanted to build my own debugger rather than use MDbg.

Enabling JMC in the stepper object is trivial:

If I make that single change and run ipydbg, any step effectively turns into a full continue since none of the code has been marked as “My Code” yet. As you see, the tricky part of JMC isn’t enabling it on the stepper, it’s “painting” the parts of the code where you want JMC stepping to work. You can set JMC status at the module, class or the method level. In the case of ipdbg, it’s easiest to work at the class level:

OnClassLoad is where the action is. This event handler is responsible for enabling JMC for all class methods that map to python code. To understand how the logic in OnClassLoad works, you need to understand a little about the .NET types and code that IronPython generates. Note, the following description is for the IronPython 2.0 branch. Code generation evolves from release to release and I know for a fact there are changes in the upcoming 2.6 version. I assume that I’ll eventually have to sniff the IronPython version in order to set JMC correctly.

Today, IronPython generates all code into dynamic modules and methods. Since I want to limit stepping to python code only, I automatically disable JMC for non-dynamic modules. I can imagine a scenario where I want to step into non-dynamically generated code, but I think the best way to handle that would be to disable JMC at the stepper rather than widening the amount of code marked as JMC enabled.

For every module that gets loaded, IronPython generates a type. At a minimum you’re going to load two modules: site.py and whatever python script you ran. If you have the python standard library installed, site.py loads a bunch of other modules as well. Each of these module types have a bunch of standard methods that always get generated. For example, the global scope code in the module is placed in a static method on the module type called Initialize. Any python functions you define get generated static methods with mangled names on the module type [1]. All these methods have corresponding python code and should be JMC enabled. The other standard methods on a module type should not be JMC enabled. So in my debugger, I mark the class as JMC enabled but then iterate over the list of methods and mark any in the list of standard methods (except for Initialize) as JMC disabled.

Of course, you can also create classes in Python. As you might expect, classes in Python are generated as .NET types. However, the semantics of Python classes are very different than .NET types. For example, you can change the inheritance hierarchy of python classes at runtime. That’s obviously not allowed for .NET types. So the .NET types we generate have all the logic to implement Python class semantics. As it turns out, these .NET types *only* have the logic to implement Python class semantics, which is to say they have *none* of Python class methods code. This makes sense when you think about it – since Python can add and remove methods from a class at runtime, IronPython can’t put the method code in the .NET type itself. Instead, Python class methods are generated as static methods on the module type, just like top-level functions are. Since Python class types only contain Python class semantics logic, we never want to enable JMC for Python class types. Python class types get generated in the IronPython.NewTypes namespace, so it’s fairly easy to check the class name in OnClassLoad and automatically disable JMC for classes any in that namespace.

Adding JMC support makes ipydbg significantly more usable. It’s almost like a real tool now, isn’t it? Latest bits are up on GitHub.

[1] FYI, IronPython generates python functions as dynamic methods in release mode and static module class methods in debug mode since you can’t step into dynamic methods. The description above is specific to debug mode since ipydbg exclusively runs in debug mode.

Writing an IronPython Debugger: Stepping Thru Code

So far, I’ve written seven posts about my IronPython debugger, but frankly it isn’t very functional yet. It runs, breaks on the first line and can show a stack trace. Not exactly Jolt award material. In this post, I’m going to add one of the core functions of any debugger: stepping. Where previously I’ve written a bunch of code but had little to show in terms of features, now I’m getting three new features (basic step, step in and step out) at once!

Here you can see the _input clauses for step, step in and step out. Of the three, step out is the simplest to implement: create the stepper object and call StepOut. For step and step in, I could simply call Step (the boolean argument indicates if you want to step into or over functions) but that only steps a single IL statement. The vast majority of the time there are multiple IL instructions for every line of source code, so IL statement stepping is very tedious. As we learned when setting a breakpoint, debug symbols contain sequence points that map between source and IL locations. If they’re available, I use the sequence points to determine the range of IL statements to step over so that I can step single source statements instead.

The stepping code above depends on three helper functions defined at global scope.

The first function, create_stepper, simply constructs and configures the stepper object. The call to SetUnmappedStopMask tells the debugger not to stop if it encounters code that can’t be mapped to IL. If you need to debug at that level, ipydbg is *not* for you.

Next is create_step_range, which exists purely for .NET interop purposes. There are three interop warts hidden in this function. First is creating a .NET array of COR_DEBUG_STEP_RANGE structs. Every time I write Array code like this, I wish for a CreateFromCollection static method on Array. However, in this case it isn’t that big a deal since it’s a one element array. Second wart is having to set the values of COR_DEBUG_STEP_RANGE via constructor keyword arguments. It turns out that IronPython disallows direct updates to value type fields (read this for the reason why). Instead, I pass in the field values into the constructor as keyword arguments. Finally, you have to explicitly convert the start and end offsets to a unsigned int in order to set the offset fields in the COR_DEBUG_STEP_RANGE struct constructor.

Finally is get_step_ranges, which iterates thru the list of sequence points in the current method looking for the one with the smallest offset that is larger than the current offset position. If it can’t find a matching sequence point, it sets the range to the end of the current function. The start range offset is always the current offset. I did make a significant change to get_sequence_points – it no longer yields sequence points that have a start line of 0xfeefee. By convention, that indicates a sequence point to be skipped. Originally, the logic to ignore 0xfeefee sequence points was in get_location. But when I originally wrote get_step_ranges, it had essentially the same sequence point skipping logic, so I moved it to get_location instead.

Technically, I’ve built three new features but the reality is that if you end up in IronPython infrastructure code it’s really hard to find your way back to python code. Step in is particularly useless right now. Luckily, the .NET debugger API supports a feature called “Just My Code” that will make stepping much more useful. In the meantime, the latest version of ipydbg is up on GitHub as usual.

Writing an IronPython Debugger: Refactoring

When we last left ipydbg, it was up to about 200 lines of code. Not bad in terms of overall length, but I started to detect some code smell. I was relying pretty heavily on global variables and the structure of my code made it difficult to control how the debugger was run. I wanted to change ipydbg so it would automatically spin up an MTA thread if I forgot to add the –X:MTA command line parameter. But since by debugger and process objects were global, they’d get created on the main thread of ipydbg, regardless if it was STA or MTA. So for this “release” (I’d say I’m almost to version 0.0.0.1), I decided on focusing on enginering and refactoring rather than new features.

The big new addition is the IPyDebugProcess class, which is clearly the workhorse of the application. All of the previously global variables are now class instance variables on IPyDebugProcess. Input and run along with all the event handlers as well as do_break_event and get_location are now class methods, as they need to access instance variables (setting the break event, accessing the symbol reader dictionary, etc.). Functions that didn’t need to access instance variables (get_sequence_points, create_breakpoint, get_dynamic_frames and get_method_info_for_frame) I left as top-level functions. If they get more complex, I may break them out into their own modules, but for now I left them in ipydbg.py.

The conversion process was fairly trivial. I had to add “self.” lots of places and change the indention level all over but that was pretty much it. Once I finished the conversion, I was able to add the run_debugger function to handle the thread creation, if necessary.

Originally, I tried to put this logic in IPyDebugProcess.run. However, since I’m creating the debugger object in the __init__ function, that meant it would be created on the wrong thread. I could have moved the debugger creation to the run method or move the thread management code to __init__, but I decided to factor that logic into a separate function completely. Felt cleaner that way.

IronPython at PyCon

Here’s a quick quiz. Which of these tasks is harder to accomplish:

1. Getting $6,000 from a variety of groups within Microsoft to pay for a Gold PyCon 2009 sponsorship.
2. Sending PSF a check

If you guessed #2, you’d be right. It’s amazing how difficult the seemly trivial task of “give those PSF folks money” turned out to be. But it’s done now, and you can see the MS logo there on the side of all the PyCon pages.

In addition to the sponsorship, there are some great looking IronPython sessions at PyCon.

• Michael Foord and Jonathan Hartley are running a Developing with IronPython tutorial. Michael is also delivering a talk Functional Testing of Desktop Applications.
• Jim Hugunin is delivering an invited talk IronPython: Directions, Data and Demos
• Dino Viehland is sitting on the Python VMs panel and doing a talk IronPython Implementation
• Sarah Dutkiewicz (aka Coding Geekette) is doing a talk Pumping Iron into Python: Intro to FePy

devhawk_ipy

As I write various python modules (many of which get blogged about), I dump them into a special folder on my machine(s). In my powershell profile script, I set the IRONPYTHONPATH environment variable so that these modules are available to the IPy interpreter (i.e. ipy.exe). To date, I’ve been pretty haphazard about this. But I decided to get a little more structured and put that folder under source control and make it available as “devhawk_ipy”.

So far, I’ve only got three scripts (plus an empty __init__.py) in devhawk_ipy.

• codedom.py (from IronPython and CodeDOM: Dynamically Compiling C# Files)
• ipypulldom.py (from Stream Processing XML in IronPython)
• xml2py.py (from Deserializing XML with IronPython)

Eventually I’ll put my code for working with WPF, LiveFX and Azure into this package, but I’m not happy with where they are yet.

Like ipydbg, devhawk_ipy is up on GitHub. For those non-Git users, I’m will continue to these files up on my SkyDrive. I kind of see SkyDrive as a dumping ground for random content while devhawk_ipy is where stuff goes when it’s a little more polished.

Like IronPython, devhawk_ipy is licensed under the MS-PL. If you’re interested in contributing, feel free to fork and send me patches.

Writing an IronPython Debugger: Dynamic Stack Trace

Now that I can interact with my debugger, it’s time to add a command. I decided to start with something simple – or at least something I thought would be simple - printing a stack trace.

In the unmanaged debugger API, threads have the concept of both stack chains and stack frames. A stack chain represents a segment of the physical stack. In a typical managed app, you’ll have at least two stack chains: the unmanaged stack chain and the managed stack chain. You can interate through the stack chains for a given thread via the Chains property. However, ipydbg is a managed only debugger, so I can ignore the unmanaged stack chain. Instead, I just retrieve the current (managed) chain via the thread’s ActiveChain property.

Within a managed stack chain, there is a collection of stack frames. This is the call stack that managed developers are typically used to working with. It turns out that printing a raw stack trace is very easy to do. Here was my first stab at it:

This elif block is part of the input method I showed last time. It loops thru the frames in the Active Chain of the active thread and prints some data to the console. As I said, pretty easy. Of course, the devil is in the details.

First detail I should call out is that active_thread variable. As per Mike Stall, “there is no notion of "active thread" in the underlying debug APIs. It's purely a construct in a debugger UI to make it easier for end-users.” My console based UI may be rudimentary, but it’s still a UI. Events like OnBreakpoint include the active thread as a event argument, so I stash that away in a variable so it’ll be available to the input loop.

Second detail is the call to get_location. When we last saw get_location, it was returning a formatted string. Since my last post, I’ve refactored the code so it returns the raw location data – a tuple of the raw IP offset and the associated sequence point, if available. I’ve also added a __str__ method to my sequence point object, so when I print it to the console, I get the filename and line nicely formatted.

Finally, there’s all CorMetadataImport code. In addition to wrapping the unmanaged debugger API, CorDebug also wraps the unmanaged metadata API. This code lets me get MethodInfo compatible view of the function metadata for a given stack frame. I use it here to get the type and function name for each frame on the stack.

The end result looks something like this. Note, I’ve replaced “Microsoft.Scripting” with “MS.Scripting” to avoid word wrapping.

OnBreakpoint Initialize Location: simpletest.py:1 (offset: 84)
» t
Managed Stack Trace
   S$2::Initialize simpletest.py:1 (offset: 84)
   MS.Scripting.Runtime.OptimizedScriptCode::InvokeTarget (offset 72)
   MS.Scripting.ScriptCode::Run (offset 0)
   IronPython.Hosting.PythonCommandLine::RunFileWorker (offset 77)
   IronPython.Hosting.PythonCommandLine::RunFile (offset 15)
   MS.Scripting.Hosting.Shell.CommandLine::Run (offset 46)
   IronPython.Hosting.PythonCommandLine::Run (offset 240)
   MS.Scripting.Hosting.Shell.CommandLine::Run (offset 74)
   MS.Scripting.Hosting.Shell.ConsoleHost::RunCommandLine (offset 158)
   MS.Scripting.Hosting.Shell.ConsoleHost::ExecuteInternal (offset 32)
   MS.Scripting.Hosting.Shell.ConsoleHost::Execute (offset 63)
   MS.Scripting.Hosting.Shell.ConsoleHost::Run (offset 390)
   PythonConsoleHost::Main -- (offset 125)

As we can see, we may be on the first line of the python script, but we’ve got a pretty deep stack trace already. Everything but the top-most frame are from the underlying IronPython implementation. Those extra frames obscure the stack frames I actually care about, so it would be nice to hide any stack frames from IronPython or the DLR. It’s easy enough to write a python generator function that filters out frames that from the DLR or IronPython namespaces. In order to get the type name, we need the method_info like we did above. I’ve factored that code into a separate function in order to avoid code duplication.

You’ll notice I’ve added a guard to get_method_info_for_frame in order to ensure that the frame argument is an IL Frame. There are three types of stack frames in the debugger API: IL, native and internal. Most of the frames we’re dealing with are IL frames, but you do run into the occasional lightweight function (i.e. DynamicMethod) frame when debugging IronPython code. Typically, IronPython generates DynamicMethods for all python code except for a few cases related to .NET interop. However, you can’t debug DynamicMethods, so when you run with –D, we generate normal non-dynamic methods instead. However, even when running with –D, we still use DynamicMethods for call site dispatch. Since they’re an implementation detail, we want to filter those out in get_dynamic_frames too.

This gives us a much more manageable stack trace:

OnBreakpoint Initialize Location: simpletest.py:1 (offset: 84)
» t
Stack Trace
   S$2::Initialize -- simpletest.py:1 (offset: 84)

As usual, the latest ipydbg source is up on GitHub.

Writing and IronPython Debugger: Adding Interactivity

Now that ipydbg can set a breakpoint, it’s time to add some interactivity to the app. MDbg supports dozens of commands and currently ipydbg supports none. I’d love for ipydbg to support a wide range of commands like MDbg does, but for now let’s keep it simple and start with two: Continue and Quit. These aren’t very interesting as commands go, but that lets me focus this blog post on adding basic interactivity and future posts on specific commands.

First off, we have to understand how the CorDebug managed API supports interactivity. As we’ve seen, callbacks into the debugger are surfaced as managed events. If we look at the base class for all the debugger event arguments, we see that it exposes a Continue property. If you want the debugger to automatically continue after the event handler finishes running, you set the Continue property to true (which is the default). If you want the debugger to stay paused while you provide the developer a chance to poke around, you set Continue to false. In that case, the debugger stays paused until call process.Continue explicitly.

Once we set the Continue property to false, we need a mechanism to signal the main thread of execution that it’s time to wake up and ask the user what they want to do next. Of course, that’s what WaitHandle and it’s descendents are for. In fact, we’re already using an AutoResetEvent in OnProcessExit to signal that the debugged app has exited so we should exit the debugger. However, now we have two different signals that we want to send: exit the debugger or enter the input loop. I decided to differentiate by using two separate AutoResetEvents:

Instead of a single call to process.Continue I had before, I’ve created an infinite “while True” loop that calls Continue, waits for one of the events to signal, then either exits the loop of enters the input loop (via the input function). Since there are two AutoResetEvents, I need to use the WaitAny method to wait for one of them to signal. WaitAny takes an array, which is kind of clunky to use from IronPython since the array has to be strongly typed. It would be much more pythonic if I could call WaitHandle.WaitAny([terminate_event, break_event]). WaitAny then returns an index into the array indicating which one received the signal. If it was the terminate_event that signaled, I exit the loop (and the application). Otherwise, I enter the input loop. Notice, by the way, in OnBreakpoint that I’m both setting Continue to false and signaling the break_event.

The “input loop” needs to be a loop because the user may want to type in multiple commands before letting the debugged app continue to execute. This means that the input function is implemented as another “while True” loop. When the user does chooses a command that implies the process should continue, I simply exit out of the input function and the outer “while True” loop above executes the continue and waits for a signal.

Here’s what the input function looks like right now with our two basic commands:

I’ve mapped “q” to quit the debugger and spacebar to continue. Since I’m using Console ReadKey, you only have to type the key in question – no return needed. For continue, we don’t do anything but exit the input loop by returning. Continue gets called as part of the other loop and since we haven’t/can’t add additional breakpoints the debugged app will run until it ends. For quit, I call the Terminate method on process, hard coding the return value to 255. However, Terminate implicitly continues the debugged process. Since you can’t continue a running process, the call to Continue in the outer loop throws an exception. I avoid this exception by adding the call to Stop before Terminate. As per the Stop docs, the debugger maintains a “stop counter” and only resumes the debugged process when the counter reaches zero.  Calling Stop increases the stop counter by one, calling Terminate decreases it by one, then the outer loop Continue  call decreases it to zero and the process continues, terminates and fires the OnProcessExit event handler as usual.

Now that we have a basic interactive loop, I’ll be able to add more interesting commands. I’m guessing at some point, I’ll need to refactor input a bit – I’m guessing a huge if/elif/else statement is going to get ugly fast, but I’ll worry about that when it gets out of hand. As usual, the latest ipydbg source is up on GitHub.

Writing an IronPython Debugger: Setting a Breakpoint

Now that we have a debugger process up and running, let start adding some actual features. First up, we want to be able to set breakpoints. One of the nice things MDbg does is auto-set a breakpoint on the entrypoint function. For ipydbg, we’re going to auto-set a breakpoint on the first line of the python file being debugged.

In order to set a breakpoint, we need debugger symbols. They allow us to translate between “line one of simpletest.py” and the actual location in the code and back. We’re all used to seeing the PDB files that are produced when we compile a C# assembly. Unsurprisingly, the symbol store binder provides a method to load these PDB files from disk. But where do IronPython debug symbols come from? I know from my extensive reading of the ipy.exe command line parameters that you pass –D to enable application debugging, but since all the IL is being generated in memory, how does the debugger get access to the PDB files?

It turns out the debugger API includes a UpdateModuleSymbols callback method that the runtime uses to notify the debugger when the symbols change. The debugger symbols are provided in an IStream, and then you use the symbol binder to get a symbol reader. The .NET Framework already provides a managed API for reading and writing debug symbols. However, that API doesn’t support loading symbols from a stream, so the MDbg code includes it’s own wrapper around the symbol binder API to include that functionality. Here’s some code to get the debug symbol reader for an updated module and iterate through the associated files:

If we run this version of ipydbg on simpletest.py with the IPy 2.0.1 release and the Python standard library installed, OnUpdatedModuleSymbols gets called six times, once for each python file that gets loaded when simpletest runs. (site.py, os.py, ntpath.py, stat.py, UserDict.py and simpletest.py). BTW, I tried running this code on the latest build of IPy (changeset 47624) and I’m getting a COM Interop exception. So for now, stick with 2.0.1.

Now that we can get these dynamically generated debug symbols, we can use them to create a breakpoint on the first line of the script being debugged. Everytime OnUpdateModuleSympols is called, I try to bind the initial breakpoint (unless it’s already been bound of course) by calling the following create_breakpoint function.

This code translates a given document/line into a function/offset where we can set a breakpoint. To do this, we use sequence points which as per Rick Byers are “used to mark a spot in the IL code that corresponds to a specific location in the original source”. So once we find the function that corresponds to a given line of code, we iterate over the sequence points until we find the one that matches the line we want to break on. If we find a matching sequence point, we set the breakpoint there. If we don’t, we set the breakpoint on the function itself. get_sequence_points is a simple wrapper around ISymbolMethod GetSequencePoints. The original API is pretty ugly to use – managing six separate arrays of information – so get_sequence_points turns it into a generator function you can iterate over.

Now that the breakpoint is set, we want to trap the breakpoint event as well. That’s easy enough, we create an event handler for process.OnBreakpoint similar to the OnUpdateModuleSymbols event above. Eventually, we’ll have the ability to step when we break, but for now I’m just going to print out the current location when the breakpoint is hit. This is kind of the reverse of the operation above. Setting a breakpoint means going from a source location to an IL offset within a function. Printing the current location means going from an IL offset in a function back to the source location. Here’s the function to do that:

Given a symbol reader and a debug thread, get_location returns a location string. It loops thru the sequence points, similar to create_breakpoint, in order to find the closest corresponding line of python code to the current offset (check out Mike Stall’s blog as for why I’m checking for 0xfeefee). In order to make this work, I need the symbol reader for the module that I retrieved in OnUpdateModuleSymbols. For now, I’m stashing the reader in a global dictionary keyed by the module named symbol_readers where OnBreakpoint can access it.

Ipydbg isn’t interactive yet, but it is now running, setting a breakpoint and successfully breaking at that breakpoint. As usual, the latest version of ipydbg is up on GitHub.

CodeHTMLer Language Definition for Python

As I’ve blogged before, I use CodeHTMLer to post code snippets on my blog. I hear SyntaxHighlighter is the new hotness, but since it relies on CSS the syntax highlighting only appears on the website and not in the RSS reader.

The problem with CodeHTMLer is that it only supports a handful of languages out of the box. But the language definition file is simple enough – just an XML file with a bunch of regular expressions. When I was doing a lot of F# work, I wrote an F# language definition. Now that I’m on the IronPython team, go figure I’m writing a lot of code in Python. I *know* I’ve written a Python language definition for CodeHTMLer more than once, but I would forget to post it and then lose it when I paved my laptop hard drive. So after doing this three or four times, I’ve finally remembered to put it up on my SkyDrive.

If you want to install this yourself to colorize Python code snippets with CodeHTMLer, follow the directions I posted earlier with the F# language definition.

Writing an IronPython Debugger: Hello, Debugger!

Since I’m guessing most of my readers have never build a debugger before (I certainly hadn’t), let’s start with the debugger equivalent of Hello, World!

I start by adding a reference to the CorDebug library I discussed at the end of my last post (that’s the low level managed debugger API plus the C# definitions of the various COM APIs). Then I need both the path to the IPy executable as well as the script to be run, which is passed in on the command line (sys.argv). For now, I just use Reflection to find the path to the current ipy.exe and use that. I use those to build a command line – you’ll notice I’m adding the –D on the command line to generate debugger symbols.

Next, I define two event handlers: OnCreateAppDomain and OnProcessExit. When the AppDomain is created, the debugger needs to explicitly attach to it. When the process exits, we signal an AutoResetEvent to indicate our program can exit.

Then it’s a simple process of creating the CorDebugger object, creating a process, setting up the process event handlers and then running the process via the call to Continue. We then wait on the AutoResetEvent for the debugged process to exit. And voila, you have the worlds simplest debugger in about 30 lines of code.

To run it, you run the ipy.exe interpreter and pass in the ipydbg script above and the python script to be debugged. You also have to pass –X:MTA on the command line, as the ICorDebug objects only work from a multi-threaded apartment. When you run it, you get something that looks like this:

» ipy -X:MTA ipydbg.py simpletest.py
OnCreateAppDomain DefaultDomain
35
OnProcessExit

Simpletest.py is a very simple script that prints the results of adding two numbers together. Here, you see the event handlers fire by writing text out to the console.

For those of you who’d like to see this code actually run on your machine, I’ve created an ipydbg project up on GitHub. The tree version that goes with this blog post is here. If you’re not running Git, you can download a tar or zip of the project via the “download” button at the top of the page. It includes both the CorDebug source as well as the ipydbg.py file (shown above) and the simpletest.py file. It also has a compiled version of CorDebug.dll, so you don’t have to compile it yourself (for those IPy only coders who don’t have VS on their machine).

Writing an IronPython Debugger: MDbg 101

Before I start writing any debugger code, I thought it would help to quickly review the .NET debugger infrastructure that is available as well as the design of the MDbg command line debugger. Please note, my understanding of this stuff is fairly rudimentary – Mike Stall is “da man” if you’re looking for a .NET debugger blogger to read.

The CLR provides a series of unmanaged APIs for things like hosting the CLR, reading and writing CLR metadata and – more relevant to our current discussion – debugging as well as reading and writing debugger symbols. These APIs are exposed as COM objects. The CLR Debugging API allows you to do those all the things you would expect to be able to do in a debugger: attach to processes (actually, app domains), create breakpoints, step thru code, etc. Of course, being an unmanaged API, it’s pretty much unavailable to be used from IronPython. Luckily, MDbg wraps this unmanaged API for us, making it available to any managed language, including IronPython.

The basic design of MDbg looks like this:

At the bottom is the “raw” assembly, which contains the C# definitions of the unmanaged debugger API – basically anything that starts with ICorDebug and ICorPublish. Raw also defines some of the metadata API, since that’s how type information is exposed to the debugger.

The next level up is the “corapi” assembly, which I refer to as the low-level managed debugger API. This is a fairly thin layer that translates the unmanaged paradigm into something more palatable to managed code developers. For example, COM enumerators such as ICorDebugAppDomainEnum are exposed as IEnumerable types. Also, the managed callback interface gets exposed as .NET events. It’s not perfect – the code is written in C# 1.0 style so there are no generics or yields.

Where corapi is the low-level API, “mdbgeng” is the high-level managed debugger API. As you would expect, it wraps the low-level API and provides automatic implementations of common operations. For example, this layer maintains a list of breakpoints so you can create them before the relevant assembly has been loaded. Then when assemblies are loaded, it goes thru the list of unbound breakpoints to see if any can be bound. It’s also this layer that automatically creates the main entrypoint breakpoint.

Finally, at the top we have the MDbg application itself, as well as any MDbg extensions (represented by the … in the diagram above). The mdbgext assembly defines the types shared between MDbg.exe and the extension assemblies. MDbg has some cool extensions – including an IronPython extension – but for now I’m focused on building something as lightweight as possible, so I’m going to forgo an extensibility mechanism, at least for now.

My initial prototype was written against the high-level API. There were two problems with this approach. The first is that there’s no support for Just My Code in the high-level API. As I mentioned in my last post, JMC support is critical for this project. Adding JMC support isn’t hard, but I’m trying to make as few changes as possible to the MDbg source, since I’m not interested in forking and maintaining that code. Second, while the low-level API provides an event-based API (OnModuleLoad, OnBreakpoint, OnStepComplete, etc), the high-level API provides a more console-oriented looping API. I found the event-driven API to be cleaner to work with and I’m thinking it will work better if I ever build a GUI version of ipydbg. So I’ve decided to work against the low-level API (aka corapi).

I mentioned above that I didn’t want to change the MDbg source, but I did make one small change. The separation of corapi and raw into two separate assemblies is an outdated artifact of an earlier version of MDbg. So I decided to combine these two into a single assembly called CorDebug. Other than some simple cleanup to assembly level attributes to make a single assembly possible, I haven’t changed the source code at all.

Writing an IronPython Debugger: Introduction

A while back I showed how you can use Visual Studio to debug IronPython scripts. While that works great, it’s lots of steps and lots of mouse work. I yearned for something lighter weight and that I could drive from the command line.

The .NET framework includes a command line debugger called MDbg, but after using it for a bit, I found it didn’t like it very much for IronPython debugging. Mdbg automatically sets a breakpoint on the main entrypoint function, but only if it can find the debugging symbols. So when you use Mdbg with the released version of IPy, the breakpoint never gets set. Instead, you have to trap the module load event, set a breakpoint in the python file you’re debugging, then stop trapping the module load event. Every Time. That gets tedious.

Another problem with MDbg is that it’s not Just-My-Code (aka JMC) aware. JMC is this awesome debugging feature that was introduced in .NET 2.0 that lets the debugger “paint” the parts of the code that you want to step thru (aka “My Code”). By default, Visual Studio marks code with symbols as “my code” and code without symbols as “not my code”. [1] We don’t ship symbols with IronPython releases, so Visual Studio does only steps thru the python code. MDbg doesn’t support JMC, so I often found myself stepping into random parts of the IronPython implementation. That’s even more tedious.

Luckily, the source code to MDbg is available. So I got the wacky idea to build a debugger specifically for IronPython. CPython includes pdb (aka Python Debugger, not Program Database) but we don’t support it because we haven’t implemented settrace. Thus, ipydbg was born.

Over the course of this series of blog posts, I’m going to build out ipydbg. I have built out a series of prototypes so I fairly confident that I know how to build it. However, I’m not sure what it will look like at the end. If you’ve got any strong opinions on it one way or the other, be sure to email me or leave me comments.

BTW, major thanks to my VSL teammate Mike Stall (of Mike Stall's .NET Debugging Blog). Without his help, I would probably still be trying to make heads or tails of the MDbg source.

[1] VS uses the DebuggerNonUserCode attribute to provide fine grained control of what is considered “my code” and should be stepped thru.

IronPython 2.0.1

I’m on vacation this week, but I wanted to quickly point out that we shipped IronPython v2.0.1 last Friday. This has been a performance focused release, as you can see via our 2.0 vs. 2.0.1 benchmarks. We have improved our PyStone performance by about 11.5% and our Richards performance by just over 4%. Thanks to Dino for the perf improvements and Dave for the great performance report.

IronPython and CodeDOM: Dynamically Compiling C# Files

As part of my series on using IronPython with WPF [1], I built an extension method in C# that does dynamic member resolution on WPF FrameworkElements. The upshot of this code is that I can write “win1.listbox1” instead of “win1.FindName(‘listbox1’)” when using WPF objects from Python or any DLR language. Convenient, right?

The problem with this approach is that the C# extension method gets compiled into an assembly that’s bound to a specific version of the DLR. I recently started experimenting with a more recent build of IronPython and I couldn’t load the extension method assembly due to a conflict between the different versions of Microsoft.Scripting.dll. Of course, I could have simply re-compiled the assembly against the new bits, but that would mean every time I moved to a new version of IronPython, I’d have to recompile. Worse, it would limit my ability to run multiple versions of IronPython on my machine at once. I currently have three – count ‘em, *three* – copies of IronPython installed: 2.0 RTM, nightly build version 46242, and an internal version without the mangled namespaces of our public CodePlex releases. Having to manage multiple copies of my extension assembly would get annoying very quickly.

Instead of adding a reference to the compiled assembly, what if I could add a reference to a C# file directly? Kinda like how adding references to Python files works, but for statically compiled C#. That would let me write code like the following, which falls back to adding a reference to the C# file directly if adding a reference to the compiled assembly fails.

Since this technique uses CodeDOM, I decided to encapsulate the code in a Python module named codedom, which is frankly pretty simple. As a shout-out to my pals on the VB team, I broke compiling out into it’s own separate function so I could easily support adding VB as well as C# files.

The compile function uses a CodeDOM provider, which provides a convenient function to compile an assembly from a single file. The only tricky part was adding the references correctly. Of the five references in this example, the only one CodeDOM can locate automatically is System.dll. For the others, it appears that CodeDOM needs the full path to the assembly in question.

Of course, hard-coding the assembly paths in my script would be too fragile, so instead I use partial names. I load each referenced assembly via Assembly.LoadWithPartialName then pass it’s Location to the CodeDOM provider via the CompilerParameters object. I realize that loading an assembly just to find its location it kind of overkill but a) I couldn’t find another mechanism to locate an assemblies location given only a partial name and b) I’m going to be loading the referenced assemblies when I load the generated assembly anyway, so I figured it loading them to find their location wasn’t a big deal. Note, that typically you’re used to passing a string to clr.AddReference, but it also can accept an assembly object directly.

Of course, this approach isn’t what you would call “fast”. Loading the pre-compiled assembly is much, much faster than compiling the C# file on the fly. But I figure slow code is better than code that doesn’t work at all. Besides, the way the code is written, I only take the extra compile hit if the pre-compiled assembly won’t load.

I stuck my codedom.py file up on my SkyDrive. Feel free to leverage as you need.

[1] I had to put that series on the back burner in part because the December update to Windows Live totally broke my WPF photo viewing app. I’ve got a new WPF app I’m working on, but I’m not quite ready to blog about it yet.

Nightly Builds Technical Info

Here are some technical details on my Nightly Builds solution. I broke them into a separate post because I figured most people are more interested in the actual service than how it’s built.

As you might expect, I built most of the solution in IronPython. All of the download, build, compress and Azure upload code was written in IPy. The one part I didn’t write in IPy was the Azure cloud web app, which I wrote in C#. Jon Udell’s been investigating getting IPy to run in Azure, but I just wanted something quick and dirty (as you can see from the utter lack of formatting) so I decided to use C# instead. Man, were my ASP.NET skills rusty.

As for the IronPython parts, for the most part I’m using external tools for downloading, building and compressing. I use the Source Control RSS Feed to discover recent source code changesets, CodePlex Client to download source from CodePlex, MSBuild to build the binaries, 7-zip to compress the binaries and the StorageClient library sample to upload the compressed binaries up to Azure blob storage.

For building and compressing, I’m literally shelling out to MSBuild and 7-Zip via os.system. I looked at programmatically building via the MSBuild API, but I ran into an assembly binding bug that I wasn’t motivated enough to work around. As for creating zip files programmatically, IronPython doesn’t have a zlib module implementation yet so I just used 7-Zip’s command line utility instead.

For downloading form CodePlex, I originally started by shelling out to CodePlex Client. However, I wanted the ability to cloak folders – for example \Tutorial and \Src\Tests – that weren’t required to build. CodePlex Client has a very useful TFS library embedded in it – the build process combines all the libraries into a single executable via ILMerge. I could have compiled my own version of the TFS library, but instead I just load cpc.exe as an assembly reference via clr.AddReferenceToFileAndPath. It’s a nifty trick Jim Hugunin showed me once.

Uploading to Azure was very straightforward because of the StorageClient library. Here’s the code to create a blob container object (creating the actual blob container if it doesn’t already exist) and to upload a file to a container.

I’ve been working on some pure IronPython code to access the blob storage REST API directly, but that’s primarily to familiarize myself with the service. At some point, I’m going to want to leverage Table Storage but my brief experimentation with the StorageClient Table Storage interface makes me think that it depends on static typing too much to be useful for IPy. If that turns out to be true, the Table Storage REST API will be my only option.

As you can see in the code above, these Azure blob containers are set to be publically accessible (via ContainerAccessControl.Public argument passed to CreateContainer). So for my C# app, I’m simply using calling XDocument.Load with the List Blobs operation url, shaping the results via LINQ to XML and binding them to nested ASP.NET Repeater controls.

Assuming people find this useful, I’m thinking of some additional improvements, in order of what I’m likely to get to first:

• Caching Project Info in the cloud app
  Currently, I’m hitting getting and processing the list of binary releases on every request. I’m sure caching that data to make it more efficient.
• Virtual Build Environment
  Currently, I’m just building on my laptop. It would be nice to have a clean environment dedicated to running the build script.
• Auto-Build
  My script uses the RSS feed to find the recent checkins, but I have to manually kick off the process. I’d like it to set it up as a service that periodically checks the source code RSS feed automatically and downloads and builds any new releases that it finds.
• Table Storage for Build Metadata
  Today, I am simply grabbing the list of all uploaded compressed binaries for a given project, parsing their names, and displaying that as a hierarchical list on the project page. If I used Table Storage, I could add additional metadata including social software features like ratings and comments.
• Amazon EC2 Virtual Build Environment
  If I’m creating a virtual machine for my build environment, I could look at hosting it on Amazon EC2. They support Windows now after all. Ideally, I’d use an Azure worker role for compiling and compressing builds, but our build tools need access to the file system.

IronPython Nightly Builds

IronPython 2.0 shipped about a month ago, but we’re still chugging along with our post 2.0 work. We’ve shipped seven source code releases since we shipped 2.0 and we should be back to our normal schedule of updating the source 2-3 times a week schedule by next week. Given how often we ship source, we’re thinking of extending the the time between binary drops. Binary releases have to be signed and there’s a fairly arduous process we have to go thru in order to get each binary release out the door.

However, there’s something nice and convenient about downloading a pre-compiled binary release. So I spent my Christmas vacation building a script to download and build IronPython nightly builds. Once built, I compress the binaries and upload them to Azure blob storage. Finally, I built a *very* simple cloud app for users to view and download available nightly builds. As an extra benefit, I’m also providing nightly builds of the DLR.

Please note, these are *NOT* official Microsoft releases of IronPython and/or DLR. They aren’t signed and they haven’t gone through the aforementioned release process. I’m just downloading the public source, building it with the publicly available tools, then making them available on a a publicly accessible website.

The website for the IronPython (and DLR) nightly builds is http://nightlybuilds.cloudapp.net.

As usual, I welcome any feedback. Is having prebuilt unsigned binaries of IPy releases useful? Do you want IronRuby binaries as well? What about social features (rating releases, comments, etc)? Please let me know what you think.

IronPython and LiveFX: Raw HTTP Access

One of the cool things about the Live Framework is that while there’s a convenient .NET library available, you can use the raw HTTP interface from any platform. LiveFX data is served up over HTTP and is available in ATOM, RSS, JSON or POX formats. As I’ve already shown, you can easily use the .NET library from IronPython, but I wanted to try working with the raw HTTP interface to get a feel for that as well.

Unfortunately, it was harder than I expected it to be. The big issue is that the documentation on how to LiveFX authorization tokens via raw HTTP is fairly sparse and occasionally contradictory. For example, there’s a whole section on Authentication and Live Framework, but it doesn’t cover this scenario. Luckily, I was able to figure it out with the help of AtomPub Project Manager LiveFX Sample, a post on Alex Feinman’s blog, a post on Emmanuel Mesas’ blog and a little groveling around with Reflector. It does appear that the auth docs are in flux –Emmanuel refers to this MSDN article as being about RPS Soap requests, but it’s actually about delegated authority. (Is MSDN reusing URLs? Bad idea.) Also, the sample code has a comment that reads “to be replaced by delegated authorization” so it looks like changes are coming. In other words, no promises on how long this code will work!

If you look at the AtomPub Project Manager sample, there’s a WindowsLiveIdentity.cs file that implements static GetTicket method that looks similar to both the code on Alex’s blog as well as the implementation of GetWindowsLiveAuthenticationToken. The upshot is that there’s a WS-Trust endpoint for Windows Live at https://dev.login.live.com/wstlogin.srf. You send it a RequestSecurityToken (aka RST) message (with a couple of extra WL specific extensions) and it responds with the security token you’ll need for accessing the LiveFx HTTP endpoints.

I ported the GetTicket function over to IronPython. I’m using .NET classes like WebRequest and XmlReader, but there’s nothing fancy here so I would expect it to be easy enough to port over to the standard Python library.

This code simply uses a WebRequest object to post the RST message to the WS-Trust enpoint then parses the result to find the token. get_RST_message uses standard Python string formatting to generate the RST message that gets posted to the WS-Trust endpoint. I’m not exactly sure why you need to convert the token value to a byte array and then Base64 encode it, but that’s what the sample code does so I did it to.

Once you have the authentication ticket, you need to download root service endpoint document in order to get the base URL and the profiles link. Then you can download all the profiles or you can download a specific one if you know it’s leet-speak identifier. LiveFX data can be downloaded in a variety of formats: ATOM, JSON, RSS or POX. You choose your format by setting the Accept and Content-Type headers.

I wrote the following functions, the generic boilerplate download function as well a specific versions for downloading JSON and POX:

Using JSON in Python is really easy, since I can simply eval the returned string and get back Python dictionary objects, similar to what you can do in Javascript.

Here’s some code that uses the get_WL_ticket and download_json functions above to retrieve the the user’s Personal Status Message

POX is also fairly easy, though a bit more verbose than JSON. The sample code, which I have stuck on my SkyDrive, includes both POX and JSON code, so you can compare and contrast the differences.

IronPython and LiveFX: Ori’s LiveOE.py

Ori Amiga is a Group Program Manager over in the Live Framework team whom you might have seen at PDC08 delivering the Lap Around LiveFX & Mesh Services and LiveFX Programming Model Architecture and Insights talks. And apparently, he’s an IronPython fan as posted a small LiveFX Python module to his blog. It’s pretty simple – it only wraps Connect and ConnectLocal - but it does cut about ten lines of path appending, reference adding and module importing code into a single import statement. Here’s the profile access script from my last post rewritten to use Ori’s LiveOE module.

FYI, make sure you update the sdkLibsPath in LiveOE.py – I’m not sure where Ori has installed the LiveFX SDK, but it’s *not* in the location suggested by the read me file.

BTW, it turns out the WL Profile information is read only which answers a question I had. However, reading the thread it sounds like they will eventually get around to making it read-write at some point.

IronPython and LiveFX: Accessing Profiles

I recently got access to both the Windows Azure and Live Framework CTP programs. Frankly, I’m very interested in Live Mesh, so I decided to start with a simple LiveFX program. Scott (aka ScottIsAFool) at LiveSide posted a “quick and dirty” console app that pulls info from a user’s profile via LiveFx. It’s not Mesh per se, but it does use the same framework and resource model so I decided to port it to IronPython. FYI, this app won’t run unless you’ve been received a LiveFx CTP token and provisioned yourself.

I did modify the app slightly, reading the WLID and password off the console – I was *sure* I would accidently post my personal credentials if I left them embedded in the app. Otherwise, it’s a straight port. First, I add references the LiveFX dlls. Since they’re not local to my script, I add the directory where they’re installed to sys.path, which lets me call clr.AddReference directly. Then I retrieve the user’s ID and password using raw_input (Python’s equivalent to Console.ReadLine). Finally, I connect to the user’s LiveOperatingEnvironment and pull their name, email address, personal status message and the number of contacts they have.

As per the original app, I use LINQ to find the right profile as well as count the number of contacts. I was able to reuse the linq.py file I wrote for my Rock Band song list screen scraper (though I did have to add the Count function since I hadn’t needed it previously). I’ve posted this script on my SkyDrive, and it includes my most recent linq.py file.

BTW, it doesn’t appear that you can set the PersonalStatusMessage programmatically, at least not currently. I was thinking it would be cool to build an app that sets your PSM via Twitter, but the set method of PersonalStatusMessage is marked internal. In fact, all the set methods of all the profile properties I looked at are marked internal. If someone knows how to update LiveFX resource objects in the current CTP, I’d appreciate it if you dropped me a line or left me a comment.

IronPython RTM News Gets Around

I just hit the MSDN home page, and what should I see?

It’s cool to see JasonZ, aka my group’s general manager, blogging about our product.

I also fired up Visual Studio, and IronPython is the top headline there too:

Not sure why the news is dated September 18th, but hey it’s really cool to see IronPython (not to mention the DLR, with the second headline) getting this kind of visibility.

IPy RTW FTW!

This is a very pretty sight. It’s a screenshot from the IronPython CodePlex home page showing that 2.0 is the “current release”. Yes that’s right, dear reader, IronPython 2.0 has officially been released!

Get it now!

This release marks the end of a very busy year for me, nine months to the day since I accepted the offer to join the dynamic languages team. Between helping ship IronPython 2.0 and helping manage the languages and tools PDC08 track, I’ve been swimming in the deep end of the pool all year. Feels good to not have any immediate deliverables for the next month or two.

Major, major props to Dino, IronCurt, Dave and Srivatsn who have done the heavy lifting on the IPy side this release. Also major props to the DLR team, who are releasing the final 0.9 version of the DLR later today in concert with IPy 2.0. (Update: the DLR 0.9 RTW bits are now available) And of course, HUGE HUGE HUGE thanks to the vibrant IPy community, many of whom are listed by name in the release notes.

Even with 2.0 finally out the door, there’s no rest for the dynamic. As per the release notes, “we’re planning on releasing IronPython 2.0.1 fairly soon” so keep those bug reports coming. Going forward, we’ve got big plans for IronPython and we rely heavily on the continued input from our community, so please keep telling us where we can improve.

On a personal note, the past nine months have been busy – very busy – but they’ve also been a blast. Frankly, I was hesitant about joining the product groups for a long time because I was worried about the grind, the culture, the overall experience. Turns out my fears were overblown, though I’m thinking that’s at least partially related to the fact that I work on a “little” project like IronPython rather than a huge project like Visual Studio.

IronPython and Linq to XML Part 4: Generating XML

Now that I have my list of Rock Band songs and I can get the right Zune metadata for most of them, I just need to write out the playlist XML. This is very straight forward to do with the classes in System.Xml.Linq.

XElement’s constructor takes a name (XName to be precise) and any number of child objects. These child objects can be XML nodes (aka XObjects) or simple content objects like strings or numbers. If you pass an IEnumerable, the XElement constructor will iterate the collection and add all the items as children of the element. If you’ve had the displeasure of building an XML tree using the DOM, you’ll really appreciate XElements’s fluent interface. I was worried that Python’s significant whitespace would force me to put all the nested XElements on a single line, but luckily Python doesn’t treat whitespace inside parenthesis as significant. 

Creating collections in Python is even easier than it is in C#. Python’s supports a yield keyword which is basically the equivalent of C#’s yield return. However, Python also supports list comprehensions (known as generator expressions), which are similar to F#’s sequence expressions. These are nice because you can specify a collection in a single line, rather than having to create a separate function, which is what you have to do to use yield. I have two generator expressions: (XAttribute(key, trk[key]) for key in trk) creates a collection of XAttributes, one for every item in the trk dictionary and (GenMediaElement(song) for song in songs) which generates a collection of XElements, one for every song in the song collection.

Once I’ve finished building the playlist XML, I need to write it out to a file. Originally, I used Python’s built in open function, but the playlist file had to be UTF-8 because of band names like Mötley Crüe. Zune’s software appears to always use UTF-8. In addition to setting the encoding, I also specify to use indentation, so the resulting file is somewhat readable by humans.

The playlist works great in the Zune software, but since it’s a streaming playlist there’s no easy way to automatically download all the songs and sync them to your Zune device. I expected to be able to right click on the playlist and select “download all", but there’s no such option. Zune does have a concept called Channels where the songs from a regularly updated feed are downloaded locally and synced to the device. However, the Zune software appears to be hardcoded to only download channels from the catalog service so I couldn’t tap into that. If anyone knows how to sign up to become a Zune partner channel, please drop me a line.

Otherwise, that’s So there you have it. As usual, I’ve stuck the code up on my SkyDrive. If I can remember, I’ll try and run the script once a week and upload the new playlist to my SkyDrive as well.

IronPython and Linq to XML Part 3: Consuming Atom Feeds

Now that I have my list of Rock Band songs, I need to generate a Zune playlist. I wrote that Zune just uses the WMP playlist format, but that’s not completely true. Media elements in a Zune playlist have several attributes that appear unique to Zune.

Because of Zune Pass, Zune supports the idea of streaming playlists where the songs are downloaded on demand instead of played from the local hard drive. In order to enable this, media elements in Zune playlists can have a serviceID attribute, a GUID that uniquely identifies the song on the Zune service. We also need the song’s album and duration – the Zune software summarily removes songs that don’t include the duration.

Of course, the Rock Band song list doesn’t include the Zune song service ID. It also doesn’t include the song’s album or duration. So we need a way, given the song’s title and artist (which we do have) to get its album, duration and service ID. Luckily, the Zune service provides a way to do exactly this, albeit an undocumented way. Via Fiddler2, I learned that Zune exposes a set of Atom feed web services on catalog.zune.net that the UI uses when you search the marketplace from the Zune software. There are feeds to search by artist and by album but the one we care about is the search by track. For example, here’s the track query for Pinball Wizard by The Who.

Since these feeds are real XML, I can simply use XDocument.Load to suck down the XML. Then I look for the first Atom entry element using similar LINQ to XML techniques I wrote about last time. If there’s no Atom elements, that means that the search failed – either Zune doesn’t know about the song or it can’t find it via the Rock Band provided title and artist. Of the 461 songs on Rock Band right now, my script can find 417 of them on Zune automatically.

Of course, since the Zune data is in XML instead of HTML, finding the data I’m looking for is much easier that it was to find the Rock Band song data. Here’s the code pull the relevant information out of the Zune catalog feed that we need.

A few quick notes:

• The code above isn’t valid Python, I added a couple of carriage returns (albumArtist and albumTitle) to get it to read well on the blog without wrapping badly.
• song.search_string returns the song title and artist as a plus delimited string. i.e. pinball+wizard+the+who. However, many Rock Band songs end in a parenthetical like (Cover Version) so I automatically strip that off for the search string
• duration in the Atom feed is stored like PT3M23S, which means the song is 3:23 long. The playlist file expect the song length in milliseconds, so I use a .NET regular expression to pull out the minutes and seconds and do the conversion. It’s not exact – songs lengths usually aren’t exactly a factor of seconds, but as far as I can understand, Zune just uses that to display in the UI – it doesn’t affect playback at all.

Now I have a list of songs with all the relevant metadata, next time I’ll write it out into a Zune playlist file.

Early Christmas from Iron Languages and DLR

Tomorrow may be Thanksgiving, but the Microsoft DevDiv dynamic language teams are trying to make it feel like Christmas with three separate pre-holiday releases.

1. IronPython 2.0 RC2 
   We were really hoping to only have one release candidate, but we ended up with a couple of significant bugs that we couldn’t push off to 2.0.1. With December holidays coming soon, RC2 has a pretty small window before we declare RTM so now is the time to download the release and try your code out.
2. IronRuby 1.0 Alpha 2 
   There’s been zero blog traffic on this, just a notice on the IronRuby mailing list. As per said notice, “Notable features” include “the inclusion of iirb.bat, igem.bat, irails.bat, irake.bat”.
3. New DLR CodePlex Project T
   he DLR source has been available as part of IronPython for over a year but now they have their own home on CodePlex. Check out the Release Notes for an overview, reads some Docs and Specs or just download their initial v0.9 beta. Their v0.9 beta is synced with IPy 2.0 RC2 (and their v0.9 final will sync with IPy 2.0 RTM) but it also includes synced versions of IronRuby and ToyScript in both source and binaries. Plus, Sesh has promised “weekly code drops”. Finally, unlike IronPython and IronRuby, DLR is using the discussion section of their CodePlex site – I’m eager to see how well the new-ish discussion/mailing list integration works.

So there you go, new versions of IronPython and IronRuby plus a whole new DLR CodePlex project to boot. Enjoy.

IronPython and Linq to XML Part 2: Screen Scraping

First, I need to convert the HTML list of Rock Band songs into a machine readable format. That means doing a little screen scraping. Originally, I used Beautiful Soup but I found that UnicodeDammit got confused on names like Blue Öyster Cult and Mötley Crüe. I’m guessing it’s broken because IronPython doesn’t have non-unicode strings.

Instead, I used SgmlReader to provide an XmlReader interface over the HTML, then queried that data via Linq to XML. I used the version of SgmlReader from MindTouch since they include a compiled binary and it seems to be the only active maintained version. I wrapped it all up in a function called load that loads HTML from either disk or the network (based on the URI scheme) into an XDocument.

I call load, passing in the URL to the list of songs. The “official” Rock Band song page loads the actual content from a different page via AJAX, so I just load the actual list directly via my load function.

Once the HTML is loaded as an XDocument, I need a way to find the specific HTML nodes I was looking for. As I said earlier, XDocument uses Linq to XML – there is not other API for querying the XML tree. In the HTML, there’s a div tag with the id “content” that contains all the song rows as table row elements. I built a simple function that uses the LINQ Single method to find the tag by it’s id attribute value.

(Side note – I didn’t like the verbosity of the “a != None and a.Value == id” line of code, by XAttributes are not comparable by value. That is, I can’t write “node.Attribute(‘id’) == XAttribute(‘id’, id)”. And writing “node.Attribute(‘id’).Value == id” only works if every node has an id attribute. Not making XAttribute comparable by value seems like a strange design choice to me.)

LINQ to objects works just fine from IronPython, with a few caveats. First, IronPython doesn’t have extension methods, so you can’t chain calls together sequentially like you can in C#. So instead of collection.Where(…).Select(…), you have to write Select(Where(collection, …), …). Second, all the LINQ methods are generic, so you have to use the verbose list syntax (for example: Single[object] or Select[object,object]). Since Python doesn’t care about the generic types, I wrote a bunch of simple helper functions around the common LINQ methods that just use object as the generic type. Here are a few examples:

Once I have the content node, all the songs are in tr nodes beneath it. I wrote a function called ScrapeSong that transforms a song tr node into a Song object (which I’ll talk about in the next installment of this series). I use LINQ methods Select, OrderBy and ThenBy to provide me an enumeration of Song objects, ordered by date added (descending) than artist name.

And that’s pretty much it. Next, I’ll iterate thru the list of songs and get the details I need from Zune’s catalog web services in order to write out a playlist that the Zune software will understand.

IronPython and Linq to XML Part 1: Introduction

Shortly after I joined the VS Languages team, we had a morale event that included a Rock Band tournament. I didn’t play that day in the tournament since I had never played before, but I was hooked just the same. I got Rock Band for my birthday, Rock Band 2 shortly after it came out in September and I’m hoping to get the AC/DC Track Pack for Christmas.

There are lots of songs available for Rock Band - 461 currently available between on-disc and downloadable tracks – with more added every week. Frankly, there’s lots of music on that list that I don’t recognize. Luckily, I’m also a Zune Pass subscriber, so I can go out and download all the Rock Band tracks and listen to them on my Zune. But who has time to manually search for 461 songs? Not me. So I wrote a little Python app to download the list of Rock Band songs and save it as a Zune playlist.

I ended up use Linq to XML very heavily in this project. Zune playlists use the same XML format as Windows playlists, Zune exposes the backend music catalog via a Atom feeds and I used Chris Lovett’s SgmlReader to expose the HTML list of Rock Band songs as XML. I realize Linq to XML wasn’t on “the list”, but I had a specific need so it got bumped to the head of the line.

BTW, for those who just want the playlist, I stuck it on my Skydrive. Unfortunately, there’s no Skydrive API right now, so I can’t automate uploading the new playlist every week. If anyone has alternative suggestions or a way to programmatically upload files to SkyDrive, let me know.

IronPython and WPF Part 5: Interactive Console

One of the hallmarks of dynamic language programming is the use of the interactive prompt, otherwise known as the Read-Eval-Print-Loop or REPL. Even though I’m building a WPF client application, I’d still like to have the ability to poke around and even modify the app as it’s running from the command prompt, REPL style.

If you work thru the IronPython Tutorial, there are exercises for interactively building both a WinForms and a WPF application. In both scenarios, you create a dedicated thread to service the UI so it can run while the interactive prompt thread is blocked waiting for user input. However, as we saw in the last part of this series, UI elements in both WinForms and WPF can only be accessed from the thread they are created on. We already know how to marshal calls to the correct UI thread – Dispatcher.Invoke. However, what we need is a way to intercept commands entered on the interactive prompt so we can marshal them to the correct thread before they execute.

Luckily, IronPython provides just such a mechanism: clr module’s SetCommandDispatcher. A command dispatcher is a function hook that gets called for every command the user enters. It receives a single parameter, a delegate representing the command the user entered. In the WPF and WinForms tutorials, you use this function hook to marshal the commands to the right thread to be executed. Here’s the command dispatcher from the WPF tutorial:

The dispatcher.Invoke call looks kinda like the UIThread decorator from the Background Processing part of this series, doesn’t it?

Quick aside: I looked at using SyncContext here instead of Dispatcher, since I don’t care about propagating a return value back to the interactive console thread. However, SyncContext expects a SendOrPostDelegate, which expects a single object parameter. The delegate passed to the console hook function is an Action with no parameters. I could have built a wrapper function that took a single parameter which it would ignore, but I decided it wasn’t worth it. The more I look at it, the more I believe SyncContext is a good idea with a bad design.

I wrapped all the thread creation and command dispatching into a reusable helper class called InteractiveApp.

The code is pretty self explanatory. The constructor (__init__) creates the UI thread, starts it, waits for it to signal that it’s ready via an AutoResetEvent and then finally sets the command dispatcher. The UI thread creates and runs the WPF application, saves the dispatcher object as a field on the object, then signals that it’s ready. DispatchConsoleCommand is nearly identical to the earlier version, I’ve just made it an instance method instead of a stand-alone function. Finally, I define __getattr__ so that any operations invoked on InteractiveApp are passed thru to the contained WPF Application instance.

In my app.py file, I look to see if the module has been started directly or if it’s been imported into another module. If the module is run directly (aka ‘ipy app.py’) then the global __name__ variable will be ‘__main__’. In that case, we start the application up normally (i.e. without the interactive prompt) by just creating an Application then running it with a Window instance. Otherwise, we are importing this app into another module (typically, the interactive console), so we create an InteractiveApp instance and we create an easy to use run method that can create the instance of the main window.

If you want to run the app interactively, you simply import the app module and call run. Here’s a sample session where I iterate thru the items bound to the first list box. Of course, I can do a variety of other operations I can do such as manipulate the data or create new UI elements.

One small thing to keep in mind: if you exit the command prompt, the UI thread will also exit since it’s marked as a background thread. Also, it looks like you could shut the client down then call run again to restart it, but you can’t. If you shut the client down, the Run method in InteractiveApp.thread_start exits, resets the Command Dispatcher to nothing and the thread terminates. I could fix it so that you could run the app multiple times, but I find I typically only run the app once for a given session anyway.

Resolver One 1.3 Released

IronPython’s biggest customer is Resolver Systems, makers of Resolver One, “a familiar spreadsheet-like interface with the powerful Python programming language to give you a tool to analyse and present your data.” While I think they have a great product on pure merit - I’ve been impressed with their product since I was introduced to it at Lang.NET this year - I’m particularly interested in Resolver One as it’s written in IronPython. They use IPy not only as the embedded language exposed to end users but as the underlying implementation language as well.

Furthermore, these guys are heavily involved in the IPy community. Resolver developer Michael Foord is writing a book on IronPython and was our first Dynamic Language MVP. Michael’s Resolver colleague Jonathan Hartley did me a solid by taking my space at ØreDev. Even the CTO and Co-founder Giles Thomas is a regular blogger and speaker at events. Let me tell you, having guys this great in the community sure makes my job easier.

I just wanted to give the Resolver folks a shout out and say major congratulations on shipping a new version of their core Resolver One product. Michael has more info on this release as well as a glance forward with their plans for their next (IPy 2.0, woot!)

Background Processing Re-Revisited

OK, here’s the last word on this whole background processing / concurrency decorators thing. I went back and re-wrote the original decorators, but using the approach I used with the SyncContext version. I don’t want to rehash it again, here are the main points:

• Instead of using a property to retrieve the dispatcher, I get it via Application.Current.MainWindow.Dispatcher (checking to be sure Current and MainWindow aren’t null…err, None). This way, I pick up the dispatcher automatically rather than forcing a specific interface on the class with decorated methods. In fact, this approach should work with pure functions as well.
• Since I don’t have a convenient function like SetSynchronizationContext, I store the dispatcher in thread local storage for later use in calling back to the UI thread.
• Unlike the SyncContext version, this version propagates the return value of @UIThread decorated functions. I don’t propagate the return value of @BGThread functions – there’d be no point farming a task to a background thread then blocking the UI thread waiting for a response.

As usual, the code is on my SkyDrive. It includes both the SyncContext and Dispatcher version of the decorators.

Introducing IronPython Article

FYI, my Introducing IronPython article from the .NET Languages issue CoDe magazine is now available online in it’s entirety. Previously, only the introduction was available online. And while we’re on the subject, major thanks to the folks who at the CoDe magazine booth at PDC, who gave me several copies of that issue.

IronPython and WPF Background Processing Revisited

Yesterday, I blogged about using decorators to indicate if a given function should execute on the UI or background thread. While the solution works, I wrote “I’m thinking there might be a way to use SynchronizationContext to marshal it automatically, but I haven’t tried to figure that out yet.” I had some time this morning so I figured out how to use SynchronizationContext instead of the WPF dispatcher.

Leslie Sanford wrote a pretty good overview, but the short version is that SyncContext is an abstraction for concurrency management. It lets you write code that is ignorant of specific synchronization mechanisms in concurrency-aware managed frameworks like WinForms and WPF. For example, while my previous version worked fine, it was specific to WPF. If I wanted to provide similar functionality that worked with WinForms, I’d have to rewrite my decorators to use Control.Invoke. But if I port them over to use SyncContext, they would work with WinForms, WPF and any other library that plugs into SyncContext.

SyncContext abstracts away both initially obtaining the sync context as well as marshaling calls back to the UI thread. SyncContext provides a static property to access  current context, instead of a framework specific mechanism like accessing the Dispatcher property of the WPF Window class. Once you have a context, you can call Send or Post to marshal the call back to the UI thread (Send blocks the calling thread, Post doesn’t).

With that in mind, here’s the new version of BGThread and UIThread. Slightly more complex, but still pretty simple clocking in at just under 30 lines.

In the BGThread wrapper, I add the current SyncContext to the parameter tuple that I pass to the background thread. Once on the background thread, I set the current SyncContext to the last element of the the parameter tuple then call the decorated function with the remaining parameters. (for the non pythonic: args[:-1] is Python slicing syntax that means “all but the last element of args”). Using a try/finally block is probably overkill – I expect the current SyncContext to be either None or leftover garbage – but the urge to clean up after myself is apparently much stronger on the background thread than it is in say my office. :)

In the UIThread wrapper, I grab the current context and invoke the decorated method via the Send method. Like QueueUserWorkItem, SyncContext Send and Post only support a single parameter, so I use the same *args trick I described in my last post. (I changed the name to unpack in the code above for blog formatting purposes)

One major caveat about this approach is that there’s no way to return a value from a function decorated as UIThread. I understand why SyncContext.Post doesn’t return a value (it’s async) but SyncContext.Send is synchronous call, so why doesn’t it marshal the return value back to the calling thread? WPF’s Dispatcher.Invoke and WinForm’s Control.Invoke both return a value. I didn’t handle the return value in my original version of UIThread, but now that I’ve moved over to using SyncContext, I can’t. Not sure why the SyncContext is designed that way – seems like a design flaw to me. Since the return value won’t propagate, I sniff the result decorated function’s return value and raise a warning if it’s not None.

I’ve uploaded the SyncContext version to my SkyDrive in case you want the code for yourself. Note, I’ll thinking I’ll revise code this one more time – I want to rebuild the WPF version so that it propagates return values and picks up an dispatcher via Application.Current.MainWindow rather than having to have a dispatcher property on my class.

IronPython and WPF Part 4: Background Processing

Like many apps today, my WL Spaces photo viewer is a connected app. The various WL Spaces RSS feeds that drive the app can take a several seconds to download. Unless you like annoying your users, it’s a bad idea to lock up your user interface while you make you make synchronous network calls on your UI thread. Typically, this long running processing gets farmed out to a background thread which keeps the UI thread free to service the user events.

.NET provides a variety of mechanisms for doing long running processing on a background thread. For example you can create a new thread, you can queue a work item to the ThreadPoool or use the BackgroundWorker component. However, none of these are particularly pythonic, so I set out to see if I could leverage any of Python’s unique capabilities to make background processing as easy as possible. This is what I ended up with:

By using the cool decorators feature of Python, I’m able to declaratively indicate whether I want a given method to be executed on the UI thread or on a background thread. Doesn’t get much easier than that. Even better, the implementations of BGThread and UIThread are only about twenty lines of Python code combined!

Decorators kinda look like custom .NET attributes. However, where .NET attributes are passive (you have to ask for them explicitly), decorators act as an active modifier to the functions they are attached to. In that respect, they’re kind of like aspects. Certainly, I would consider which thread a given method executes on to be a cross-cutting concern.

The Completed function above is exactly the same as if I had written the following:

In C#, you can’t pass a function as a parameter to another function – you have to first wrap that function in a delegate. Python, like F#, directly supports higher-order functions. This lets you easily factor common aspectual code out into reusable functions then compose them with your business logic. The decorators have no knowledge of the functions they are attached to and the code that calls those functions are written in complete ignorance of the decorators. Python goes the extra mile beyond even F# by providing the ‘@’ syntax.

Here are the implementations of my the UIThread and BGThread decorators:

BGThread defines a wrapper function that queues a call to the decorated function to the .NET thread pool.  UIThread defines a wrapper that marshals the call to the UI thread by using a WPF Dispatcher. I’m thinking there might be a way to use SynchronizationContext to marshal it automatically, but I haven’t tried to figure that out yet. The above approach does require a dispatcher property hanging off the class, but that’s fairly trivial to implement and seems like a small price to pay to get declarative background thread processing.

A couple of quick implementation notes:

• The ‘*args’ syntax used in those methods above means “given me the rest of the positional arguments in a tuple”. Kinda like the C# params keyword. But that syntax also lets you pass a tuple of parameters to a function, and have them broken out into individual parameters. QueueUserWorkItem only supports passing a single object into the queued function, so I pass the tupled arguments to the argUnpacker method, which in turn untuples the arguments and calls the decorated function.
• The System assembly includes the single parameter Action<T> delegate. The current DLR provides Action delegates with zero, two and up to sixteen parameters. However, those are in a separate namespace (remember?) and IPy seems to have an issue with importing overloaded type names into the current scope. I could have used their namespace scoped name, but instead I redefined the version from System to be called Action1.
• To interop with .NET generic types, IPy uses the legal but rarely used Python syntax type[typeparam]. For example, to create a List of strings, you would say “List[str]()”. The type parameter is a tuple, so in UIThread I build a tuple of objects based on the number of arguments passed into wrapper (with the special case of a single type parameter using Action1 instead of Action).

I haven’t uploaded my WL Spaces Photo Viewer app because I keep making changes to it as I write this blog post series. However, for this post I built a simple demo app so I could focus on just the threading scenario. I’ve stuck the code for that demo up on my SkyDrive, so feel free to leverage it as you need.

IronPython and WPF Part 3: Data Binding

Here’s the short version of this post: data binding in WPF to IPy objects just works...mostly. However, I’m guessing you are much more interested in the long version.

Typically, data binding depends on reflection. For example, the following snippet of XAML defines a data bound list box where the title property of each object in the bound collection gets bound to the text property of a text block control. WPF would typically find the title property of the bound objects via reflection.

The problem is that IronPython objects don’t support reflection – or more accurately, reflection won’t give you the answer you’re expecting. Every IPy object does have a static type, but it implements Python’s dynamic type model. [1] Thus, if you reflect on the IPy object looking for the title property or field, you won’t find it. It might seem we’re in a bit of a bind (pun intended). However, WPF does provide an out:

“You can bind to public properties, sub-properties, as well as indexers of any common language runtime (CLR) object. The binding engine uses CLR reflection to get the values of the properties. Alternatively, objects that implement ICustomTypeDescriptor or have a registered TypeDescriptionProvider also work with the binding engine.”
WPF Binding Sources Overview, MSDN Library

Luckily for us, IronPython objects implement ICustomTypeDescriptor [2]. That snippet of XAML above? It’s straight from my photo viewing app. All I had to do was define the data template in the list box XAML then set the ItemsSource property of the list box instance.

As I said, it just works. However, I did hit one small snag – hence the “mostly” caveat above.

If you look at the top level WL Spaces photos feed, you’ll see that each item’s title starts with “Photo Album:”. Yet in the screenshot of my app, you’ll notice that I’ve stripped that redundant text out of the title. Typically, if you want to change the bound value during the binding process, you build an IValueConverter class. I needed two value conversions in my app, stripping “Photo Album:” for the album list box and converting a string URL into a BitmapImage for the image list box.

IronPython objects can inherit from a .NET interface, so there’s no problem building an IValueConverter. However, in order to use a custom IValueConverter from XAML, you need to declare it in XAML as a static resource. However, as you might imagine, dynamic IPy objects don’t work as static resources. So while I can define an IValueConverter in Python, I can’t create one from XAML.

There are a few possible solutions to this. The first is to build up the data template in code. If you do that, they you can programmatically add the converter to the binding. I was hopeful that I could define the data template in XAML then manipulate the binding, but there doesn’t appear to be any way to do that. Another option would be to build some type of generic IValueConverter class in C# that loads either an IPy based IValueConverter or embedded python conversion code. That’s problematic because those IPy object would need to be created in the right ScriptRuntime, and there’s no built-in way to access that. There are also a small set of XamlReader extensions such as XamlTypeMapper that might be able to provide the right hook into the XAML parsing to allow IronPython based conversion.

In the end, I took the easiest way out – I transformed the data to be bound before binding it. It’s cheating of sorts, but given the read-only nature of this app, it was the easiest thing to do. So the actual line of code to set listbox1’s ItemsSource looks like this:

I create a Python class for each RSS item in the feed, saving the stripped title and the album RSS URL as fields. It’s kinda annoying to basically be parsing the feed twice, but at least it’s not much code. Python’s list comprehension syntax makes creating a list of Albums from a list of RSS items a single line of code. I do something very similar for data binding the second list box:

Here I’m not only converting the raw data (adding “:thumbnail” at the end of the URL) but also changing the data type from string to BitmapImage. I’m binding to an image object in the second list box, but to do that I need a BitmapImage instead of a string.

This “convert the data first” approach feels like a hack to me. After I get this series of posts done, I am planning on going back and improving this sample. Hopefully, I can find a better approach to value conversions. Any gurus out there on XAML parsing, please feel free to drop me a line or leave me a comment.

[1] you can access the underlying CLR type for any Python type via the clr.GetClrType method. You an also check out the CreateNewType method from NewTypeMaker.cs

[2] I spent the better part of an afternoon trying to make TypeDescriptionProviders work before Dino pointed out that we already support ICustomTypeDescriptor in Python objects. I didn’t realize at first because I had a case sensitivity bug in my original prototype code - it turns out that “Title” != “title”.

IronPython and WPF Part 2: Loading XAML

If we’re going to build a WPF app, we’re going to want to be able to load some XAML. Sure, you can programmatically build up your UI, but WPF and more importantly WPF tools like Expression Blend are designed to work with XAML. Luckily, loading XAML is fairly easy:

We simply open the filename provided and use XamlReader to build out the corresponding WPF object graph. Note, this is very different from the XAML approach used by C#/VB or even by IronPythonStudio. In those scenarios, the XAML is compiled into a binary format (BAML) and embedded in the compiled assembly. For my TechieWife Photo viewer, it’s all script so there’s neither a XAML to BAML compile step nor a compiled assembly to embed the BAML into, so we’re just loading raw XAML.

Since we’re using raw XAML, there are additional rules we need to follow. First, when using compiled XAML, we can specify the name of the event handler in the XAML directly. For XamlReader, that’s no allowed since there’s no C#/VB class associated with the XAML. Speaking of class, you can’t specify x:Class either. Finally, anywhere you want to use a static resource, as far as I can tell those need to be compiled in a static language. I think you could build one in C#, add a reference to that assembly via clr.AddReference, then use it from XAML and it should just work. However, since I’m trying to stick to IronPython exclusively, I didn’t try that scenario out. 

Since you can’t specify the event handlers in XAML loaded by XamlReader, you have to bind the event handlers in code. There are two listboxes in my photo viewing app, and I want to capture the SelectionChanged event of both of them. Binding event handlers in IronPython code uses the same += syntax as C# uses.

My win1.xaml file has a Window type instance as the root. You don’t need to be a deep WPF expert to realize that the WPF Window doesn’t have listbox1 or listbox2 properties. Yet, in the code snippet above, I was able to say win1.listbox1 and get back the WPF ListBox element with that name. Cool trick, eh? Well, I can’t take credit for it – I copied the code from our Silverlight integration for dynamic languages. Unfortunately, this code has to be written in C# code, but it is the only C# code in my whole solution (and it’s reusable!)

GetBoundMember is kinda like Python’s __getattr__ or Ruby’s method_missing. Of course, it doesn’t work with C#, but it does lets us trap dynamic member resolution when calling a C# object from a DLR language. Srivatsn has a great write up on using GetBoundMember and the four other special methods you can use to make your CLR objects act more dynamic.

In this case, if the standard reflection-based member name resolution fails, we try calling FrameworkElement’s FindName method to see if there’s a corresponding control with the provided name. So win.listbox1 is the equivalent to win.FindName(‘listbox1’), but with less code and a much more pythonic feel.

You’ll notice that we’re attaching this GetBoundMember method to FrameworkElement as an extension method. It’s kinda cool that we can inject a new method into an existing class to provides dynamic behavior and it all works seamlessly from Python. However, DLR uses a different mechanism to locate and bind extension methods than C# or VB. Those languages use ExtensionAttribute to mark extension methods and the assemblies and classes that contain them. However, that approach forces you to examine ever single class in marked assemblies and every single method in marked classes. Examining every class and method is no big deal to do at compile time, but it would be a significant perf issue at runtime. By using ExtensionType attribute, the DLR only has to look at assembly attributes in order to bind extension methods.

Once you’ve got the compiled FrameworkElementExtension assembly, you just need to load it via clr.AddReference. I called the assembly Devhawk.Scripting.Wpf and I load it automatically in my wpy.py module. So if you’re building a WPF app in IronPython, you can simply “import wpy” and you get the GetBoundMember extension method, the LoadXaml function, and a bunch of WPF related namespaces imported into the wpf scope. That way, you can write wpf.Button() instead of System.Windows.Control.Button() to programmatically create a new button.

IronPython and WPF Part 1: Introduction

I decided to start my IronPython and “veritable universe of cool technologies” examples with WPF. I figured that since we already have Silverlight support, there might be some overlap (there was). Futhermore, after seeing BabySmash on Surface I’m jonesing to build a Surface app of my own. Getting vanilla WPF working with IPy seems like a smart step before trying to build a Surface WPF app with IPy.

WPF is all about cool graphics, so I decided to build a photo viewing app. Kinda boring, I know. But it turns out my wife has posted hundreds of photos to her WL Space, and WL Spaces provides convenient RSS feeds of both photo albums as well as photos in specific albums. So I built out a simple WPF based photo viewer for my wife’s WL Space photos in IronPython.

As you can see, I'm not quitting my job to go pursue a career in design anytime soon. But hey, the point is demonstrate building a WPF app in IPy, not to be a great designer. Plus, don’t those cute kids make up for the ugliness of the app?

Turns out building this app in IPy was fairly straightforward, with a few pitfalls. I wasted half a day digging thru data binding before realized that data binding against IPy objects works out of the box - but only if you type the case of the property correctly (Title != title). Also, I couldn’t make TypeConverters work the way I wanted, but python list comprehensions made it enough to transform the feed data before binding it to the UI. That approach worked great for this scenario but maybe not so much for others. (I’ve got feelers out to the WPF data binding wonks, so maybe there’s still hope for type converters)

Over the next several posts, I’m going to show you all the code for this app. It’s pretty small, only about 50 lines of app-specific python code + 50 lines of XAML to describe the window. There’s also some reusable code – 50 lines of WPF module code (mostly stolen from avalon.py in the IPy tutorial), 200 lines of xml2py code which I’ve discussed before and a very small C# based assembly to make accessing WPF elements by name very pythonic.

IronPython and <Insert MSFT Technology Here>

Now that PDC08 is in my rear view mirror, I’m back to doing IronPython stuff. One of the things I’m looking at is making IronPython work with a variety of Microsoft technologies. Given the usage of dynamic languages in web scenarios, most of our focus to date has been on using Iron languages in Silverlight. Being able to program the browser with the same language you program the server is a fairly compelling scenario. We’re also starting to see new progress on ASP.NET support for Iron languages.

But those are only two out of a veritable universe of cool technologies. Now that I’m done with PDC, I can start to explore some of the others. Some ideas include:

• IPy and WPF
• IPy and Surface
• IPy and XNA (desktop only – Xbox and Zune use the Compact Framework with doesn’t support DLR)
• IPy and WCF
• IPy and WF

Any other suggestions? Please leave them in the comments.

PDC08 Day –2

I’m in Los Angeles for PDC. Now that all the prep work is done, I’m going to to try and get back to regular blogging and I figure that daily reports from PDC is as good a way as any to get started.

I’m in town early for two reasons. We’re doing some last minute dry runs tomorrow afternoon and I wanted to make sure I was in town in case there was any other last minute stuff to do. Additionally, there’s a Code Camp in SoCal this weekend, so I volunteered to do my Pumping Iron talk.

The talk went pretty well – the room was mostly full (though small) and many folks stayed as much as 30 minutes over to ask questions. The Code Camp is being held at USC, my alma mater, so it was kind of strange to be standing in the front of the classroom in Vivian Hall rather than taking notes at the back. I made a trip over to the campus bookstore (thought I kept calling it the ‘company store’) for a new T-shirt for me and some SC gear for the kids.

Hung out most of the day with Mike VIncent whom I’ve gotten to know over the past couple of years thru his IASA and INETA involvement. He’s doing a talk on Dynamic Languages and the DLR tomorrow at Code Camp, so I’m looking forward to that. Also spent time with Chris Smith (F# SDET), Dustin Campbell (VB PM), Charlie Calvert (C# PM) and ran into DonXML as I was heading out.

I headed out early because I’m going up over the hill to Burbank tonight to see some of my old college / LA buddies, drink some beer and watch the SC game. With our early loss to Oregon State, we can’t make the Rose Bowl, much less the BCS Championship, without help. If win out but don’t get help, we’ll still probably get a BCS at-large bid. For a team that’s been in the championship hunt for the past five years, it’s disappointing, but it’s also like “no pressure” – at least for me, an non-obsessive alumni fan. (I typically save my obsession for Capitals hockey, but even that takes a back seat to the presidential election for the next 11 days.)

I forget who said it, but someone said today that “Los Angeles was like paradise 50 years ago”. Truer words were rarely ever said. It’s nice to be on campus and see friends and all that, but I can’t wait to go home. Why does PDC always have to be in LA? Mike? Well, at least it’s not on fire this time.

IronPython 2.0 Release Candidate

I’ve been so busy with last minute PDC prep that I totally missed that my teammates shipped the IronPython 2.0 Release Candidate. Awesome work guys!

If you’re an IPy user, this is your last chance to hammer on the release and raise any issues to our attention before we ship. If you find anything, please file a bug and report it on the mailing list right away. For those who haven’t seen it, Dave from our test team has a handy guide for reporting bugs, including a link to the bug template.

Next step, RTM!

The Fifth Assembly

As I’ve written previously, we’ve had a few challenges recently with name collisions in the DLR. In that post, I described how we had solved – or thought we solved, as it turned out – the problem with ExtensionAttribute name collisions between Microsoft.Scripting.Core.dll and System.dll.

However, as it turns out, having lots of copies of the same type didn’t solve the problem. Since ExtensionAttribute is a known type to the C# 3.0 compiler, it has to choose one of the multiple copies that are in the project. We thought that given a choice, it would favor the System.Core version. However, what folks discovered after we released Beta 5 is that C# 3.0 will choose the first copy of ExtensionAttribute that it finds. So if you have System.Core.dll and IronPython referenced in your project, and you define your own extension methods, the compile fails if the C# 3.0 compiler finds one of the IronPython or DLR private copies of ExtensionAttribute before the public copy in System.Core.

Furthermore, there doesn’t seem to be any way to set the reference order in MSBuild files. I’ve never dug deep into the MSBuild file format, but changing the order of the references in the csproj file didn’t seem to effect the order the references were passed to the C# compiler. I’m guessing we might be able to change this by fiddling with the ResolveAssemblyReference task, but we didn’t want to force low level MSBuild file hacking on our user base.

We looked at a variety of other solutions, including rewriting the IL after compilation to change the namespace of the ExtensionAttribute. However, we had trouble making that solution work and besides, changing the ExtensionAttribute namespace would have broken anyone using the existing DLR or IPy extension methods. So instead, we went with a different solution that we like to refer to as “The Fifth Assembly” around the office.

In IronPython 2.0 Beta 5, there were four DLLs that implement IronPython: IronPython.dll, IronPython.Modules.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll. In our RC1 release, we’ve added “The Fifth Assembly”: Microsoft.Scripting.ExtensionAttribute.dll. As you might guess from its name, it has only a single public type: ExtensionAttribute. By having ExtensionAttribute in its own dedicated assembly, we can avoid the type collision at compile time by not referencing both System.Core.dll and Microsoft.Scripting.ExtensionAttribute.dll in the same project.

In IronPython, we reference the ExtensionAttribute assembly because we use the C# 3.0 complier but IPy has to be able to run on .NET Framework 2.0 SP1. However, projects that embed IronPython in a .NET 3.5 project (aka C# 3.0 or VB 9.0) will reference System.Core instead. The only reason why you would explicitly use the ExtensionAttribute assembly was that if you, like us, wanted to build your app with .NET 3.5, use extension methods but still be compatible with .NET 2.0 SP1. We’re guessing there aren’t many of our customers doing that, but if you are, explicitly referencing ExtensionAttribute will work just as it does for compiling IronPython itself.

It’s important to remember two things about the Fifth Assembly:

1. Never reference System.Core and Microsoft.Scripting.ExtensionAttribute in the same project.
2. Always deploy Microsoft.Scripting.ExtensionAttribute.dll as part of any solution that embeds IronPython (or IronRuby or vanilla DLR for that matter), even if you don’t reference it explicitly within your project.

This change is public in the source code as of change set 42076 and will also be in the nearly-ready RC1 release of IronPython 2.0. If you’ve got any questions or <shudder> find any more issues with this solution, please let us know right away.

DLR Namespace Change Fire Drill

[Ed. Note – This long post is about changes we made in the DLR to avoid type collisions with System.Core. The short version of this post is “You can safely ignore the CS1685 warning you’ll get if you embed IPy 2.0 Beta 5 or later in your C# 3.0 application”. The long version below has the gory details of what we did, why we did it, a little about how extension methods work and why you can ignore warning CS1685.]

[Author note – I don’t really have an editor.]

Between Beta 3 and Beta 4 of IronPython 2.0, the DLR team made a very significant change to Microsoft.Scripting.Core.dll. As JB noticed, the DLR expression trees have merged with the LINQ expression trees. As part of this effort, they moved the newly merged expression tree types in Microsoft.Scripting.Core.dll into the System.Linq.Expressions namespace – the same namespace where those types live in System.Core.dll in .NET Framework 3.5.

This change caused quite a few issues with our users. Basically, because of the expression tree merge and the namespace change, the beta 4 version of Microsoft.Scripting.Core.dll had type collisions with System.Core.dll all over the place, making it hard or impossible to use them together. If you're using C# you could get around these type collisions by using an assembly reference alias. However, assembly reference aliases aren’t supported for Web Sites projects or in Visual Basic.

To fix this, we're changing the top level namespace in Microsoft.Scripting.Core.dll from System to Microsoft. We're not going back to the namespaces as they were in Beta 3 - for example, DLR expression trees were originally in the Microsoft.Scripting.Ast namespace but now they'll be in Microsoft.Linq.Expressions. We don't think this change will be much of an issue because most people don't use types from Microsoft.Scripting.Core.dll directly. Unless you're building your own DLR language, this namespace change shouldn't affect you at all except to solve the type collision problem.

However, we did hit a small snag.

The LINQ expression tree code, having been written for .NET 3.5, is a heavy user of extension methods. This means IronPython is now also a heavy user of extension methods. However, unlike LINQ, IronPython has to run against .NET 2.0 SP1. That means we can't reference System.Core.dll in IPy or DLR. If you try to compile C# code with extension methods but without a reference System.Core.dll, you get a compiler error complaining that it can't find the required ExtensionAttribute type, which is defined in System.Core.dll.

This might appear to be an unsolvable problem, but it turns out the C# compiler doesn't actually care where the ExtensionAttribute type comes from. You can actually define your own copy of ExtensionAttribute (in the right namespace) and C# will happily compile extension methods without complaint. Furthermore, ExtensionAttribute is only used as a marker - there's no real code in it - so implementing your own copy is trivial. In the DLR source, you'll find they have defined their own copy of ExtensionAttribute so they can use extension methods and remain .NET 2.0 SP1 compatible. Since we were using them in Microsoft.Scripting.Core.dll, we started using extension methods in Microsoft.Scripting.dll and IronPython.dll as well.

If you'll recall back to the start of this post, we're changing namespaces in order to eliminate type collisions. The snag we hit was that we couldn’t change the ExtensionAttribute namespace without breaking all the extension methods. But we couldn’t leave it the same without having a type collision with the ExtensionAttribute defined in System.Core.dll. If you had a project including both copies of ExtensionAttribute, C# would generate a multiple type definition warning and VB would generate a multiple type definition error.

We've looked at a several possible solutions to this. One idea was to ship two completely different sets of binaries - one for .NET 2.0 and one for .NET 3.5. But the upgrade story for that stinks - you want to upgrade your app to .NET 3.5 you have to swap out all your IPy and DLR dlls. Yuck. We considered having separate copies of just Microsoft.Scripting.Core.dll - one defining ExtensionAttribute and the other linked to System.Core.dll and using TypeForwardedTo - but since the assemblies are strongly typed they'd have to same exact version number in order to be swappable. Double yuck.

In the end, we decided to put an internal copy of ExtensionAttribute in each assembly that needs it. As previously indicated, that's IronPython.dll and Microsoft.Scripting.dll as well as making the copy already in Microsoft.Scripting.Core.dll internal. For IronRuby fans reading this, we also added a copy of ExtensionAttribute to IronRuby.dll and IronRuby.Libraries.Scanner.dll as well.

It seems counterintuitive, doesn't it? To solve a multiple type definition problem, we defined even more copies of the type in question.

The key thing is that all these copies of ExtensionAttribute (except the one in System.Core.dll) are internal rather than public types. If you build a VB app that references System.Core.dll and Microsoft.Scripting.Core.dll (beta 4), you end up with multiple public copies of ExtensionAttribute and are rewarded with a VB compiler error. However, as long as there's only one public copy of ExtensionAttribute - regardless of the number of internal copies of that type - VB is happy. So if you're building a VB app against Microsoft.Scripting.Core.dll (beta 5) and System.Core, you should be golden.

In C# 3.0, on the other hand, continues to throw a warning. If ExtensionAttribute was a user-defined type, we'd be fine. However, since Extension attribute is a "predefined system type", you get C# warning 1685 even though the copies of ExtensionAttribute are all internal. Furthermore, since there are multiple internal copies of ExtensionAttribute in the IPy dlls, you'll get this warning even if you're not referencing System.Core. It seems here that C# 3.0 considers ExtensionAttribute a predefined system type while VB doesn't.

I realize that always having a warning in C# 3.0 - even if you're not referencing System.Core.dll - doesn't feel particularly clean. Given our desire to support both .NET v2 and v3.5 with the same binaries, it was the only choice. Remember that ExtensionAttribute has literally no code and is only used to signal the compiler for extension methods, so we decided it was fairly ignorable as warnings go.

If you're willing to compile from source yourself, it's fairly easy to build a set of binaries for a specific version .NET that doesn't have the warning. If you're building for v3.5, you need to remove Extension.cs from the three projects that have a copy of it (Microsoft.Scripting.Core, Microsoft.Scripting, IronPython) and add a reference to System.Core.dll. If you're building for v2.0, remove the Extension.cs from Microsoft.Scripting and IronPython then change the visibility in the Microsoft.Scripting.Core version from internal to public. Note, we treat warnings as errors in IPy, but we did add CS1685 to the list of WarningsNotAsErrors so the code still compiles. Of course, if you’re defining a framework specific version, you won’t get the warning anyway.

As usual, we appreciate all feedback from our community so hammer on this build as much as you can - esp. if you've been having type conflict errors with Beta 4. As I said in an earlier post, this is our last planned beta, so now’s the time put it thru the paces to make sure there’s nothing blocking you before we get to 2.0.

Finally, major props to Curt...aka IRON CURT...for driving these dev changes and putting up with the constant barrage of "where are we now?" status requests from yours truly. I'm sure he now regrets sitting across the hall within easy earshot.

IronPython Beta 5

Dave just posted the announcement to the IPy mailing list, Beta 5 has been released. You can download the release from CodePlex. Like Beta 4, this release is available in MSI or ZIP format, with the MSI including the Python Standard Library as an option.

As per the release notes, this is our last planned beta for 2.0 – our next planned release is RC1. So if you’ve been holding off on upgrading to the latest betas or to 2.0 in general, now is the time to upgrade if you want us to fix your bug before we ship 2.0.

Afternoon Coffee 174

You know, this gets pretty long when I go a week between morning coffee posts.

Dynamic Language Stuff

• There's a new version of the DLR Hosting spec. One of the biggest changes is the removal of DLR's pre-canned knowledge of IPy and IRuby. Now, you use app.config to specify DLR languages and map them to extensions. Sesh has the details and a sample. If you only care about hosting IPy, you can use our language specific hosting API which doesn't require app.config.
• Jimmy pushed out a new version of the Silverlight Dynamic Languages SDK. He's on the lookout for a better name, one of his commenters suggested "SilverWhisper Kit".
• Srivatsn shows how to pass Python keyword arguments to a C# method using ParamDictionaryAttribute, similar to how C# supports the params keyword via the ParamArrayAttribute. Unfortunately, there's no syntactic sugar for calling param dictionary methods from C#. I wonder, do most people like or dislike the idea of language level support for keyword arguments in C#? I'm guessing most would be against, though I think it's a cool idea.
• Maybe it's me, but I've noticed an uptick in twitter posts containing "ironpython" of late. For example, via Summize I discovered Joseph Jude's blog finally{} where he has posted a time tracking tool he built in IPy as well as his initial IPy learnings.
• Michael Foord, as usual, has been generating quite a bit of good IPy content. He did a tutorial on IPy at PyCon UK,  his book is almost done, he has some interesting thoughts on dynamic language performance and even admits that he likes C#, though not as much as he likes Python.
• Vernon Cole announced that the adodbapi project now supports IronPython 2.0 as well as CPython. This library is a DB-API 2.0 compliant interface for any database that support ADO (note, that's native ADO not ADO.NET). I love to see Python projects with explicit support for IPy.
• My boss - who apparently will be called "Rem" from now on in the office - saves you "gazillions" by explaining non-positional negotiating strategy and draws the comparison to agile software development. Not exactly related to dynamic languages, but I'd like to see him further explore the relationship super-optimal, non-positional, significantly-hyphenated negotiation and software design. (OK, I made up "significantly-hyphenated")

Other Stuff

• Don Syme blogs about an update to the F# CTP, a mere week after the original release. One week? That's more often than even IPy releases. I can't wait to see what they ship in next week's release! :) Seriously, I hope they can keep the release sprints short, but every week would be a bit crazy!
• Speaking of F#, Matt Podwysocki updates FsTest for the F# CTP and posts about Extension Everything in F#. Unlike C#, which only supports extension methods, F# also supports extensions properties, static methods and events, though like Matt I can't think of a good use for extension events.
• Still speaking about F#, Andrew Kennedy has a three part series on the new units of measure feature of F#. If you were going to use F# to build the physics engine of a game, I would suspect UoM would be extremely useful. (via Don Syme)
• Oh look, Chris Smith built an F# version of artillery game that uses Units of Measure for the physics code. I'll bet UoM was extremely useful. :)
• Talking about Live Mesh at TechEd Australia - where much to my surprise frankly they were demoing Live Mesh Apps - I pointed out to Scott Hanselman that Mesh is running an embedded CoreCLR (aka the same CLR from Silverlight 2). Scott went poking around and posted what he discovered. Looking forward to finding out what he digs up on using CoreCLR outside the browser.
• Speaking of Scott, I need to set up a family video conference solution like Scott's before my next trip.
• Congrats to Glenn Block and the MEF team for their initial CodePlex source drop! I've been hearing about this possibility since Glenn joined the team, so I'm really excited to see it happen. I need to take a look at it in detail (in my copious spare time) because I want to find out how to make it work with IPy.
• Bart de Smet has a whole series (starting here) on Dynamic Expression Trees. However, given that he specifically writes "This blog series is not about DLR itself" makes it seem pretty conceptual to me. Why not talk about DLR expression trees instead Bart?
• I'm sure you noticed ASP.NET MVC preview 5 dropped last week. I really liked Brad Wilson's discussion of the new view engine design.
• Tomas Restrepo has started publishing his source code on GitHub. Personally, I haven't published any source code lately but I am using Git for all of my non IPy core work (which is stored in TFS). Like Tomas, I'm still getting the hang of Git but I'm really digging it's speed, it's branching and the fact that there's zero infrastructure requirements. SVN provides the lightweight svnserve, but Git is even lighter weight than that.
• I liked Steve Yegge's post on typing. I am a touch typer, but I doubt I type 70 words a minutes. I do know where the number keys are without looking though, so I guess that's pretty good. I remember seeing Chris Anderson demo Avalon WPF long before it was public and being impressed at how fast he could type.

Morning Coffee 173

I'm on my way out the door for New Zealand and Australia, but I wanted to push out a few things.

• F# August September CTP is out! Don Syme has the announcement, Jomo Fisher has the link roundup and details are on the brand-spanking-new MSDN F# Dev Center. Major congrats to the F# team. I've been running a pre-release version of these bits, and they are a huge step forward if you're an F# developer.
• I've got an article on IronPython in the latest issue of CoDe magazine. Also check out Brad Wilson's IronRuby article, Ted Neward's F# article and Neil Ford's Polygot Programming article.
• Via Michael Foord I discovered that IronPython tester Dave Fugate is back on the blog. He starts with a couple of posts about measuring IronPython performance.
• Speaking of blogging teammates, I think the dynamic languages team has the highest percentage of bloggers in any group at MSFT. All four Program Managers (Dave (lead), John, Jimmy and me), four of five developers (Shri (lead), Dino, Curt and Oleg) and all three Testers (Jim, Dave and Srivatsn). The only non blogger right now is Tomas - who at least has a home page - and  the lead tester which is an open position right now. 11 bloggers out of 12 team members equals 91.67% team blogger coverage.
• I was really impressed with Newspeak when I saw it at Lang.NET so I'm very excited to see they have a new website. No public bits yet, but I like the part where they point out Newspeak "can be implemented independently of Squeak, Smalltalk or any particular VM or IDE". How about implementing a version on DLR guys?
• Maurice de Beijer shows off embedding IronPython inside a WF application. Kinda cool, but he's primarily showing off implementing a CLR interface in IPy. How about a WF activity that execute arbitrary IPy code. That would be cool. (via IronPython URLs)
• Ironclad has reached their 0.5 milestone, being able to import numpy from IronPython. BTW, guys - I'm not sure commenting out one line that appears to be unreferenced qualifies as a "monstrous caveat". Congrats guys! (via IronPython URLs)