The Fifth Assembly

As I’ve written previously, we’ve had a few challenges recently with name collisions in the DLR. In that post, I described how we had solved – or thought we solved, as it turned out – the problem with ExtensionAttribute name collisions between Microsoft.Scripting.Core.dll and System.dll.

However, as it turns out, having lots of copies of the same type didn’t solve the problem. Since ExtensionAttribute is a known type to the C# 3.0 compiler, it has to choose one of the multiple copies that are in the project. We thought that given a choice, it would favor the System.Core version. However, what folks discovered after we released Beta 5 is that C# 3.0 will choose the first copy of ExtensionAttribute that it finds. So if you have System.Core.dll and IronPython referenced in your project, and you define your own extension methods, the compile fails if the C# 3.0 compiler finds one of the IronPython or DLR private copies of ExtensionAttribute before the public copy in System.Core.

Furthermore, there doesn’t seem to be any way to set the reference order in MSBuild files. I’ve never dug deep into the MSBuild file format, but changing the order of the references in the csproj file didn’t seem to effect the order the references were passed to the C# compiler. I’m guessing we might be able to change this by fiddling with the ResolveAssemblyReference task, but we didn’t want to force low level MSBuild file hacking on our user base.

We looked at a variety of other solutions, including rewriting the IL after compilation to change the namespace of the ExtensionAttribute. However, we had trouble making that solution work and besides, changing the ExtensionAttribute namespace would have broken anyone using the existing DLR or IPy extension methods. So instead, we went with a different solution that we like to refer to as “The Fifth Assembly” around the office.

In IronPython 2.0 Beta 5, there were four DLLs that implement IronPython: IronPython.dll, IronPython.Modules.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll. In our RC1 release, we’ve added “The Fifth Assembly”: Microsoft.Scripting.ExtensionAttribute.dll. As you might guess from its name, it has only a single public type: ExtensionAttribute. By having ExtensionAttribute in its own dedicated assembly, we can avoid the type collision at compile time by not referencing both System.Core.dll and Microsoft.Scripting.ExtensionAttribute.dll in the same project.

In IronPython, we reference the ExtensionAttribute assembly because we use the C# 3.0 complier but IPy has to be able to run on .NET Framework 2.0 SP1. However, projects that embed IronPython in a .NET 3.5 project (aka C# 3.0 or VB 9.0) will reference System.Core instead. The only reason why you would explicitly use the ExtensionAttribute assembly was that if you, like us, wanted to build your app with .NET 3.5, use extension methods but still be compatible with .NET 2.0 SP1. We’re guessing there aren’t many of our customers doing that, but if you are, explicitly referencing ExtensionAttribute will work just as it does for compiling IronPython itself.

It’s important to remember two things about the Fifth Assembly:

1. Never reference System.Core and Microsoft.Scripting.ExtensionAttribute in the same project.
2. Always deploy Microsoft.Scripting.ExtensionAttribute.dll as part of any solution that embeds IronPython (or IronRuby or vanilla DLR for that matter), even if you don’t reference it explicitly within your project.

This change is public in the source code as of change set 42076 and will also be in the nearly-ready RC1 release of IronPython 2.0. If you’ve got any questions or <shudder> find any more issues with this solution, please let us know right away.

The PDC Prep Death March – It’s Almost Over

So it’s been just over a month since my last post, and I think it’s safe to say it’s been one of the busiest of my career to date. If you’ve been following my Twitter stream, you already know that I’ve spent the last two weeks in PDC dry runs – we went thru almost every session in the track, reviewing content and giving feedback to the speakers. Some were very good (as I twittered at the time, Ed Pinto’s Building WCF Services with WF blew my mind). Others, needed more work, but I think will be great by PDC. I got into several disagreements about the best way to present content, had to raise my voice once, and called some speakers “a little creepy”. (No, I’m not telling you which sessions those were.)

After 9 days of dry runs, I spent Friday with Jason Zander, General Manager for Visual Studio, again reviewing almost all the decks with him and some marketing folks. Frankly, spending that much time with my boss’s boss’s boss’s boss is a bit intimidating, but Jason’s been great and I’m guessing the visibility will be great for my career. I mean, come review time, he won’t be saying “Harry who?” I’ve also gotten to meet people from far and wide across my division, which has been great since I’m still new over here.

Unfortunately,  I can’t talk much about what I’ve seen, since as you know most of it is new and being revealed at PDC for the first time publicly. For example, I can tell you Ed’s talk was frakking awesome, but I can’t tell you why I think it’s frakking awesome, yet. But I’m queuing up some posts now that I will publish later once the sessions are public.

Tomorrow, I’m back to my “day job” as the IronPython PM. My teammates have been soldering on in my absence – the one time I was in my office in the past two weeks I joked that no one would recognize me since I’ve been gone so long. We’re coming up on the final release of IronPython 2.0, and my exclusive focus on PDC has left me a mountain of work to do here in the final stretch.

In addition to my teammates, I need to give a quick shout out to Shoshanna Budzianowski and Mike Swanson, without whom I’m not sure I would have survived the past two weeks. Mike is the PDC content owner, so as hard as the past two weeks have been for me, I’m sure they’re worse for him. Shoshanna is the track owner for the “Tools and Languages” track (that’s what the TL in the session codes stands for) and I’ve been the main representative for the VS group in the track. I don’t know if she’s ever done something like PDC before, but she’s awesome. 

I leave Friday for southern California. I’m going down early to see a few friends and to do my Pumping Iron talk at the SoCal Code Camp next Saturday. That should be fun since it’s being held at my alma mater. Then Monday starts PDC proper. If you’re going to PDC, some find me in the PDC Lounges. I’ll be spending the vast majority of my time there, since I’ve seen all the content in my track already!

DLR Namespace Change Fire Drill

[Ed. Note – This long post is about changes we made in the DLR to avoid type collisions with System.Core. The short version of this post is “You can safely ignore the CS1685 warning you’ll get if you embed IPy 2.0 Beta 5 or later in your C# 3.0 application”. The long version below has the gory details of what we did, why we did it, a little about how extension methods work and why you can ignore warning CS1685.]

[Author note – I don’t really have an editor.]

Between Beta 3 and Beta 4 of IronPython 2.0, the DLR team made a very significant change to Microsoft.Scripting.Core.dll. As JB noticed, the DLR expression trees have merged with the LINQ expression trees. As part of this effort, they moved the newly merged expression tree types in Microsoft.Scripting.Core.dll into the System.Linq.Expressions namespace – the same namespace where those types live in System.Core.dll in .NET Framework 3.5.

This change caused quite a few issues with our users. Basically, because of the expression tree merge and the namespace change, the beta 4 version of Microsoft.Scripting.Core.dll had type collisions with System.Core.dll all over the place, making it hard or impossible to use them together. If you're using C# you could get around these type collisions by using an assembly reference alias. However, assembly reference aliases aren’t supported for Web Sites projects or in Visual Basic.

To fix this, we're changing the top level namespace in Microsoft.Scripting.Core.dll from System to Microsoft. We're not going back to the namespaces as they were in Beta 3 - for example, DLR expression trees were originally in the Microsoft.Scripting.Ast namespace but now they'll be in Microsoft.Linq.Expressions. We don't think this change will be much of an issue because most people don't use types from Microsoft.Scripting.Core.dll directly. Unless you're building your own DLR language, this namespace change shouldn't affect you at all except to solve the type collision problem.

However, we did hit a small snag.

The LINQ expression tree code, having been written for .NET 3.5, is a heavy user of extension methods. This means IronPython is now also a heavy user of extension methods. However, unlike LINQ, IronPython has to run against .NET 2.0 SP1. That means we can't reference System.Core.dll in IPy or DLR. If you try to compile C# code with extension methods but without a reference System.Core.dll, you get a compiler error complaining that it can't find the required ExtensionAttribute type, which is defined in System.Core.dll.

This might appear to be an unsolvable problem, but it turns out the C# compiler doesn't actually care where the ExtensionAttribute type comes from. You can actually define your own copy of ExtensionAttribute (in the right namespace) and C# will happily compile extension methods without complaint. Furthermore, ExtensionAttribute is only used as a marker - there's no real code in it - so implementing your own copy is trivial. In the DLR source, you'll find they have defined their own copy of ExtensionAttribute so they can use extension methods and remain .NET 2.0 SP1 compatible. Since we were using them in Microsoft.Scripting.Core.dll, we started using extension methods in Microsoft.Scripting.dll and IronPython.dll as well.

If you'll recall back to the start of this post, we're changing namespaces in order to eliminate type collisions. The snag we hit was that we couldn’t change the ExtensionAttribute namespace without breaking all the extension methods. But we couldn’t leave it the same without having a type collision with the ExtensionAttribute defined in System.Core.dll. If you had a project including both copies of ExtensionAttribute, C# would generate a multiple type definition warning and VB would generate a multiple type definition error.

We've looked at a several possible solutions to this. One idea was to ship two completely different sets of binaries - one for .NET 2.0 and one for .NET 3.5. But the upgrade story for that stinks - you want to upgrade your app to .NET 3.5 you have to swap out all your IPy and DLR dlls. Yuck. We considered having separate copies of just Microsoft.Scripting.Core.dll - one defining ExtensionAttribute and the other linked to System.Core.dll and using TypeForwardedTo - but since the assemblies are strongly typed they'd have to same exact version number in order to be swappable. Double yuck.

In the end, we decided to put an internal copy of ExtensionAttribute in each assembly that needs it. As previously indicated, that's IronPython.dll and Microsoft.Scripting.dll as well as making the copy already in Microsoft.Scripting.Core.dll internal. For IronRuby fans reading this, we also added a copy of ExtensionAttribute to IronRuby.dll and IronRuby.Libraries.Scanner.dll as well.

It seems counterintuitive, doesn't it? To solve a multiple type definition problem, we defined even more copies of the type in question.

The key thing is that all these copies of ExtensionAttribute (except the one in System.Core.dll) are internal rather than public types. If you build a VB app that references System.Core.dll and Microsoft.Scripting.Core.dll (beta 4), you end up with multiple public copies of ExtensionAttribute and are rewarded with a VB compiler error. However, as long as there's only one public copy of ExtensionAttribute - regardless of the number of internal copies of that type - VB is happy. So if you're building a VB app against Microsoft.Scripting.Core.dll (beta 5) and System.Core, you should be golden.

In C# 3.0, on the other hand, continues to throw a warning. If ExtensionAttribute was a user-defined type, we'd be fine. However, since Extension attribute is a "predefined system type", you get C# warning 1685 even though the copies of ExtensionAttribute are all internal. Furthermore, since there are multiple internal copies of ExtensionAttribute in the IPy dlls, you'll get this warning even if you're not referencing System.Core. It seems here that C# 3.0 considers ExtensionAttribute a predefined system type while VB doesn't.

I realize that always having a warning in C# 3.0 - even if you're not referencing System.Core.dll - doesn't feel particularly clean. Given our desire to support both .NET v2 and v3.5 with the same binaries, it was the only choice. Remember that ExtensionAttribute has literally no code and is only used to signal the compiler for extension methods, so we decided it was fairly ignorable as warnings go.

If you're willing to compile from source yourself, it's fairly easy to build a set of binaries for a specific version .NET that doesn't have the warning. If you're building for v3.5, you need to remove Extension.cs from the three projects that have a copy of it (Microsoft.Scripting.Core, Microsoft.Scripting, IronPython) and add a reference to System.Core.dll. If you're building for v2.0, remove the Extension.cs from Microsoft.Scripting and IronPython then change the visibility in the Microsoft.Scripting.Core version from internal to public. Note, we treat warnings as errors in IPy, but we did add CS1685 to the list of WarningsNotAsErrors so the code still compiles. Of course, if you’re defining a framework specific version, you won’t get the warning anyway.

As usual, we appreciate all feedback from our community so hammer on this build as much as you can - esp. if you've been having type conflict errors with Beta 4. As I said in an earlier post, this is our last planned beta, so now’s the time put it thru the paces to make sure there’s nothing blocking you before we get to 2.0.

Finally, major props to Curt...aka IRON CURT...for driving these dev changes and putting up with the constant barrage of "where are we now?" status requests from yours truly. I'm sure he now regrets sitting across the hall within easy earshot.

IronPython Beta 5

Dave just posted the announcement to the IPy mailing list, Beta 5 has been released. You can download the release from CodePlex. Like Beta 4, this release is available in MSI or ZIP format, with the MSI including the Python Standard Library as an option.

As per the release notes, this is our last planned beta for 2.0 – our next planned release is RC1. So if you’ve been holding off on upgrading to the latest betas or to 2.0 in general, now is the time to upgrade if you want us to fix your bug before we ship 2.0.

Webdev Script and KillWebDevServer

I was updating my webdev powershell script today. I wanted to add support for a -browser switch that would automatically launch a browser window the way chiron from the Silverlight Dynamic Languages SDK. does. I also set the script to serve up the current directory by default. I posted the new version up on my SkyDrive.

While I was working on the script, I thought about how one might shutdown the WebDev server from the command line. That turned out to be much harder. Basically, you have to look thru all the top level windows for one that has "ASP.NET Development Server" in the window text, then you send that window two messages - WM_QUERYENDSESSION and WM_QUIT. Not sure why the WebDev server uses WM_QUERYENDSESSION to shut down it's tray icon, but if you look at WebDev.WebServer.exe in Reflector, you'll see the tray icon form overrides WinProc in order to look for message 0x11, i.e. WM_QUERYENDSESSION.

I threw together a quick little C# console app to shutdown the WebDev server and stuck it up on my SkyDrive as well. Source code is up there too. I had to use a bunch of P/Invokes to make it work, or I would have written it in Powershell or IronPython.

Enjoy.