As I’ve written previously, we’ve had a few challenges recently with name collisions in the DLR. In that post, I described how we had solved – or thought we solved, as it turned out – the problem with ExtensionAttribute name collisions between Microsoft.Scripting.Core.dll and System.dll.

However, as it turns out, having lots of copies of the same type didn’t solve the problem. Since ExtensionAttribute is a known type to the C# 3.0 compiler, it has to choose one of the multiple copies that are in the project. We thought that given a choice, it would favor the System.Core version. However, what folks discovered after we released Beta 5 is that C# 3.0 will choose the first copy of ExtensionAttribute that it finds. So if you have System.Core.dll and IronPython referenced in your project, and you define your own extension methods, the compile fails if the C# 3.0 compiler finds one of the IronPython or DLR private copies of ExtensionAttribute before the public copy in System.Core.

Furthermore, there doesn’t seem to be any way to set the reference order in MSBuild files. I’ve never dug deep into the MSBuild file format, but changing the order of the references in the csproj file didn’t seem to effect the order the references were passed to the C# compiler. I’m guessing we might be able to change this by fiddling with the ResolveAssemblyReference task, but we didn’t want to force low level MSBuild file hacking on our user base.

We looked at a variety of other solutions, including rewriting the IL after compilation to change the namespace of the ExtensionAttribute. However, we had trouble making that solution work and besides, changing the ExtensionAttribute namespace would have broken anyone using the existing DLR or IPy extension methods. So instead, we went with a different solution that we like to refer to as “The Fifth Assembly” around the office.

In IronPython 2.0 Beta 5, there were four DLLs that implement IronPython: IronPython.dll, IronPython.Modules.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll. In our RC1 release, we’ve added “The Fifth Assembly”: Microsoft.Scripting.ExtensionAttribute.dll. As you might guess from its name, it has only a single public type: ExtensionAttribute. By having ExtensionAttribute in its own dedicated assembly, we can avoid the type collision at compile time by not referencing both System.Core.dll and Microsoft.Scripting.ExtensionAttribute.dll in the same project.

In IronPython, we reference the ExtensionAttribute assembly because we use the C# 3.0 complier but IPy has to be able to run on .NET Framework 2.0 SP1.

However, projects that embed IronPython in a .NET 3.5 project (aka C# 3.0 or VB 9.0) will reference System.Core instead. The only reason why you would explicitly use the ExtensionAttribute assembly was that if you, like us, wanted to build your app with .NET 3.5, use extension methods but still be compatible with .NET 2.0 SP1. We’re guessing there aren’t many of our customers doing that, but if you are, explicitly referencing ExtensionAttribute will work just as it does for compiling IronPython itself.

It’s important to remember two things about the Fifth Assembly:

1. Never reference System.Core and Microsoft.Scripting.ExtensionAttribute in the same project.
2. Always deploy Microsoft.Scripting.ExtensionAttribute.dll as part of any solution that embeds IronPython (or IronRuby or vanilla DLR for that matter), even if you don’t reference it explicitly within your project.

This change is public in the source code as of change set 42076 and will also be in the nearly-ready RC1 release of IronPython 2.0. If you’ve got any questions or <shudder> find any more issues with this solution, please let us know right away.

Update: This approach doesn’t work. Please see the followup article for the gory details.

[Ed. Note – This long post is about changes we made in the DLR to avoid type collisions with System.Core. The short version of this post is “You can safely ignore the CS1685 warning you’ll get if you embed IPy 2.0 Beta 5 or later in your C# 3.0 application”. The long version below has the gory details of what we did, why we did it, a little about how extension methods work and why you can ignore warning CS1685.]

[Author note – I don’t really have an editor.]

Between Beta 3 and Beta 4 of IronPython 2.0, the DLR team made a very significant change to Microsoft.Scripting.Core.dll. As JB noticed, the DLR expression trees have merged with the LINQ expression trees. As part of this effort, they moved the newly merged expression tree types in Microsoft.Scripting.Core.dll into the System.Linq.Expressions namespace – the same namespace where those types live in System.Core.dll in .NET Framework 3.5.

This change caused quite a few issues with our users. Basically, because of the expression tree merge and the namespace change, the beta 4 version of Microsoft.Scripting.Core.dll had type collisions with System.Core.dll all over the place, making it hard or impossible to use them together. If you’re using C# you could get around these type collisions by using an assembly reference alias. However, assembly reference aliases aren’t supported for Web Sites projects or in Visual Basic.

To fix this, we’re changing the top level namespace in Microsoft.Scripting.Core.dll from System to Microsoft. We’re not going back to the namespaces as they were in Beta 3 – for example, DLR expression trees were originally in the Microsoft.Scripting.Ast namespace but now they’ll be in Microsoft.Linq.Expressions. We don’t think this change will be much of an issue because most people don’t use types from Microsoft.Scripting.Core.dll directly. Unless you’re building your own DLR language, this namespace change shouldn’t affect you at all except to solve the type collision problem.

However, we did hit a small snag.

The LINQ expression tree code, having been written for .NET 3.5, is a heavy user of extension methods. This means IronPython is now also a heavy user of extension methods. However, unlike LINQ, IronPython has to run against .NET 2.0 SP1. That means we can’t reference System.Core.dll in IPy or DLR. If you try to compile C# code with extension methods but without a reference System.Core.dll, you get a compiler error complaining that it can’t find the required ExtensionAttribute type, which is defined in System.Core.dll.

This might appear to be an unsolvable problem, but it turns out the C# compiler doesn’t actually care where the ExtensionAttribute type comes from. You can actually define your own copy of ExtensionAttribute (in the right namespace) and C# will happily compile extension methods without complaint. Furthermore, ExtensionAttribute is only used as a marker – there’s no real code in it – so implementing your own copy is trivial. In the DLR source, you’ll find they have defined their own copy of ExtensionAttribute so they can use extension methods and remain .NET 2.0 SP1 compatible. Since we were using them in Microsoft.Scripting.Core.dll, we started using extension methods in Microsoft.Scripting.dll and IronPython.dll as well.

If you’ll recall back to the start of this post, we’re changing namespaces in order to eliminate type collisions. The snag we hit was that we couldn’t change the ExtensionAttribute namespace without breaking all the extension methods. But we couldn’t leave it the same without having a type collision with the ExtensionAttribute defined in System.Core.dll. If you had a project including both copies of ExtensionAttribute, C# would generate a multiple type definition warning and VB would generate a multiple type definition error.

We’ve looked at a several possible solutions to this. One idea was to ship two completely different sets of binaries – one for .NET 2.0 and one for .NET 3.5. But the upgrade story for that stinks – you want to upgrade your app to .NET 3.5 you have to swap out all your IPy and DLR dlls. Yuck. We considered having separate copies of just Microsoft.Scripting.Core.dll – one defining ExtensionAttribute and the other linked to System.Core.dll and using TypeForwardedTo – but since the assemblies are strongly typed they’d have to same exact version number in order to be swappable. Double yuck.

In the end, we decided to put an internal copy of ExtensionAttribute in each assembly that needs it. As previously indicated, that’s IronPython.dll and Microsoft.Scripting.dll as well as making the copy already in Microsoft.Scripting.Core.dll internal. For IronRuby fans reading this, we also added a copy of ExtensionAttribute to IronRuby.dll and IronRuby.Libraries.Scanner.dll as well.

It seems counterintuitive, doesn’t it? To solve a multiple type definition problem, we defined even more copies of the type in question.

The key thing is that all these copies of ExtensionAttribute (except the one in System.Core.dll) are internal rather than public types. If you build a VB app that references System.Core.dll and Microsoft.Scripting.Core.dll (beta 4), you end up with multiple public copies of ExtensionAttribute and are rewarded with a VB compiler error. However, as long as there’s only one public copy of ExtensionAttribute – regardless of the number of internal copies of that type – VB is happy. So if you’re building a VB app against Microsoft.Scripting.Core.dll (beta 5) and System.Core, you should be golden.

In C# 3.0, on the other hand, continues to throw a warning. If ExtensionAttribute was a user-defined type, we’d be fine. However, since Extension attribute is a “predefined system type”, you get C# warning 1685 even though the copies of ExtensionAttribute are all internal. Furthermore, since there are multiple internal copies of ExtensionAttribute in the IPy dlls, you’ll get this warning even if you’re not referencing System.Core. It seems here that C# 3.0 considers ExtensionAttribute a predefined system type while VB doesn’t.

I realize that always having a warning in C# 3.0 – even if you’re not referencing System.Core.dll – doesn’t feel particularly clean. Given our desire to support both .NET v2 and v3.5 with the same binaries, it was the only choice. Remember that ExtensionAttribute has literally no code and is only used to signal the compiler for extension methods, so we decided it was fairly ignorable as warnings go.

If you’re willing to compile from source yourself, it’s fairly easy to build a set of binaries for a specific version .NET that doesn’t have the warning. If you’re building for v3.5, you need to remove Extension.cs from the three projects that have a copy of it (Microsoft.Scripting.Core, Microsoft.Scripting, IronPython) and add a reference to System.Core.dll. If you’re building for v2.0, remove the Extension.cs from Microsoft.Scripting and IronPython then change the visibility in the Microsoft.Scripting.Core version from internal to public. Note, we treat warnings as errors in IPy, but we did add CS1685 to the list of WarningsNotAsErrors so the code still compiles. Of course, if you’re defining a framework specific version, you won’t get the warning anyway.

As usual, we appreciate all feedback from our community so hammer on this build as much as you can – esp. if you’ve been having type conflict errors with Beta 4. As I said in an earlier post, this is our last planned beta, so now’s the time put it thru the paces to make sure there’s nothing blocking you before we get to 2.0.

Finally, major props to Curt…aka IRON CURT…for driving these dev changes and putting up with the constant barrage of “where are we now?” status requests from yours truly. I’m sure he now regrets sitting across the hall within easy earshot.

Dave just posted the announcement to the IPy mailing list, Beta 5 has been released. You can download the release from CodePlex. Like Beta 4, this release is available in MSI or ZIP format, with the MSI including the Python Standard Library as an option.

As per the release notes, this is our last planned beta for 2.0 – our next planned release is RC1. So if you’ve been holding off on upgrading to the latest betas or to 2.0 in general, now is the time to upgrade if you want us to fix your bug before we ship 2.0.

Fans of Python use the phrase “batteries included” to describe the standard library, which covers everything from asynchronous processing to zip files.
About Python

I’m very happy to announce that as of IronPython Beta 4, we are including the Python standard library in an IronPython release. In addition to the usual binary and source zip archives, we’re shipping an MSI installer that gives you the option to include the library files. I mentioned a couple of weeks ago that I’ve been spending a lot of time with lawyers, and this is the first step towards making the IronPython project as open as possible. Note the use of the word “first” in that previous statement. In other words, I’ve got more lawyers to talk to.

We’ve added a new standard library component category to the IronPython Issue Tracker. So if you find any issues or incompatibilities with using the standard library with IronPython, please files bugs on them. We haven’t figured out what set of libraries we’re going to ship, but we don’t want to include ones that we know wont work. For example, Kevin Chu reported on the mailing list that importing the sqlite3 module fails. This is because we don’t have a managed version of the _sqlite3 C-based module. So that’s one we’ll probably remove outright.

Also, this is the first release we’ve distributed as an MSI file, and it looks some folks on the mailing list have had some troubles with it. If you have issues, please file setup bugs as well.

Two weeks ago, I posted a roadmap for where IPy is heading after we get 2.0 out the door later this year. I got two questions in the comments that I wanted to address.

• Michael Foord wants to know “will you actively maintain IronPython 2 and IronPython 3 in parallel?”
• francois wants to know for the VS integration “Will it be necessary to buy VS standard/Pro to benefit from it?”

As to the first question, Curt Hagenlocher (aka Iron Curt) from the IPy dev team wrote the following on the IPy mailing list:

“I think it’s fair to say that this will be driven mostly by the needs of our community and of the larger Python community. By the time IronPython 3.0 ships, it’s likely that CPython 3.0 will have been out for a year or even longer, and we’ll have a much better idea of what the uptake is like in the community.”

Let me second Curt’s comment. I used to think getting to 3.0 as quickly as possible was one of our highest priorities. However, it looks like the CPy 3.0 uptake rate is a real open question right now. If the Python community takes 3-4 years to fully embrace 3.0 (as Michael suggested) then I don’t see how we will have any choice but to maintain both in parallel. It also suggests that better 2.6 support may be more important than 3.0 support, at least for the next 18-24 months.

Francois’s question is much easier: No, we will not require you to buy any version of VS in order to get IPy VS integration. It will work with the Visual Studio Shell, much like the current IronPythonStudio VSX sample does today.

For out first release, we are focusing on the integrated mode scenario. This means that if you own a copy of VS (other than express), the IPy tools will snap into your existing VS installation. If you don’t own a copy of VS, you’ll be able to download and install the integrated shell for free and our IPy tools will snap into that. Down the road, we might investigate an “IronPython Express” version that leverages the isolated shell, but that’s not a priority right now.

The VS shell includes support for the WinForms and WPF designers and we expect to support both eventually. WPF support is fairly straightforward since the designer works with XAML files rather than code files. WinForms support is harder, since it wants to emit statically typed code into code files – a neat trick for a dynamically typed language like Python. The current implementation that the IronPythonStudio sample uses requires us to inject a bunch of custom type hints disguised as python @decorators in order to round trip between the designer and code views. However, we’re not bringing those custom type hints forward in the 2.0 release so we’ll need to find a better way to integrate with the WinForms designer.

As for web apps, ASP.NET support is currently on the back burner while Jimmy and David drink coffee and expensive juice while they figure out the ASP.NET / dynamic language story. We also want to support Silverlight development for IronPython in VS, but I’m exactly sure how we get Silverlight designer support yet. Silverlight Tools for VS08 today require VS Standard, so I’m guessing there’s some leg work to do if we want to support the SL designer in VS Shell.

As always, feel free to ask here or on the mailing list if you’ve got any questions or feedback on our plans.