As I’ve written previously, we’ve had a few challenges recently with name collisions in the DLR. In that post, I described how we had solved – or thought we solved, as it turned out – the problem with ExtensionAttribute name collisions between Microsoft.Scripting.Core.dll and System.dll.

However, as it turns out, having lots of copies of the same type didn’t solve the problem. Since ExtensionAttribute is a known type to the C# 3.0 compiler, it has to choose one of the multiple copies that are in the project. We thought that given a choice, it would favor the System.Core version. However, what folks discovered after we released Beta 5 is that C# 3.0 will choose the first copy of ExtensionAttribute that it finds. So if you have System.Core.dll and IronPython referenced in your project, and you define your own extension methods, the compile fails if the C# 3.0 compiler finds one of the IronPython or DLR private copies of ExtensionAttribute before the public copy in System.Core.

Furthermore, there doesn’t seem to be any way to set the reference order in MSBuild files. I’ve never dug deep into the MSBuild file format, but changing the order of the references in the csproj file didn’t seem to effect the order the references were passed to the C# compiler. I’m guessing we might be able to change this by fiddling with the ResolveAssemblyReference task, but we didn’t want to force low level MSBuild file hacking on our user base.

We looked at a variety of other solutions, including rewriting the IL after compilation to change the namespace of the ExtensionAttribute. However, we had trouble making that solution work and besides, changing the ExtensionAttribute namespace would have broken anyone using the existing DLR or IPy extension methods. So instead, we went with a different solution that we like to refer to as “The Fifth Assembly” around the office.

In IronPython 2.0 Beta 5, there were four DLLs that implement IronPython: IronPython.dll, IronPython.Modules.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll. In our RC1 release, we’ve added “The Fifth Assembly”: Microsoft.Scripting.ExtensionAttribute.dll. As you might guess from its name, it has only a single public type: ExtensionAttribute. By having ExtensionAttribute in its own dedicated assembly, we can avoid the type collision at compile time by not referencing both System.Core.dll and Microsoft.Scripting.ExtensionAttribute.dll in the same project.

In IronPython, we reference the ExtensionAttribute assembly because we use the C# 3.0 complier but IPy has to be able to run on .NET Framework 2.0 SP1.

However, projects that embed IronPython in a .NET 3.5 project (aka C# 3.0 or VB 9.0) will reference System.Core instead. The only reason why you would explicitly use the ExtensionAttribute assembly was that if you, like us, wanted to build your app with .NET 3.5, use extension methods but still be compatible with .NET 2.0 SP1. We’re guessing there aren’t many of our customers doing that, but if you are, explicitly referencing ExtensionAttribute will work just as it does for compiling IronPython itself.

It’s important to remember two things about the Fifth Assembly:

1. Never reference System.Core and Microsoft.Scripting.ExtensionAttribute in the same project.
2. Always deploy Microsoft.Scripting.ExtensionAttribute.dll as part of any solution that embeds IronPython (or IronRuby or vanilla DLR for that matter), even if you don’t reference it explicitly within your project.

This change is public in the source code as of change set 42076 and will also be in the nearly-ready RC1 release of IronPython 2.0. If you’ve got any questions or <shudder> find any more issues with this solution, please let us know right away.

Update: This approach doesn’t work. Please see the followup article for the gory details.

[Ed. Note – This long post is about changes we made in the DLR to avoid type collisions with System.Core. The short version of this post is “You can safely ignore the CS1685 warning you’ll get if you embed IPy 2.0 Beta 5 or later in your C# 3.0 application”. The long version below has the gory details of what we did, why we did it, a little about how extension methods work and why you can ignore warning CS1685.]

[Author note – I don’t really have an editor.]

Between Beta 3 and Beta 4 of IronPython 2.0, the DLR team made a very significant change to Microsoft.Scripting.Core.dll. As JB noticed, the DLR expression trees have merged with the LINQ expression trees. As part of this effort, they moved the newly merged expression tree types in Microsoft.Scripting.Core.dll into the System.Linq.Expressions namespace – the same namespace where those types live in System.Core.dll in .NET Framework 3.5.

This change caused quite a few issues with our users. Basically, because of the expression tree merge and the namespace change, the beta 4 version of Microsoft.Scripting.Core.dll had type collisions with System.Core.dll all over the place, making it hard or impossible to use them together. If you’re using C# you could get around these type collisions by using an assembly reference alias. However, assembly reference aliases aren’t supported for Web Sites projects or in Visual Basic.

To fix this, we’re changing the top level namespace in Microsoft.Scripting.Core.dll from System to Microsoft. We’re not going back to the namespaces as they were in Beta 3 – for example, DLR expression trees were originally in the Microsoft.Scripting.Ast namespace but now they’ll be in Microsoft.Linq.Expressions. We don’t think this change will be much of an issue because most people don’t use types from Microsoft.Scripting.Core.dll directly. Unless you’re building your own DLR language, this namespace change shouldn’t affect you at all except to solve the type collision problem.

However, we did hit a small snag.

The LINQ expression tree code, having been written for .NET 3.5, is a heavy user of extension methods. This means IronPython is now also a heavy user of extension methods. However, unlike LINQ, IronPython has to run against .NET 2.0 SP1. That means we can’t reference System.Core.dll in IPy or DLR. If you try to compile C# code with extension methods but without a reference System.Core.dll, you get a compiler error complaining that it can’t find the required ExtensionAttribute type, which is defined in System.Core.dll.

This might appear to be an unsolvable problem, but it turns out the C# compiler doesn’t actually care where the ExtensionAttribute type comes from. You can actually define your own copy of ExtensionAttribute (in the right namespace) and C# will happily compile extension methods without complaint. Furthermore, ExtensionAttribute is only used as a marker – there’s no real code in it – so implementing your own copy is trivial. In the DLR source, you’ll find they have defined their own copy of ExtensionAttribute so they can use extension methods and remain .NET 2.0 SP1 compatible. Since we were using them in Microsoft.Scripting.Core.dll, we started using extension methods in Microsoft.Scripting.dll and IronPython.dll as well.

If you’ll recall back to the start of this post, we’re changing namespaces in order to eliminate type collisions. The snag we hit was that we couldn’t change the ExtensionAttribute namespace without breaking all the extension methods. But we couldn’t leave it the same without having a type collision with the ExtensionAttribute defined in System.Core.dll. If you had a project including both copies of ExtensionAttribute, C# would generate a multiple type definition warning and VB would generate a multiple type definition error.

We’ve looked at a several possible solutions to this. One idea was to ship two completely different sets of binaries – one for .NET 2.0 and one for .NET 3.5. But the upgrade story for that stinks – you want to upgrade your app to .NET 3.5 you have to swap out all your IPy and DLR dlls. Yuck. We considered having separate copies of just Microsoft.Scripting.Core.dll – one defining ExtensionAttribute and the other linked to System.Core.dll and using TypeForwardedTo – but since the assemblies are strongly typed they’d have to same exact version number in order to be swappable. Double yuck.

In the end, we decided to put an internal copy of ExtensionAttribute in each assembly that needs it. As previously indicated, that’s IronPython.dll and Microsoft.Scripting.dll as well as making the copy already in Microsoft.Scripting.Core.dll internal. For IronRuby fans reading this, we also added a copy of ExtensionAttribute to IronRuby.dll and IronRuby.Libraries.Scanner.dll as well.

It seems counterintuitive, doesn’t it? To solve a multiple type definition problem, we defined even more copies of the type in question.

The key thing is that all these copies of ExtensionAttribute (except the one in System.Core.dll) are internal rather than public types. If you build a VB app that references System.Core.dll and Microsoft.Scripting.Core.dll (beta 4), you end up with multiple public copies of ExtensionAttribute and are rewarded with a VB compiler error. However, as long as there’s only one public copy of ExtensionAttribute – regardless of the number of internal copies of that type – VB is happy. So if you’re building a VB app against Microsoft.Scripting.Core.dll (beta 5) and System.Core, you should be golden.

In C# 3.0, on the other hand, continues to throw a warning. If ExtensionAttribute was a user-defined type, we’d be fine. However, since Extension attribute is a “predefined system type”, you get C# warning 1685 even though the copies of ExtensionAttribute are all internal. Furthermore, since there are multiple internal copies of ExtensionAttribute in the IPy dlls, you’ll get this warning even if you’re not referencing System.Core. It seems here that C# 3.0 considers ExtensionAttribute a predefined system type while VB doesn’t.

I realize that always having a warning in C# 3.0 – even if you’re not referencing System.Core.dll – doesn’t feel particularly clean. Given our desire to support both .NET v2 and v3.5 with the same binaries, it was the only choice. Remember that ExtensionAttribute has literally no code and is only used to signal the compiler for extension methods, so we decided it was fairly ignorable as warnings go.

If you’re willing to compile from source yourself, it’s fairly easy to build a set of binaries for a specific version .NET that doesn’t have the warning. If you’re building for v3.5, you need to remove Extension.cs from the three projects that have a copy of it (Microsoft.Scripting.Core, Microsoft.Scripting, IronPython) and add a reference to System.Core.dll. If you’re building for v2.0, remove the Extension.cs from Microsoft.Scripting and IronPython then change the visibility in the Microsoft.Scripting.Core version from internal to public. Note, we treat warnings as errors in IPy, but we did add CS1685 to the list of WarningsNotAsErrors so the code still compiles. Of course, if you’re defining a framework specific version, you won’t get the warning anyway.

As usual, we appreciate all feedback from our community so hammer on this build as much as you can – esp. if you’ve been having type conflict errors with Beta 4. As I said in an earlier post, this is our last planned beta, so now’s the time put it thru the paces to make sure there’s nothing blocking you before we get to 2.0.

Finally, major props to Curt…aka IRON CURT…for driving these dev changes and putting up with the constant barrage of “where are we now?” status requests from yours truly. I’m sure he now regrets sitting across the hall within easy earshot.

You know, this gets pretty long when I go a week between morning coffee posts.

Dynamic Language Stuff

• There’s a new version of the DLR Hosting spec. One of the biggest changes is the removal of DLR’s pre-canned knowledge of IPy and IRuby. Now, you use app.config to specify DLR languages and map them to extensions. Sesh has the details and a sample. If you only care about hosting IPy, you can use our language specific hosting API which doesn’t require app.config.
• Jimmy pushed out a new version of the Silverlight Dynamic Languages SDK. He’s on the lookout for a better name, one of his commenters suggested “SilverWhisper Kit”.
• Srivatsn shows how to pass Python keyword arguments to a C# method using ParamDictionaryAttribute, similar to how C# supports the params keyword via the ParamArrayAttribute. Unfortunately, there’s no syntactic sugar for calling param dictionary methods from C#. I wonder, do most people like or dislike the idea of language level support for keyword arguments in C#? I’m guessing most would be against, though I think it’s a cool idea.
• Maybe it’s me, but I’ve noticed an uptick in twitter posts containing “ironpython” of late. For example, via Summize I discovered Joseph Jude’s blog finally where he has posted a time tracking tool he built in IPy as well as his initial IPy learnings.
• Michael Foord, as usual, has been generating quite a bit of good IPy content. He did a tutorial on IPy at PyCon UK,  his book is almost done, he has some interesting thoughts on dynamic language performance and even admits that he likes C#, though not as much as he likes Python.
• Vernon Cole announced that the adodbapi project now supports IronPython 2.0 as well as CPython. This library is a DB-API 2.0 compliant interface for any database that support ADO (note, that’s native ADO not ADO.NET). I love to see Python projects with explicit support for IPy.
• My boss – who apparently will be called “Rem” from now on in the office – saves you “gazillions” by explaining non-positional negotiating strategy and draws the comparison to agile software development. Not exactly related to dynamic languages, but I’d like to see him further explore the relationship super-optimal, non-positional, significantly-hyphenated negotiation and software design. (OK, I made up “significantly-hyphenated”)

Other Stuff

• Don Syme blogs about an update to the F# CTP, a mere week after the original release. One week? That’s more often than even IPy releases. I can’t wait to see what they ship in next week’s release! 😄 Seriously, I hope they can keep the release sprints short, but every week would be a bit crazy!
• Speaking of F#, Matt Podwysocki updates FsTest for the F# CTP and posts about Extension Everything in F#. Unlike C#, which only supports extension methods, F# also supports extensions properties, static methods and events, though like Matt I can’t think of a good use for extension events.
• Still speaking about F#, Andrew Kennedy has a three part series on the new units of measure feature of F#. If you were going to use F# to build the physics engine of a game, I would suspect UoM would be extremely useful. (via Don Syme)
• Oh look, Chris Smith built an F# version of artillery game that uses Units of Measure for the physics code. I’ll bet UoM was extremely useful. 😄
• Talking about Live Mesh at TechEd Australia – where much to my surprise frankly they were demoing Live Mesh Apps – I pointed out to Scott Hanselman that Mesh is running an embedded CoreCLR (aka the same CLR from Silverlight 2). Scott went poking around and posted what he discovered. Looking forward to finding out what he digs up on using CoreCLR outside the browser.
• Speaking of Scott, I need to set up a family video conference solution like Scott’s before my next trip.
• Congrats to Glenn Block and the MEF team for their initial CodePlex source drop! I’ve been hearing about this possibility since Glenn joined the team, so I’m really excited to see it happen. I need to take a look at it in detail (in my copious spare time) because I want to find out how to make it work with IPy.
• Bart de Smet has a whole series (starting here) on Dynamic Expression Trees. However, given that he specifically writes “This blog series is not about DLR itself” makes it seem pretty conceptual to me. Why not talk about DLR expression trees instead Bart?
• I’m sure you noticed ASP.NET MVC preview 5 dropped last week. I really liked Brad Wilson’s discussion of the new view engine design.
• Tomas Restrepo has started publishing his source code on GitHub. Personally, I haven’t published any source code lately but I am using Git for all of my non IPy core work (which is stored in TFS). Like Tomas, I’m still getting the hang of Git but I’m really digging it’s speed, it’s branching and the fact that there’s zero infrastructure requirements. SVN provides the lightweight svnserve, but Git is even lighter weight than that.
• I liked Steve Yegge’s post on typing. I am a touch typer, but I doubt I type 70 words a minutes. I do know where the number keys are without looking though, so I guess that’s pretty good. I remember seeing Chris Anderson demo Avalon WPF long before it was public and being impressed at how fast he could type.

• Ben Hall announces IronEditor, a simple dev tool for IronPython and IronRuby. Pretty nice, though fairly simplistic (as Ben readily admits). For example, it doesn’t have an interactive mode, only the ability to execute scripts and direct the output to IronEditor’s output window. However, it is a good start and I’m sure it’ll just get better. One thing he’s apparently considering is a Silverlight version. (via Michael Foord)
• Speaking of “Iron” tools, Sapphire Steel have had an IronRuby version (in alpha) of their Ruby in Steel product for several months now. I wonder if John’s had a chance to play with it.
• Speaking of John, the ASP.NET MVC / IronRuby prototype he talked about @ TechEd is now available on ASP.NET MVC Preview 4 via Phil Haack.
• Ted Neward has an article exploring the IronPython VS Integration sample that ships in the VS SDK. As I mentioned the other day, we’re starting working on a production quality implementation of VS Integration for IPy.
• Ophir Kra-Oz (aka Evil Fish) blogs Python for Executives. I like his “Risk, Recruiting, Performance and Maturity” model – four boxes, perfect for keeping an executive’s attention! 😄 Plus Ophir has some nice things to say about IronPython. (via Michael Foord)
• Ronnie Maor blogs an extension method for PythonEngine to make Eval simpler. I especially like how he uses string format syntax so you can dynamically generate the code to eval. I wonder what this would look like in IPy 2.0 with DLR Hosting API. (via IronPython URLs)
• Speaking of DLR Hosting, Seshadri has another great DLR hosting post, this time hosting IPy inside of VS08 so you can script VS08 events (document saved, window created, etc) with Python.
• Justin Etheredge has a bunch of IronRuby posts – Getting IronRuby Up and Running, Running Applications in IronRuby, Learning Ruby via IronRuby and C# Part 1. (via Sam Gentile)
• Don Syme links to several F# related posts by Ray Vernagus, though he’s apparently also experimenting with IronRuby. I’m really interested in his Purely Functional Data Structures port to F#.
• Speaking of F#, Brian has a teaser screenshot of F# upcoming CTP. However, he chooses the New Item dialog to tease, which looks pretty much like the current new item dialog (the new one does have fewer F# templates). However, if you look in the Solution Explorer, you’ll notice a real “References” node. No more #I/#R! Yeah!
• The interactive graphic in Kevin Kelly’s One Machine article is fascinating. It really highlights that the vast vast vast majority of power, storage, CPU cycles and RAM come from personal computers on the edge. Even in bandwidth, where PC’s still have the highest share but it looks to be around 1/3rd, the aggregate of all edge devices (PCs, mobile phones, PDAs, etc.) still dominates the data centers.

• Check out the crowd
  
  for a the Washington Capitals developmental camp scrimmage last week (My parents are in their somewhere). Standing room only in the practice facility to watch a bunch of kids, most of whom won’t ever make it to the NHL, in July. If you think Washington can’t be a hockey town, you are sorely mistaken.
• Speaking of the Caps, they are establishing a “spirit squad“? Is that really necessary? (short answer: no). Peerless’ take is hilarious.
• Seshadri Vijayaraghavan is a tester on the DLR team and he’s been writing quite a bit about the DLR hosting API. He’s got a series of posts about hosting, invoking and redirecting output from IronPython in a C# application.
• I haven’t seen an official announcement, but mobile access to Live Mesh is available by pointing your phone browser to http://m.mesh.com. It’s mostly a web view of the Live Desktop, though there is a feature to upload photos from your phone. However, for some reason that feature doesn’t work for me right now. I don’t get the “browse” button.
• ASP.NET MVC Preview 4 is available for download. Phil Haack has a few details that ScottGu didn’t cover. Scott Hanselman shows off some AJAX stuff.
• Speaking of Scott Hanselman, he highlights the return of Terrarium from Bil Simser. Scott mentions that most Terrarium animal implementations were big collections of nested if statements. I wonder if F# pattern matching would be a cleaner approach?
• Ted Neward obviously never “even tangentially” touched politics, as I think they have far worse flame wars far more often than we have in the software industry. However, certainly the Scala flame war he’s commenting on seems fairly counterproductive.
• Brad Wilson runs into a wall trying to convert a string to an arbitrary Nullable<T>.He doesn’t find an answer, but I found reading thru the steps he took to try and find an answer strangely compelling.
• Jeff Atwood argues that Maybe Normalization isn’t Normal. It’s mostly a collection of information from other places, including a compilation of high-scale database case studies. But it’s a useful collection of info and links, with a little common-sense thrown in for good measure.
• I have a hard time imagining Pat Helland camping.