I LOVE the new boot from VHD feature in Win7. I am primarily using it for doing some VS 2010 dogfooding without messing up my primary drive partition. But man, the process for setting up a VHD for booting is brutal. Scott Hanselman did a great job laying out the steps, but I wanted something a bit more productive.

First, I created a clean Win7 RC VHD and zipped it up for easy storage. The basic Win7 RC VHD is just under 5GB, but compresses down to about 1.5GB with 7-zip. I used the ImageX process Aviraj described though in the future I’ll use the Install-WindowsImage script. Install-WindowsImage is more convenient to use because it will list the indexes within a given .wim file instead of making you grovel thru an XML file like ImageX does. Also Install-WindowsImage is 27k download while ImageX is part of the 1.4 gigabyte Windows Automated Installation Kit. Look, I’m not hurting for bandwidth, but I don’t see the point of downloading 54442 times more data for a utility that isn’t as useful.

Once you’ve created the VHD, you need to update your Boot Configuration Data, or BCD for short, using the appropriately named BCDEdit utility. The process is fairly straight forward, if tedious. You have to run BCDEdit four times, copy the configuration GUID to the clipboard and type out the path to the VHD in a slightly funky syntax. Blech. So I built a PowerShell script to automate updating the BCD, called add-bcd-vhd. You can get it from my SkyDrive. Pass in the name of the BCD entry and the path to the VHD and add-bcd-vhd will do the rest.

I was whining on Twitter yesterday that there’s no PowerShell specific tools for managing the BCD data. Add-bcd-vhd just runs bcdedit behind the scenes and processes the text output with regular expressions. Ugly, but effective. I decided to spend some time trying accessing the BCD data from its WMI provider, but that turned out to be way too much of a hassle to be effective. If someone else out there knows how to use the BCD WMI provider from PowerShell, I’d appreciate some sample code.

In my last post, I added support for custom attribute positional parameters . To finish things off, I need to add support for named parameters as well. Custom attributes support named parameters for public fields and settable properties. It works kind of like C# 3.0’s object initalizers. However, unlike object initalizers, the specific fields and properties to be set on a custom attribute as well as their values are passed to the CustomAttributeBuilder constructor. With six arguments – five of which are arrays – it’s kind of an ugly constructor. But luckily, we can hide it away in the make_cab function by using Python’s keyword arguments feature.

def make_cab(attrib_type, *args, **kwds):
  clrtype = clr.GetClrType(attrib_type)
  argtypes = tuple(map(lambda x:clr.GetClrType(type(x)), args))
  ci = clrtype.GetConstructor(argtypes)

  props = ([],[])
  fields = ([],[])

  for kwd in kwds:
    pi = clrtype.GetProperty(kwd)
    if pi is not None:
      props[0].append(pi)
      props[1].append(kwds[kwd])
    else:
      fi = clrtype.GetField(kwd)
      if fi is not None:
        fields[0].append(fi)
        fields[1].append(kwds[kwd])
      else:
        raise Exception, "No %s Member found on %s" % (kwd, clrtype.Name)

  return CustomAttributeBuilder(ci, args,
    tuple(props[0]), tuple(props[1]),
    tuple(fields[0]), tuple(fields[1]))

def cab_builder(attrib_type):
  return lambda *args, **kwds:make_cab(attrib_type, *args, **kwds)

You’ll notice that make_cab now takes a third parameter: the attribute type and the tuple of positional arguments we saw last post. This third parameter “**kwds” is a dictionary of named parameters. Python supports both positional and named parameter passing, like VB has for a while and C# will in 4.0. However, this **kwds parameter contains all the extra or leftover named parameters that were passed in but didn’t match any existing function arguments. Think of it like the params of named parameters.

As I wrote earlier, custom attributes support setting named values of both fields and properties. We don’t want the developer to have to know if given named parameter is a field or property, so make_cab iterates over all the named parameters, checking first to see if it’s a property then if it’s a field. It keeps a list of all the field / property infos as well as their associated values. Assuming all the named parameters are found, those lists are converted to tuples and passed into the CustomAttributeBuilder constructor.

In addition to the change to make_cab, I also updated cab_builder slightly in order to pass the **kwds parameter on thru to the make_cab function. No big deal. So now, I can add an attribute with named parameters to my IronPython class and it still looks a lot like a C# attribute specification.

clr.AddReference("System.Xml")
from System.Xml.Serialization import XmlRootAttribute
from System import ObsoleteAttribute, CLSCompliantAttribute
Obsolete = cab_builder(ObsoleteAttribute)
CLSCompliant = cab_builder(CLSCompliantAttribute)
XmlRoot = cab_builder(XmlRootAttribute)

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [
    Obsolete("Warning Lark's Vomit"),
    CLSCompliant(False),
    XmlRoot("product", Namespace="http://samples.devhawk.net")]

  # remainder of Product class omitted for clarity

As usual, sample code is up on my SkyDrive.

Now that I can support custom attributes on classes, it would be fairly straightforward to add them to methods, properties, etc as well. The hardest part at this point is coming up with a well designed API that works within the Python syntax. If you’ve got any opinions on that, feel free to share them in the comments, via email, or on the IronPython mailing list.

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

def make_cab(attrib_type, *args):
  argtypes = tuple(map(lambda x:clr.GetClrType(type(x)), args))
  ci = clr.GetClrType(attrib_type).GetConstructor(argtypes)
  return CustomAttributeBuilder(ci, args)

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [make_cab(ObsoleteAttribute , "Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

In make_cab, I build a tuple of CLR types from the list of positional arguments that was passed in. If you haven’t seed the *args syntax before, it works like C#’s params keyword – any extra arguments are passed into the function as a tuple names args. I use Python’s built in map function (FP FTW!) to build a tuple of CLR types of the provided arguments, which I then pass to GetConstructor. Previously, I passed an empty tuple to GetConstructor because I wanted the default constructor. If you don’t pass any positional arguments, you still get the default constructor. Once I’ve found the right constructor, I pass it and the original tuple of arguments to the CustomAttributeBuilder constructor.

One major benefit of this approach is that it simplifies the metaclass code. Since _clrclassattribs is now a list of CustomAttributeBuilders, now I just need to iterate over that list and call SetCustomAttribute for each.

if hasattr(cls, '_clrclassattribs'):
      for cab in cls._clrclassattribs:
        typebld.SetCustomAttribute(cab)

The only problem with this approach is that specifying the list of custom attributes is now extremely verbose. Not only am I specifying the full attribute class name as well as the positional arguments, I’m also having to insert a call to make_cab. Previously, it kinda looked like a C# custom attribute, albeit in the wrong place. Not anymore. So I decided to write a function called cab_builder to generates less verbose calls to make_cab:

def cab_builder(attrib_type):
  return lambda *args:make_cab(attrib_type, *args)

from System import ObsoleteAttribute
Obsolete = cab_builder(ObsoleteAttribute)

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [Obsolete("Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

The cab_builder function returns an anonymous lambda function that closes over the attrib_type variable. Python lambdas are just like C# lambdas, except that they only support expressions ¹. The results of calling the lambda returned from cab_builder is exactly the same as calling make_cab directly, but less verbose. And since I named the function returned from cab_builder Obsolete, now my list of class custom attributes looks exactly like it does in C# (though still in a different place). As usual, the code is up on my SkyDrive.

If you’re only using the attribute once like this, it is kind of annoying to first declare the cab_builder function. If you wanted to you could iterate over the types in a given assembly, looking for ones that inherit from Attribute and generate the cab_builder call dynamically. However, I’m not sure how performant that would be. Another possibility would be to iterate over the types in a given assembly and generate a Python module on disk with the calls to cab_builder. Then, you’d just have to import this module of common attributes but still be able to include additional calls to cab_builder as needed.

I know it’s been a while since my last __clrtype__ post, but I was blocked on some bug fixes that shipped as part of IronPython 2.6 Beta 1. So now let’s start looking at one of the most requested IronPython features – custom attributes!

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [ObsoleteAttribute]

  # remainder of class omitted for clarity

Python list comprehensions use the same square bracket syntax as C# properties, so it kinda looks right to someone with a C# eye – though having the attribute specifications inside the class, rather than above it, is totally different. I wish I could use Python’s class decorators for custom class attributes, but class decorators run after metaclasses so unfortunately that doesn’t work. Also, I can’t leave off the “Attribute” suffix like you can in C#. If I really wanted to, I could provide a new type name in the import statement (“from System import ObsoleteAttribute as Obsolete”) but I thought spelling it out was clearer for this post.

Now that I have specified the class attributes, I can update the metaclass __clrtype__ method to set the attribute on the generated CLR class:

if hasattr(cls, '_clrclassattribs'):
      for attribtype in cls._clrclassattribs:
        ci = clr.GetClrType(attribtype).GetConstructor(())
        cab = CustomAttributeBuilder(ci, ())
        typebld.SetCustomAttribute(cab)

I’m simply iterating over the list of _clrclassattribs (if it exists), getting the default parameterless constructor for each attribute type, creating a CustomAttributeBuilder instance from that constructor and then calling SetCustomAttribute. Of course, this is very simple because we’re not supporting any custom arguments or setting of named properties. We’ll get to that in the next post. In the mean time, you can get the full code for this post from my SkyDrive.

There is one significant issue with this custom attribute code. Attributes are typically marked with the AttributeUsage attribute that specifies a set of constraints, such as the kind of targets a given attribute can be attached to and if it can be specified multiple times. For example, the MTAThread attribute can’t be specified multiple times and it can only be attached to methods. However, those attribute constraints are validated by the compiler, not the runtime. I haven’t written any code yet to validate those constraints, so you can specify invalid combinations like multiple MTAThread attributes on a class. For now, I’m just going to leave it to the developer not to specify invalid attribute combinations. Custom attributes are passive anyway so I’m figure no one will come looking for a MTAThread attribute on a class or other such scenarios.

However, I’m interested in your opinion: When we get to actually productizing a higher-level API for __clrtype__, what kinds of attribute validation should we do, if any?

Today was the Presentation Idol competition I blogged about a couple of weeks ago. Unfortunately, I didn’t win – but believe me when I say I was up against some serious competition. I think I was about in the middle of the pack – better than some but clearly not as good as others.

Since I made a big deal about asking for people ideas on how to present my topic of choice – external contributions to Microsoft Open Source projects – I decided I’d post my deck and my script. Yes, I said script. Usually, I don’t script what I’m going to say word for word like this. But with only three and a half minutes to present, I thought I’d be as precise as possible. You’ll notice some lines near the end are in italics – those are ones I planned on cutting if I was in danger of going over the time limit.

Feedback, as usual, is most welcome.

Hello, my name is Harry Pierson. I’m a program manager in the Visual Studio Languages group and I’m here to talk about what I would most like to do to improve Microsoft.

At Microsoft, we care an awful lot about the software ecosystem. Searching for the word “ecosystem” on Microsoft.com returns nearly eight thousand results. We talk about the ecosystem in our marketing and in our press releases.

In 2007, we commissioned a study of the global economic impact of the Microsoft ecosystem. In a word, it’s massive. It’s responsible for nearly 15 million jobs and drives over half a trillion dollars a year in tax revenue worldwide.

No wonder we care about the ecosystem so much.

But clearly, we’re not the only ones who care. Microsoft represents a fairly small percentage of the overall ecosystem. We earn just over 11% of the total revenue and account for half of one percent of the total employment within the ecosystem.

That means there are an awfully large number of people with a vested interest in the continuing success of the Microsoft platform.

With this vast ecosystem in mind, I want to talk about Open Source. Microsoft and Open Source are often portrayed as enemies. But in DevDiv, we have several high profile Open Source projects. I work on IronPython, which has been Open Source for over four years. More recently, the ASP.NET and Extensibility Framework teams have decided to release some projects as Open Source.

I believe we should have more Open Source projects at Microsoft. But more importantly, I feel that we need to go beyond the textbook definition of Open Source. Our Open Source projects are typically closed to external contributions. But for the ecosystem at large, Open Source isn’t just about the availability and distribution terms of the source code. It also implies a collaborative development model - parties working together across organizational boundaries, contributing directly to projects in question.

The thing I would most like to change about Microsoft would be to let members of our ecosystem contribute code to our Open Source projects.

I can tell you from personal experience, there are members of the IronPython community who would leap at the opportunity to contribute code. And their engineering prowess and real world would benefit the IronPython project tremendously. But the legal process for getting permission to take contributions is onerous. Worse, the legal stigma attached to code that isn’t 100% pure Microsoft intellectual property makes it nearly impossible for any other group inside Microsoft to build on it.

I realize the onerous legal process is there for a reason: to protect Microsoft’s interests. But improving IronPython and Open Source projects like it isn’t just in Microsoft’s best interest; it’s in the best interest of our ecosystem as well. We need a legal framework that protects Microsoft while allowing for code contributions. Developing such a framework will be a challenge. But competitors like Sun, Google and IBM have already demonstrated that it’s not insurmountable.

Half a trillion dollars annually. 15 Million jobs. 42% of the IT workforce. The Microsoft ecosystem is the envy of the industry. And Microsoft is in a unique position to harness the collective experience and engineering prowess of that ecosystem while simultaneously dispelling the myth that we are an enemy of Open Source.

It’s time we make this change.

Thank you very much.