__clrtype__ Metaclasses Series

Introducing __clrtype__ Metaclasses

Everyone knows Anders announced at PDC08 that C# 4.0 will include new features (aka the dynamic keyword + the DLR) that makes it much easier for C# to call into dynamically typed code. What you probably don’t know is that IronPython 2.6 includes a new feature that makes it easier for IronPython code to be called by statically typed code.

While the vast majority of .NET is available to IronPython, there are certain APIs that just don’t work with dynamic code. In particular, any code that uses Reflection over an object’s CLR type metadata won’t work with IronPython. For example, while WPF supportsICustomTypeDescriptor, Silverlight only supports data binding against reflectable properties. Furthermore, any code that uses custom attributes inherently uses Reflection. For example, Darrel Hawley recently blogged a WCF host he wrote in IronPython, but he wrote the WCF service in C#. You can’t write WCF services in IronPython because WCF expects service classes to be adorned with ServiceContract and OperationContract attributes (among many others). IronPython users want access to use these APIs. Support for custom attributes is one of the most common requests we get – it’s currently the 5th highest vote getter among open issues.

In IronPython 2.6, we’re adding the ability to customize the CLR type of Python classes. This means you can add custom attributes, emit properties, whatever you want. For those of you who’ve been dreaming of implementing WCF services or databinding in Silverlight purely in IronPython, then this is the feature for you.

In a nutshell, IronPython 2.6 extends Python’s metaclass feature that lets you to customize the creation of classes. In the metaclass, you can implement an IronPython-specific method __clrtype__ which returns a custom System.Type of your own creation that IronPython will then use as the underlying CLR type of the Python class. Implementing __clrtype__ gives you the chance to implement whatever reflectable metadata you need: constructors, fields, properties, methods, events, custom attributes, nested classes, whatever.

Over a series of posts, I’ll be demonstrating this new feature and implement some common scenario requests – including Silverlight databinding and WCF services – purely in Python. Quick warning: __clrtype__ uses low level features like Python metaclasses, Reflection.Emit and DLR Binders so these posts will be deeper technically than usual. Don’t worry – this isn’t the API interface we expect everyone to use. Eventually, we want to have an easy to use API that will sit on top of the low-level __clrtype__ hook.

__clrtype__ Metaclasses: IronPython Classes Under the Hood

Before we start using __clrtype__ metaclasses, we need to understand a bit about how IronPython maps between CLR types and Python classes. IronPython doesn’t support Reflection based APIs or custom attributes today because IronPython doesn’t emit a custom CLR types for every Python class. Instead, it typically shares a single CLR type across many Python classes. For example, all three of these Python classes share a single underlying CLR type.

class shop(object):
  pass

class cheese_shop(shop):
  def have_cheese(self, cheese_type):
    return False

class argument_clinic(object):
  def is_right_room(self, room=12):
    return "I've told you once"

import clr
print clr.GetClrType(shop).FullName
print clr.GetClrType(cheese_shop).FullName
print clr.GetClrType(argument_clinic).FullName

Even though cheese_shop inherits from shop and argument_clinic inherits from object, all three classes share the same underlying CLR type. On my machine, running IronPython 2.6 Alpha 1, that type is named IronPython.NewTypes.System.Object_1$1.

IronPython can share the CLR type across multiple Python classes because that CLR type has no code specific to a given Python class. CLR types are immutable – once you build a CLR type, you can’t do things like add new methods, remove existing method or change the inheritance hierarchy. But all those things are legal to do in Python. Here, I’m creating an instance of the cheese_shop class, but then changing that instance to be an argument_clinic instance instead.

>>> cs = cheese_shop()

>>> cs.have_cheese("Venezuelan Beaver Cheese")
False
>>> cs.is_right_room(12)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'cheese_shop' object has no attribute 'is_right_room'

>>> # Change the object's class at runtime
>>> cs.__class__ = argument_clinic # don't try this in C#!

>>> cs.have_cheese("Venezuelan Beaver Cheese")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'argument_clinic' object has no attribute 'have_cheese'
>>> cs.is_right_room(12)
"I've told you once"

When you call a method on a Python object, the name is resolved by walking a series of dictionaries. First, the dictionary of the object itself is searched for the method name. Assuming the name isn’t in the object dictionary, Python then looks in the __class__ dictionary. If it’s not there, Python recursively looks through the base classes stored in the __bases__ tuple until it finds the method or the name fails to resolve. If we re-assign __class__ at run time, we change the dictionary Python uses to resolve method names.

There are cases where IronPython generates a new underlying CLR type. For example, if you build a Python class that inherits from a CLR type, then IronPython will have to generate a new underlying CLR type that inherits from the CLR type in order to remain compatible. IronPython automatically overrides all the virtual methods of the base type, implementing the same dynamic method dispatch that I described above. This lets you pass the IronPython class wherever the base CLR type is expected.

The ability to swap Python classes at runtime depends on having the same underlying CLR type. If the underlying CLR type doesn’t match, then assigning a new value to the __class__ field of an object will fail. This applies both to IronPython classes that inherit from CLR types as well as __clrtype__ metaclass types. In the code I’ll be blogging, I always generate a unique CLR type for every Python class, which means that I can’t dynamically retype the object. Given that the point of __clrtype__ metaclasses is to generate static type information, this hardly seems like a limitation. However, it’s something to be aware of as we explore the __clrtypes__ feature.

__clrtype__ Metaclasses: Customizing the Type Name

Now that we know a little about how IronPython uses CLR types under the hood, let’s start customizing those types. In a nutshell, __clrtype__ metaclasses are metaclasses that implement a function named __clrtype__ that takes the Python class definition as a parameter and returns a System.Type. IronPython will then use the returned Type  as the underlying CLR type whenever you create an instance of the Python class.

Technically, you could emit whatever custom CLR Type you want to in the __clrtype__, but typically you’ll want to emit a class that both implements whatever static CLR metadata you need as well as the dynamic binding infrastructure that IronPython expects. The easiest way to do this is to ask IronPython emit a type that handles all the dynamic typing and then inherit from that type to add the custom CLR metadata you want.

Let’s start simple and hello-worldly by just customizing the name of the generated CLR type that’s associated with the Python class. There’s a fair amount of boilerplate code that is needed even for this simple scenario, and I can build on that as we add features that actually do stuff. If you want to follow along at home, you’ll need IronPython 2.6 Alpha 1 (or later) and you can get this code from my SkyDrive.

class ClrTypeMetaclass(type):
  def __clrtype__(cls):
    baseType = super(ClrTypeMetaclass, cls).__clrtype__()
    typename = cls._clrnamespace + "." + cls.__name__
                 if hasattr(cls, "_clrnamespace")
                 else cls.__name__

    typegen = Snippets.Shared.DefineType(typename, baseType, True, False)
    typebld = typegen.TypeBuilder

    for ctor in baseType.GetConstructors():
      ctorparams = ctor.GetParameters()
      ctorbld = typebld.DefineConstructor(
                  ctor.Attributes,
                  ctor.CallingConvention,
                  tuple([p.ParameterType for p in ctorparams]))
      ilgen = ctorbld.GetILGenerator()
      ilgen.Emit(OpCodes.Ldarg, 0)
      for index in range(len(ctorparams)):
        ilgen.Emit(OpCodes.Ldarg, index + 1)
      ilgen.Emit(OpCodes.Call, ctor)
      ilgen.Emit(OpCodes.Ret)

    return typebld.CreateType()

Like all Python metaclasses, ClrTypeMetaclass inherits from the built-in Python type object. If I wanted to customize the Python class as well, I could implement __new__ on ClrTypeMetaclass , but I only care about customizing the CLR type so it only implements __clrtype__. If you want to know more about what you can do with Python metaclasses, check out Michael Foord’s Metaclasses in Five Minutes.

First off, I want to get IronPython to generate the base class that will implement all the typical Pythonic stuff like name resolution and dynamic method dispatch. To do that, I call __clrtype__ on the supertype of ClrTypeMetaclass – aka the built-in type object. That function returns the System.Type that IronPython would have used as the underlying CLR type for the Python class if we weren’t using __clrtype__ metaclasses.

Once I have the base class, next I figure out what the name of the generated CLR type will be. This is pretty simple, I just use the name of the Python class. To make this logic a little more interesting, I added support for a custom namespace. If the Python class has a _clrnamespace field, I append that as the custom namespace for the name. I should probably be using a double underscore – i.e. __clrnamespace – but I didn’t want to wrestle with name mangling in this prototype code.

Now that I have a name and a base class, I can generate the class I’m going to use. I’m using the DefineType method in Microsoft.Scripting.Generation.Snippets DLR class for three reasons. First, there’s a CLR bug that doesn’t let you create a dynamic assembly from a dynamic method. Second, reusing the snippets assembly avoids the overhead of generating a new assembly. Finally, the types in Snippets.Shared get saved to disk if you run with the -X:SaveAssemblies flag, so you can inspect custom CLR type that gets generated. The DefineType function takes four parameters, the type name, the base class, a preserve name flag and a generate debug symbols flag. If you pass false for preserve name, you get a name like foobar$1 instead of just foobar. As for debug symbols, since I don’t have any source code that I’m generating IL from, emitting debug symbols doesn’t make a lot of sense. DefineType returns a TypeGen, but I only need the TypeBuilder.

The last thing I need to do is implement the custom CLR type constructor(s). IronPython CLR types will always have at least one parameter – the PythonType (PythonType == IronPython’s implementation of Python’s built-in type object) that’s used for dynamic name resolution. I don’t want to add any custom functionality in my custom CLR type constructors, so I simply iterate thru the list of constructors on the base class and generate a constructor on the custom CLR type with a matching parameter list and that calls the base class constructor. 

Generating the IL to emit the constructor and the base class is straightforward, if tedious. I define the constructor with the same attributes, calling convention and parameters as the base class constructor. Then I emit IL to load the local instance (i.e. ldarg 0) and all the parameters onto the stack, call the base constructor and finally return. Once all the constructors are defined, I can create the type and return.

Using the ClrTypeMetaclass is very easy – simply specify the __metaclass__ field in a class. If you want to customize the namespace, specify the _clrnamespace field as well. Here’s an example:

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"

  def __init__(self, name, cost, quantity):
    self.name = name
    self.cost = cost
    self.quantity = quantity

  def calc_total(self):
    return self.cost * self.quantity

You can verify this code has custom CLR metadata by calling GetType on a Product instance and inspecting the result via standard reflection techniques.

>>> m = Product('Crunchy Frog', 10, 20)
>>> m.GetType().Name
'Product'
>>> m.GetType().FullName
'DevHawk.IronPython.ClrTypeSeries.Product'

Great, so now I have a custom CLR type for my Python class. Unfortunately, at this point it’s pretty useless. Next, I’m going to add instance fields to the CLR type.

__clrtype__ Metaclasses: Adding CLR Fields

Now that we have the basic __clrtype__ metaclass infrastructure in place, let’s enhance it to add support for CLR fields. To do this, we’re going to need to add two things to our custom CLR type. First, we need to define the fields themselves. Second, we need to make sure that Python code will read and writes to the statically typed fields for the specified names rather than the storing them in the object dictionary as usual. Here’s the updated version of ClrTypeMetaclass (or you can get it from my SkyDrive)

class ClrTypeMetaclass(type):
  def __clrtype__(cls):
    baseType = super(ClrTypeMetaclass, cls).__clrtype__()
    typename = cls._clrnamespace + "." + cls.__name__
                 if hasattr(cls, "_clrnamespace")
                 else cls.__name__

    typegen = Snippets.Shared.DefineType(typename, baseType, True, False)
    typebld = typegen.TypeBuilder

    for ctor in baseType.GetConstructors():
      ctorparams = ctor.GetParameters()
      ctorbld = typebld.DefineConstructor(
                  ctor.Attributes,
                  ctor.CallingConvention,
                  tuple([p.ParameterType for p in ctorparams]))
      ilgen = ctorbld.GetILGenerator()
      ilgen.Emit(OpCodes.Ldarg, 0)
      for index in range(len(ctorparams)):
        ilgen.Emit(OpCodes.Ldarg, index + 1)
      ilgen.Emit(OpCodes.Call, ctor)
      ilgen.Emit(OpCodes.Ret)

    if hasattr(cls, "_clrfields"):
      for fldname in cls._clrfields:
        typebld.DefineField(
          fldname,
          clr.GetClrType(cls._clrfields[fldname]),
          FieldAttributes.Public)

    new_type = typebld.CreateType()

    if hasattr(cls, "_clrfields"):
      for fldname in cls._clrfields:
        fldinfo = new_type.GetField(fldname)
        setattr(cls, fldname, ReflectedField(fldinfo))

    return new_type

All the base type, type name, type builder and constructor code in the first half of the __clrtype__ method is the same as last time, so we’ll focus on the second half. After emitting the constructor(s), next we iterate thru a dictionary named _clrfields (if it exists in the class) that maps field names to types. For each of these dictionary entries, we emit a public field on the CLR type with the specified name and type.

The first time I tried this, I simply added the custom field generation code I just described and left it at that. Didn’t work. Python doesn’t look to store information in fields defined by the static type metadata unless explicitly instructed to. That’s why I need to iterate over the declared list of fields a second time after the type has been created. The first time creates the CLR fields, the second time inserts a ReflectedField instance into the class dictionary. ReflectedField is a Python descriptor that reads and writes the field value by calling GetValue and SetValue on the contained FieldInfo object. Python uses the same name resolution for fields as it does for method (In Python, methods are fields that store callable objects) so when IronPython discovers the ReflectedField descriptor in the class instance, it uses that to get or store the value rather than sticking it in the local dictionary.

Now here’s the new version of the Product class, this time with CLR fields as well as a custom type name:

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrfields = {
    "name":str,
    "cost":float,
    "quantity":int,
    }

  def __init__(self, name, cost, quantity):
    self.name = name
    self.cost = cost
    self.quantity = quantity

  def calc_total(self):
    return self.cost * self.quantity

As you can see, the only thing that’s changed is the addition of the _clrfields dictionary. But now, we can use reflection to get and set the Product fields, like so:

>>> p = Product("Crunchy Frog", 5.99, 10)
>>> t = p.GetType()
>>> p.name
'Crunchy Frog'
>>> namefi = t.GetField("name")
>>> namefi.GetValue(p)
'Crunchy Frog'
>>> namefi.SetValue(p, "Spring Surprise")
>>> p.name
'Spring Surprise'

This is great progress, but not enough to get us to our first “real” scenario: data binding in Silverlight. Silverlight only supports data binding against public properties, so I’ll need to wrap all these CLR fields in CLR properties in my next post.

__clrtype__ Metaclasses: Adding CLR Properties

When I was first experimenting with __clrtype__, I got to the point of making CLR fields work and then immediately tried to do some data binding with Silverlight. Didn’t work. Turns out Silverlight can only data bind against properties – fields aren’t supported. So now let’s add basic property support to ClrTypeMetaclass. Python has a rich mechanism for defining properties, but hooking that up requires DLR binders so for now I’m going to generate properties that are simple wrappers around the associated fields.

There’s enough code involved in defining a property to break it out into it’s own method:

@staticmethod
def define_prop(typebld, name, fieldtype, fieldbld):
    attribs = ( MethodAttributes.Public
              | MethodAttributes.SpecialName
              | MethodAttributes.HideBySig)
    clrtype = clr.GetClrType(fieldtype)

    getbld = typebld.DefineMethod("get_" + name, attribs, clrtype, None)
    getilgen = getbld.GetILGenerator()
    getilgen.Emit(OpCodes.Ldarg_0)
    getilgen.Emit(OpCodes.Ldfld, fieldbld)
    getilgen.Emit(OpCodes.Ret)

    setbld = typebld.DefineMethod("set_" + name, attribs, None, (clrtype,))
    setilgen = setbld.GetILGenerator()
    setilgen.Emit(OpCodes.Ldarg_0)
    setilgen.Emit(OpCodes.Ldarg_1)
    setilgen.Emit(OpCodes.Stfld, fieldbld)
    setilgen.Emit(OpCodes.Ret)

    prpbld = typebld.DefineProperty(name,
      PropertyAttributes.None, clrtype, None)
    prpbld.SetGetMethod(getbld)
    prpbld.SetSetMethod(setbld)

You provide define_prop the TypeBuilder for the Type being constructed, the name and type of the property as well as the FieldBuilder that gets returned from the call to DefineField. In the previous installment, I wasn’t bothering to save the FieldBuilder to a variable since I never used it again. Now, I’m stashing it away for the call to define_prop as I’ll show below.

For each field, we define a get method, a set method and a property. The get function first executes ldarg_0 to load the current object reference onto the execution stack, then it executes ldfld to load the specified field from the object onto the stack, then it returns. The set function executes ldarg_0 to load the current object reference and ldarg_1 to load the value passed as the first argument onto the execution stack, then it executes stfld to store the value in the specified field of the object. Once I have the two methods, I call DefineProperty to create the PropertyBuilder and then associate the get and set methods with that property.

As I said before, Reflection.Emit is straightforward though tedious. Honestly, I didn’t go thru the Emit docs to figure out what the methods should look like. Instead, I wrote a basic wrapper property in C# and looked at the generated IL in Reflector.

The only other change here is adding the call to define_prop on our first iteration thru list of _clrfields. Since the rest of __clrtype__ is the same, here’s just that code snippet:

if hasattr(cls, "_clrfields"):
    for fldname in cls._clrfields:
        fieldtype = clr.GetClrType(cls._clrfields[fldname])
        fieldbld = typebld.DefineField(fldname, fieldtype, FieldAttributes.Public)
        ClrTypeMetaclass.define_prop(typebld, fldname, fieldtype, fieldbld)

As I said above, I simply save off the result of calling DefineField so I can pass it to define_prop. I also save off the field type in a variable since I use it more than once. Avoids the second dictionary lookup and is clearer to understand what the function does.

Accessing the CLR properties via reflection is pretty straightforward – not very different than reflecting over CLR fields. The only significant difference between them is that CLR properties can be indexable and fields can’t, so you have to pass an index parameter to GetValue and SetValue. These aren’t indexed properties, so I pass in None for the index parameter.

>>> p = Product("Crunchy Frog", 10, 12)
>>> pi = p.GetType().GetProperty("name")
>>> pi.GetValue(p, None)
'Crunchy Frog'
>>> pi.SetValue(p, "Spring Surprise", None)
>>> pi.GetValue(p, None)
'Spring Surprise'
>>> p.name
'Spring Surprise'

One quick aside about the CLR type I’m generating here. I’m fairly certain this reflected object wouldn’t pass muster with the C# compiler. I’m defining a field and a property with the same name. It clearly works at the IL level, but I’m not sure what the C# compiler would do if you tried to refer to a CLR type like this. I should probably be prepending an underscore or something on the field name, but then I wonder if the field should also be private. There’s a whole API design discussion down that road, but I’m not quite ready to have that yet so I’m just leaving the fields public and having fields and properties with the same name. Luckily, I’m never generating a CLR type on disk so you can’t build a C# project that refers to it anyway.

__clrtype__ Metaclasses Demo: Silverlight Databinding

I’ve gotten to the point where I can actually demo something interesting with __clrtype__ metaclasses: Silverlight Databinding. This is a trivial sample, data binding a list of Products (aka the sample class I’ve been using all week) to a list box. But according to Jimmy, this is something he gets asked about on a regular basis and there’s a AgDLR bug open for this. The __clrtype__ feature is specific to IronPython but I bet the IronRuby guys could implement something similar if they wanted to.

When you install IronPython 2.6 (or 2.0.1 for that matter), it comes with the AgDLR bits in the Silverlight subfolder. This includes Silverlight compatible versions of the DLR and IronPython as well as the Silverlight DLR host and the development web server Chiron in the Silverlightbin directory. There is also a script in the Silverlightscript directory that will generate a dynamic Silverlight application from a template. I ran “sl.bat python sldemo” in order to build the skeleton project.

In the generated app.xaml file, I removed the default text box and replaced it with this XAML code that I stole nearly-verbatim from my blog post on data binding in WPF with IronPython. The only thing I changed was the binding path for the text block (title became name).

<ListBox x:Name="listbox1" >
  <ListBox.ItemTemplate>
    <DataTemplate>
      <TextBlock Text="{Binding Path=name}" />
    </DataTemplate>
  </ListBox.ItemTemplate>
</ListBox>

Then in the App class, I set the ItemsSource of the ListBox to a hand-built a list of Products.

class App:
  def __init__(self):
    root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    root.listbox1.ItemsSource = [
      Product("Crunchy Frog", 10, 12),
      Product("Rams Bladder Cup", 10, 12),
      Product("Cockroach Cluster", 10, 12),
      Product("Anthrax Ripple", 10, 12),
      Product("Spring Suprise", 10, 12)]

And that’s pretty much it. I used Chiron’s /z command to create a Silverlight XAP file, uploaded it to Silverlight Streaming and embedded it right here in this post. Code is up on my skydrive as well. Uusing Silverlight Streaming for this app was very easy – basically upload the XAP file to their server and embed some iframe code in this post via the source view and that was it. I’m not sure I would use it for a production app, but it rocked for hosting this demo.

The XAP is a big download for such a trivial app – about 1.3MB. The vast majority of that is the DLR and IronPython assemblies. The XAP would only be 2.9kB if it was just the Python, XAML and manifest files. This kinda stinks, but there’s a new transparent platform extensions feature in Silverlight 3 so we can at least break the DLR and IronPython DLLs out into their own separate XAPs. That way they only get downloaded once and cached in the browser instead of being included in every single IronPython Silverlight application anyone creates.

So that’s one scenario down, one to go. In order to be able to build WCF services in IronPython, I have to add a lot more infrastructure – notably emitting CLR methods that can invoke dynamic methods as well as emitting custom attributes. Invoking dynamic methods means understanding DLR binders, so look for more posts on __clrtype__ next week.

__clrtype__ Metaclasses: Simple Custom Attributes

I know it’s been a while since my last __clrtype__ post, but I was blocked on some bug fixes that shipped as part of IronPython 2.6 Beta 1. So now let’s start looking at one of the most requested IronPython features – custom attributes!

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [ObsoleteAttribute]

  # remainder of class omitted for clarity

Python list comprehensions use the same square bracket syntax as C# properties, so it kinda looks right to someone with a C# eye – though having the attribute specifications inside the class, rather than above it, is totally different. I wish I could use Python’s class decorators for custom class attributes, but class decorators run after metaclasses so unfortunately that doesn’t work. Also, I can’t leave off the “Attribute” suffix like you can in C#. If I really wanted to, I could provide a new type name in the import statement (“from System import ObsoleteAttribute as Obsolete”) but I thought spelling it out was clearer for this post.

Now that I have specified the class attributes, I can update the metaclass __clrtype__ method to set the attribute on the generated CLR class:

if hasattr(cls, '_clrclassattribs'):
      for attribtype in cls._clrclassattribs:
        ci = clr.GetClrType(attribtype).GetConstructor(())
        cab = CustomAttributeBuilder(ci, ())
        typebld.SetCustomAttribute(cab)

I’m simply iterating over the list of _clrclassattribs (if it exists), getting the default parameterless constructor for each attribute type, creating a CustomAttributeBuilder instance from that constructor and then calling SetCustomAttribute. Of course, this is very simple because we’re not supporting any custom arguments or setting of named properties. We’ll get to that in the next post. In the mean time, you can get the full code for this post from my SkyDrive.

There is one significant issue with this custom attribute code. Attributes are typically marked with the AttributeUsage attribute that specifies a set of constraints, such as the kind of targets a given attribute can be attached to and if it can be specified multiple times. For example, the MTAThread attribute can’t be specified multiple times and it can only be attached to methods. However, those attribute constraints are validated by the compiler, not the runtime. I haven’t written any code yet to validate those constraints, so you can specify invalid combinations like multiple MTAThread attributes on a class. For now, I’m just going to leave it to the developer not to specify invalid attribute combinations. Custom attributes are passive anyway so I’m figure no one will come looking for a MTAThread attribute on a class or other such scenarios.

However, I’m interested in your opinion: When we get to actually productizing a higher-level API for __clrtype__, what kinds of attribute validation should we do, if any?

__clrtype__ Metaclasses: Positional Attribute Parameters

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

def make_cab(attrib_type, *args):
  argtypes = tuple(map(lambda x:clr.GetClrType(type(x)), args))
  ci = clr.GetClrType(attrib_type).GetConstructor(argtypes)
  return CustomAttributeBuilder(ci, args)

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [make_cab(ObsoleteAttribute , "Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

In make_cab, I build a tuple of CLR types from the list of positional arguments that was passed in. If you haven’t seed the *args syntax before, it works like C#’s params keyword – any extra arguments are passed into the function as a tuple names args. I use Python’s built in map function (FP FTW!) to build a tuple of CLR types of the provided arguments, which I then pass to GetConstructor. Previously, I passed an empty tuple to GetConstructor because I wanted the default constructor. If you don’t pass any positional arguments, you still get the default constructor. Once I’ve found the right constructor, I pass it and the original tuple of arguments to the CustomAttributeBuilder constructor.

One major benefit of this approach is that it simplifies the metaclass code. Since _clrclassattribs is now a list of CustomAttributeBuilders, now I just need to iterate over that list and call SetCustomAttribute for each.

if hasattr(cls, '_clrclassattribs'):
      for cab in cls._clrclassattribs:
        typebld.SetCustomAttribute(cab)

The only problem with this approach is that specifying the list of custom attributes is now extremely verbose. Not only am I specifying the full attribute class name as well as the positional arguments, I’m also having to insert a call to make_cab. Previously, it kinda looked like a C# custom attribute, albeit in the wrong place. Not anymore. So I decided to write a function called cab_builder to generates less verbose calls to make_cab:

def cab_builder(attrib_type):
  return lambda *args:make_cab(attrib_type, *args)

from System import ObsoleteAttribute
Obsolete = cab_builder(ObsoleteAttribute)

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [Obsolete("Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

The cab_builder function returns an anonymous lambda function that closes over the attrib_type variable. Python lambdas are just like C# lambdas, except that they only support expressions 1. The results of calling the lambda returned from cab_builder is exactly the same as calling make_cab directly, but less verbose. And since I named the function returned from cab_builder Obsolete, now my list of class custom attributes looks exactly like it does in C# (though still in a different place). As usual, the code is up on my SkyDrive.

If you’re only using the attribute once like this, it is kind of annoying to first declare the cab_builder function. If you wanted to you could iterate over the types in a given assembly, looking for ones that inherit from Attribute and generate the cab_builder call dynamically. However, I’m not sure how performant that would be. Another possibility would be to iterate over the types in a given assembly and generate a Python module on disk with the calls to cab_builder. Then, you’d just have to import this module of common attributes but still be able to include additional calls to cab_builder as needed.


  1. The lack of statement lambdas in Python is one of my few issues with the language.

__clrtype__ Metaclasses: Named Attribute Parameters

In my last post, I added support for custom attribute positional parameters . To finish things off, I need to add support for named parameters as well. Custom attributes support named parameters for public fields and settable properties. It works kind of like C# 3.0’s object initalizers. However, unlike object initalizers, the specific fields and properties to be set on a custom attribute as well as their values are passed to the CustomAttributeBuilder constructor. With six arguments – five of which are arrays – it’s kind of an ugly constructor. But luckily, we can hide it away in the make_cab function by using Python’s keyword arguments feature.

def make_cab(attrib_type, *args, **kwds):
  clrtype = clr.GetClrType(attrib_type)
  argtypes = tuple(map(lambda x:clr.GetClrType(type(x)), args))
  ci = clrtype.GetConstructor(argtypes)

  props = ([],[])
  fields = ([],[])

  for kwd in kwds:
    pi = clrtype.GetProperty(kwd)
    if pi is not None:
      props[0].append(pi)
      props[1].append(kwds[kwd])
    else:
      fi = clrtype.GetField(kwd)
      if fi is not None:
        fields[0].append(fi)
        fields[1].append(kwds[kwd])
      else:
        raise Exception, "No %s Member found on %s" % (kwd, clrtype.Name)

  return CustomAttributeBuilder(ci, args,
    tuple(props[0]), tuple(props[1]),
    tuple(fields[0]), tuple(fields[1]))

def cab_builder(attrib_type):
  return lambda *args, **kwds:make_cab(attrib_type, *args, **kwds)

You’ll notice that make_cab now takes a third parameter: the attribute type and the tuple of positional arguments we saw last post. This third parameter “**kwds” is a dictionary of named parameters. Python supports both positional and named parameter passing, like VB has for a while and C# will in 4.0. However, this **kwds parameter contains all the extra or leftover named parameters that were passed in but didn’t match any existing function arguments. Think of it like the params of named parameters.

As I wrote earlier, custom attributes support setting named values of both fields and properties. We don’t want the developer to have to know if given named parameter is a field or property, so make_cab iterates over all the named parameters, checking first to see if it’s a property then if it’s a field. It keeps a list of all the field / property infos as well as their associated values. Assuming all the named parameters are found, those lists are converted to tuples and passed into the CustomAttributeBuilder constructor.

In addition to the change to make_cab, I also updated cab_builder slightly in order to pass the **kwds parameter on thru to the make_cab function. No big deal. So now, I can add an attribute with named parameters to my IronPython class and it still looks a lot like a C# attribute specification.

clr.AddReference("System.Xml")
from System.Xml.Serialization import XmlRootAttribute
from System import ObsoleteAttribute, CLSCompliantAttribute
Obsolete = cab_builder(ObsoleteAttribute)
CLSCompliant = cab_builder(CLSCompliantAttribute)
XmlRoot = cab_builder(XmlRootAttribute)

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [
    Obsolete("Warning Lark's Vomit"),
    CLSCompliant(False),
    XmlRoot("product", Namespace="http://samples.devhawk.net")]

  # remainder of Product class omitted for clarity

As usual, sample code is up on my SkyDrive.

Now that I can support custom attributes on classes, it would be fairly straightforward to add them to methods, properties, etc as well. The hardest part at this point is coming up with a well designed API that works within the Python syntax. If you’ve got any opinions on that, feel free to share them in the comments, via email, or on the IronPython mailing list.