__clrtype__ Metaclasses: Positional Attribute Parameters

The basic infrastructure for custom attributes in IronPython is in place, but it’s woefully limited. Specifically, it only works for custom attributes that don’t have parameters. Of course, most of the custom attributes that you’d really want to use require additional parameters, both the positional or named variety. Since positional parameters are easier, let’s start with them.

Positional parameters get passed to the custom attribute’s constructor. As we saw in the previous post, you need a CustomAttributeBuilder to attach a custom attribute to an attribute target (like a class). Previously, I just needed to know the attribute type since I was hard coding the positional parameters. But now, I need to know both the attribute type as well as the desired positional parameters. I could have built a custom Python class to track this information, but it made much more sense just to use CustomAttributeBuilder instances. I built a utility function make_cab to construct the CustomAttributeBuilder instances.

def make_cab(attrib_type, *args):
  argtypes = tuple(map(lambda x:clr.GetClrType(type(x)), args))
  ci = clr.GetClrType(attrib_type).GetConstructor(argtypes)
  return CustomAttributeBuilder(ci, args)

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [make_cab(ObsoleteAttribute , "Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

In make_cab, I build a tuple of CLR types from the list of positional arguments that was passed in. If you haven’t seed the *args syntax before, it works like C#’s params keyword – any extra arguments are passed into the function as a tuple names args. I use Python’s built in map function (FP FTW!) to build a tuple of CLR types of the provided arguments, which I then pass to GetConstructor. Previously, I passed an empty tuple to GetConstructor because I wanted the default constructor. If you don’t pass any positional arguments, you still get the default constructor. Once I’ve found the right constructor, I pass it and the original tuple of arguments to the CustomAttributeBuilder constructor.

One major benefit of this approach is that it simplifies the metaclass code. Since _clrclassattribs is now a list of CustomAttributeBuilders, now I just need to iterate over that list and call SetCustomAttribute for each.

if hasattr(cls, '_clrclassattribs'):
      for cab in cls._clrclassattribs:
        typebld.SetCustomAttribute(cab)

The only problem with this approach is that specifying the list of custom attributes is now extremely verbose. Not only am I specifying the full attribute class name as well as the positional arguments, I’m also having to insert a call to make_cab. Previously, it kinda looked like a C# custom attribute, albeit in the wrong place. Not anymore. So I decided to write a function called cab_builder to generates less verbose calls to make_cab:

def cab_builder(attrib_type):
  return lambda *args:make_cab(attrib_type, *args)

from System import ObsoleteAttribute
Obsolete = cab_builder(ObsoleteAttribute)

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [Obsolete("Warning Lark's Vomit")]

  # remaining Product class definition omited for clarity

The cab_builder function returns an anonymous lambda function that closes over the attrib_type variable. Python lambdas are just like C# lambdas, except that they only support expressions 1. The results of calling the lambda returned from cab_builder is exactly the same as calling make_cab directly, but less verbose. And since I named the function returned from cab_builder Obsolete, now my list of class custom attributes looks exactly like it does in C# (though still in a different place). As usual, the code is up on my SkyDrive.

If you’re only using the attribute once like this, it is kind of annoying to first declare the cab_builder function. If you wanted to you could iterate over the types in a given assembly, looking for ones that inherit from Attribute and generate the cab_builder call dynamically. However, I’m not sure how performant that would be. Another possibility would be to iterate over the types in a given assembly and generate a Python module on disk with the calls to cab_builder. Then, you’d just have to import this module of common attributes but still be able to include additional calls to cab_builder as needed.


  1. The lack of statement lambdas in Python is one of my few issues with the language.

__clrtype__ Metaclasses: Simple Custom Attributes

I know it’s been a while since my last __clrtype__ post, but I was blocked on some bug fixes that shipped as part of IronPython 2.6 Beta 1. So now let’s start looking at one of the most requested IronPython features – custom attributes!

Over the course of the next three blog posts, I’m going to build out a mechanism for specifying custom attributes on the CLR type we’re generating via __clrtype__. All the various Builder classes in System.Reflection.Emit support a SetCustomAttribute method that works basically the same way. There are two overloads – the one I’m going to use takes a single CustomAttributeBuilder as a parameter.

For this first post, I’m going to focus on the basic custom attribute infrastructure, so we’re going to use the extremely simple ObsoleteAttribute. While you can pass some arguments to the constructor, for this first post I’m going to use the parameterless constructor. To keep things less confusing, I’m going back to the original version of the Product class, before I introduced CLR fields and properties. The one change I’m making is that I’m adding a list of attributes I want to add to the class.

from System import ObsoleteAttribute

class Product(object):
  __metaclass__ = ClrTypeMetaclass
  _clrnamespace = "DevHawk.IronPython.ClrTypeSeries"
  _clrclassattribs = [ObsoleteAttribute]

  # remainder of class omitted for clarity

Python list comprehensions use the same square bracket syntax as C# properties, so it kinda looks right to someone with a C# eye – though having the attribute specifications inside the class, rather than above it, is totally different. I wish I could use Python’s class decorators for custom class attributes, but class decorators run after metaclasses so unfortunately that doesn’t work. Also, I can’t leave off the “Attribute” suffix like you can in C#. If I really wanted to, I could provide a new type name in the import statement (“from System import ObsoleteAttribute as Obsolete”) but I thought spelling it out was clearer for this post.

Now that I have specified the class attributes, I can update the metaclass __clrtype__ method to set the attribute on the generated CLR class:

if hasattr(cls, '_clrclassattribs'):
      for attribtype in cls._clrclassattribs:
        ci = clr.GetClrType(attribtype).GetConstructor(())
        cab = CustomAttributeBuilder(ci, ())
        typebld.SetCustomAttribute(cab)

I’m simply iterating over the list of _clrclassattribs (if it exists), getting the default parameterless constructor for each attribute type, creating a CustomAttributeBuilder instance from that constructor and then calling SetCustomAttribute. Of course, this is very simple because we’re not supporting any custom arguments or setting of named properties. We’ll get to that in the next post. In the mean time, you can get the full code for this post from my SkyDrive.

There is one significant issue with this custom attribute code. Attributes are typically marked with the AttributeUsage attribute that specifies a set of constraints, such as the kind of targets a given attribute can be attached to and if it can be specified multiple times. For example, the MTAThread attribute can’t be specified multiple times and it can only be attached to methods. However, those attribute constraints are validated by the compiler, not the runtime. I haven’t written any code yet to validate those constraints, so you can specify invalid combinations like multiple MTAThread attributes on a class. For now, I’m just going to leave it to the developer not to specify invalid attribute combinations. Custom attributes are passive anyway so I’m figure no one will come looking for a MTAThread attribute on a class or other such scenarios.

However, I’m interested in your opinion: When we get to actually productizing a higher-level API for __clrtype__, what kinds of attribute validation should we do, if any?

IronPython 2.6 Beta 1

In addition to the IronPython CTP for .NET Framework 4.0 Beta 1 I blogged about earlier, we also released the first beta of IronPython 2.6 today. How about that – two IronPython releases in one day! This is our second preview release as we work towards our 2.6 RTM in September. 2.6 Alpha 1 was released back in March.

There are two big new features in this release. The first is our implementation of the ctypes module. The ctypes module is like P/Invoke for Python. It allows Python code to call into unmanaged DLL functions. Here, for example, I’m calling into the standard wprintf function from msvcrt.dll

IronPython 2.6 Beta 1 (2.6.0.10) on .NET 2.0.50727.4918
>>> import ctypes
>>> libc = ctypes.cdll.msvcrt
>>> ret = libc.wprintf("%sn", "hello")
hello

Between ctypes and Ironclad, I think we’ll eventually be able to load most native Python extensions in IronPython. Woot!

The other big new feature in this release is a real implementation of sys._getframe. _getframe lets you write code that inspects the Python callstack. Previously, we supported _getframe only with a depth of zero which is to say you could inspect the current frame, but no others. Now, by default we don’t implement _getframe at all unless you pass in –X:Frames or –X:FullFrames on the command line. Removing the version of _getframe that only worked for depth zero fixes an issue with collections.py that broke much of the 2.6 standard library in IronPython 2.6 Alpha 1.

The difference between Frames and FullFrames is in what is returned by frame.f_locals member. If you’re running with FullFrames, we hoist all local variables into the heap so they can be accessed by our frame walker. If you’re running with Frames, our ability to access locals up the stack is limited. Sometimes they are available – If you called locals() in a frame up the stack for example, then f_locals will be available – but usually not. There’s a performance difference between the default (i.e. no Frames), –X:Frames and –X:FullFrames, hence why we provide the user fine grained control over the Frame support.

Our performance has gotten better relative to 2.6 Alpha 1. Our PyStone numbers have improved 80% from Alpha 1, similar to where we were in IronPython 2.0.1. We’ve also been able to cut our startup time about 25% from 2.0.1. We’re still an order of magnitude slower than CPython on startup, but we’re getting better. We’re significantly worse on PyBench than we were in 2.6 Alpha 1, but that’s primarily because there’s now a second exception test. As I described back in March, we get killed on the exceptions benchmarks – the two combine to consume nearly 62% of our total run time. Ouch!

Finally, there are bug fixes. Of particular relevance to readers of this blog are a series of fixes that allow me to continue on with my __clrtype__ series. Watch for that soon.

As I said back when we released Alpha 1, the release cycle on 2.6 will be much shorter than it was for 2.0. 2.0 had eight alphas, five betas and two release candidates over the course of around twenty months. We expect 2.6 to have one alpha, two betas and a release candidate over eight months. So please start trying using the beta as soon as you can so you can give us your feedback and we can fix your bugs!

IronPython 2.6 CTP for .NET 4.0 Beta 1

The .NET Framework 4.0 and Visual Studio 2010 Beta 1 is now generally available for download. Jason Zander has a very thorough rundown on some of the new features in this release. Of course, my favorite new features in VS2010 is the new dynamic language support in C# and Visual Basic, which let’s you easily call out to IronPython code from those languages.

For anyone who wants to experiment with interoperating C# or VB with IronPython, we released IronPython 2.6 CTP for .NET 4.0 Beta 1 today. There’s also a walkthru showing how you can use the standard Python library module random from both C# and VB. Note, there’s currently a URL bug in that walkthru – it links to IronPython 2.6 Alpha 1 rather than the .NET 4.0 Beta 1 IronPython CTP. Make sure you pick up the right version of IronPython if you want to try out the walkthru. Looks like they fixed the redirect in the walkthru.

FYI, this is a CTP quality release with about the same functionality as IronPython 2.6 Alpha 1.  Essentially, this is the version of IronPython that was in the source tree when the VS team branched for Beta 1.

If you’ve got any feedback, please drop us a line on the mailing list.

Checkin Comments for IronPython Source

We’ve been slowly but surely increasing the frequency of IronPython source drops. When I joined the team last April, we we only pushing the source about twice a month (sometimes only once a month). By last July, we were pushing source about once a week. Since mid-January, we’ve pushed out the latest source 131 times, which comes to about once a day on average since the start of the year. Big kudos to Dave Fugate, who’s primarily responsible for improving the frequency of our source code drops.

However, while we’ve been good about source code drop frequency, we haven’t been good about transparency. All those source drops have the same less-than-useful checkin comment “Latest IP sources migrated to CodePlex TFS”. If you wanted to know what was changed in a given changeset, you had to do the diff yourself.

But all that opaque code changes is a thing of the past now. Dave upgraded out source push script so that it emails a list of changes as well as the checkin comments whenever we update the source on CodePlex. For example, check out the source push announcement for our latest source drop.  Now we publish added, deleted and modified sources as well as the comments for any checkins included in the source drop.

As Dave said on the mailing list, please let us know if you have any feedback on these source update emails. I think they’re awesome (though I did have one small suggestion) but we want to know what you think.