__clrtype__ Metaclasses: IronPython Classes Under the Hood

Before we start using __clrtype__ metaclasses, we need to understand a bit about how IronPython maps between CLR types and Python classes. IronPython doesn’t support Reflection based APIs or custom attributes today because IronPython doesn’t emit a custom CLR types for every Python class. Instead, it typically shares a single CLR type across many Python classes. For example, all three of these Python classes share a single underlying CLR type.

class shop(object):
  pass

class cheese_shop(shop):
  def have_cheese(self, cheese_type):
    return False

class argument_clinic(object):
  def is_right_room(self, room=12):
    return "I've told you once"

import clr
print clr.GetClrType(shop).FullName
print clr.GetClrType(cheese_shop).FullName
print clr.GetClrType(argument_clinic).FullName

Even though cheese_shop inherits from shop and argument_clinic inherits from object, all three classes share the same underlying CLR type. On my machine, running IronPython 2.6 Alpha 1, that type is named IronPython.NewTypes.System.Object_1$1.

IronPython can share the CLR type across multiple Python classes because that CLR type has no code specific to a given Python class. CLR types are immutable – once you build a CLR type, you can’t do things like add new methods, remove existing method or change the inheritance hierarchy. But all those things are legal to do in Python. Here, I’m creating an instance of the cheese_shop class, but then changing that instance to be an argument_clinic instance instead.

>>> cs = cheese_shop()

>>> cs.have_cheese("Venezuelan Beaver Cheese")
False
>>> cs.is_right_room(12)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'cheese_shop' object has no attribute 'is_right_room'

>>> # Change the object's class at runtime
>>> cs.__class__ = argument_clinic # don't try this in C#!

>>> cs.have_cheese("Venezuelan Beaver Cheese")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'argument_clinic' object has no attribute 'have_cheese'
>>> cs.is_right_room(12)
"I've told you once"

When you call a method on a Python object, the name is resolved by walking a series of dictionaries. First, the dictionary of the object itself is searched for the method name. Assuming the name isn’t in the object dictionary, Python then looks in the __class__ dictionary. If it’s not there, Python recursively looks through the base classes stored in the __bases__ tuple until it finds the method or the name fails to resolve. If we re-assign __class__ at run time, we change the dictionary Python uses to resolve method names.

There are cases where IronPython generates a new underlying CLR type. For example, if you build a Python class that inherits from a CLR type, then IronPython will have to generate a new underlying CLR type that inherits from the CLR type in order to remain compatible. IronPython automatically overrides all the virtual methods of the base type, implementing the same dynamic method dispatch that I described above. This lets you pass the IronPython class wherever the base CLR type is expected.

The ability to swap Python classes at runtime depends on having the same underlying CLR type. If the underlying CLR type doesn’t match, then assigning a new value to the __class__ field of an object will fail. This applies both to IronPython classes that inherit from CLR types as well as __clrtype__ metaclass types. In the code I’ll be blogging, I always generate a unique CLR type for every Python class, which means that I can’t dynamically retype the object. Given that the point of __clrtype__ metaclasses is to generate static type information, this hardly seems like a limitation. However, it’s something to be aware of as we explore the __clrtypes__ feature.

The Lounge Survey

I just joined The Lounge advertising network so I wanted to pass along an opportunity to win a bunch of great technical books.

The Lounge is asking the readers of the blogs in their network to fill out a survey in order for them to improve how they target their advertising. It’s pretty much what you would expect from an advertising network focused on the .NET development platform: what language(s) do you use, what framework(s), what testing tool(s), etc, etc, etc. Takes like three minutes to fill out at most.

We all know that filling out surveys isn’t what most people consider “exciting” or “fun”. In order to incent you, dear Reader, to take a few minutes of your valuable time to fill out the survey, The Lounge is giving away all forty one of Manning’s “In Action” books, including IronPython In Action. Even if you don’t win, you still get 40% discount off any purchase from Manning.

So it’s up to you, a scant few minutes of your time in exchange for a chance to win enough technical books to keep you busy for months.

Introducing __clrtype__ Metaclasses

Everyone knows Anders announced at PDC08 that C# 4.0 will include new features (aka the dynamic keyword + the DLR) that makes it much easier for C# to call into dynamically typed code. What you probably don’t know is that IronPython 2.6 includes a new feature that makes it easier for IronPython code to be called by statically typed code.

While the vast majority of .NET is available to IronPython, there are certain APIs that just don’t work with dynamic code. In particular, any code that uses Reflection over an object’s CLR type metadata won’t work with IronPython. For example, while WPF supportsICustomTypeDescriptor, Silverlight only supports data binding against reflectable properties. Furthermore, any code that uses custom attributes inherently uses Reflection. For example, Darrel Hawley recently blogged a WCF host he wrote in IronPython, but he wrote the WCF service in C#. You can’t write WCF services in IronPython because WCF expects service classes to be adorned with ServiceContract and OperationContract attributes (among many others). IronPython users want access to use these APIs. Support for custom attributes is one of the most common requests we get – it’s currently the 5th highest vote getter among open issues.

In IronPython 2.6, we’re adding the ability to customize the CLR type of Python classes. This means you can add custom attributes, emit properties, whatever you want. For those of you who’ve been dreaming of implementing WCF services or databinding in Silverlight purely in IronPython, then this is the feature for you.

In a nutshell, IronPython 2.6 extends Python’s metaclass feature that lets you to customize the creation of classes. In the metaclass, you can implement an IronPython-specific method __clrtype__ which returns a custom System.Type of your own creation that IronPython will then use as the underlying CLR type of the Python class. Implementing __clrtype__ gives you the chance to implement whatever reflectable metadata you need: constructors, fields, properties, methods, events, custom attributes, nested classes, whatever.

Over a series of posts, I’ll be demonstrating this new feature and implement some common scenario requests – including Silverlight databinding and WCF services – purely in Python. Quick warning: __clrtype__ uses low level features like Python metaclasses, Reflection.Emit and DLR Binders so these posts will be deeper technically than usual. Don’t worry – this isn’t the API interface we expect everyone to use. Eventually, we want to have an easy to use API that will sit on top of the low-level __clrtype__ hook.

Issue Tracking for my GitHub Projects

FYI, anyone using ipydbg, devhawk_ipy or Pygments for WL Writer can now submit issues, bugs and feature requests other the associated GitHub project page. Head to the associated GitHub project page and click the “Issues” tab at the top. There’s a short video of introducing the GitHub Issues feature you can check out.

BTW, thanks to defunkt @ GitHub for the ipydbg shoutout.

Joining the Lounge Advertising Network

For those of you who read this blog primarily via my RSS feed, I made a change to my homepage over the weekend. Goodbye adSense, hello The Lounge. The Lounge is an ad network run by James Avery’s company InfoZerk. I’ve known James for a while – he included my now-obsolete SccSwitch tool in his book Visual Studio Hacks.

From a financial perspective, I’m not really sure how much of a difference this will make. I guess I’ll see when I get my first check. Given how little I was making with adSense, I’ve got nowhere to go but up. Regardless, I feel much better working with a smaller ad network run by someone I respect and that is focused on the .NET platform.

Thanks for having me in the Lounge ad network, James.