As expected, my new role is going to involve much more travel than my old role. Here’s a list of all the places I’m going / have been this year. I’ll be updating this post periodically as I get tapped for more presentations. There are several other conferences I’m considering, submitting sessions for, in discussions with, but these are the ones that are confirmed.

DevDays 2008
May 22nd – 23rd, Amsterdam, The Netherlands

This is a kinda last minute pickup. My boss was originally scheduled to do this. Or should I say, my ex-boss? (I’ve been here less than two months and already scared off my boss! 😄 I’m doing two talks, but I only have the abstract for one of them.

DEV315 – “IronPython” and Dynamic Languages on .NET
“IronPython” is the codename for a new implementation of the Python programming language on the .NET Framework. IronPython is fast—in fact, up to 1.8 times faster than Python-2.4 on the standard pystone benchmark. It supports an interactive interpreter with fully dynamic compilation as well as static compilation to produce pre-compiled executables. It’s well integrated with the rest of the framework and makes all .NET libraries easily available to Python programmers. This session shows how IronPython brings the power of .NET to Python and the power of Python to .NET. At OSCON 2004, the first public release of IronPython was announced. This session demonstrates the latest IronPython version in a range of situations from using GUI frameworks to driving Microsoft Office applications to working with a variety of external libraries. We also discuss other scripting languages on .NET.

I’m also going to do a talk on Dynamic Languages in Web Development, focused on Silverlight and ASP.NET. I got asked to do a second session at the last minute (technically, later than the last minute) so this one has no abstract.

TechEd New Zealand and Australia
Sept 1st – 5th, Auckland, New Zealand and Sydney, Australia

I did TechEd NZ & Australia back in 2004 and had a blast, so I’m looking forward to coming back this year. Content isn’t locked down yet, but I’m looking to do both a dynamic languages session as well as an architecture session. There’s also rumor of a web futures panel discussion that I’ll be participating in.

PDC08
Oct 27th – 30th, Los Angeles, California

I’m not doing a session, but I’m helping drive PDC content for my group, so I’m assuming I’ll be at the conference in some capacity. I’m thinking we need a dynamic language open space session.

patterns & practices Summit 2008
Nov 3rd-7th, Redmond, Washington

I really enjoy p&p Summit because Keith lets me experiment with somewhat off the wall sessions like “Developer 2.0” and “Moving Beyond Industrial Software”. Frankly, I have NO idea what I’m going to do at this years’ Summit, but I’m looking to stay outside the box like I have in the past.

In case I’m not the last person on the planet to figure this out…

1. In VS, click on File->Open->Project/Solution or press Ctl-Shift-O
2. Select ipy.exe from wherever you put it
3. Right click ipy.exe in Solution Explorer and select Properties
4. In the Command Arguments box, type “-D” (to generate debug code) and the full path to the script you want to execute. If you want to drop into interactive mode after the script executes, also include a “-i”
5. Open the script you specified in step 4 and place breakpoints as usual
6. Run via Debug->Start Debugging or press F5

Thanks Srivatsn for helping me out with this.

Now that I can stream process XML, the next logical step is to deserialize it into some type of object graph. As I said in my last post, there are at least three different DOM-esque options on the .NET platform as well as two in the Python library (xml.dom and xml.minidom)

However, anyone who’s ever programmed against the DOM knows just what a major PITA it is.

Instead, you could deserialize the XML into a custom object tree, based on the nodes in the XML stream. In .NET, there are at least two libraries for doing this: the old-school XmlSerializer as well as the new-fangled DataContractSerializer. In these libraries, the PITA comes in defining the static types with all the various custom attribute adornments you need to tell the deserializer how to do it’s job. Actually, if you’re defining your code first, all those adornments aren’t that big a deal. However, if you’re starting from the XML, especially XML with lots of different namespaces – like say my RSS feed – defining a static type for this gets old fast.

Of course, if you’re not using a statically typed language… 😉

One of the cool aspects of dynamic languages is the ability to easily generate new types on the fly. In Python, you can create a new type by calling the type function. Here’s an example of creating a new type for a XML node:

def create_type(node, parent):  
  return type(node.name, (parent,), {'xmlns':node.namespace})

Since I’m working with XML, I wanted to make sure I handled namespaces. Thus, I add the namespace to the class definition (the third parameter in the type function above). This lets me walk up to any arbitrary object created from an XML element and check it’s namespace.

I used this dynamic type creation functionality in my xml2py module, which I added to my IronPython SkyDrive folder. It leverages ipypulldom, so make sure you get both. The heart of the module is the xml2py function, which recursively iterates thru the node stream and builds the tree. Attributes and child elements become named attributes on the object, so I can write code that looks like this:

import xml2py  
rss = xml2py.parse('http://feeds.feedburner.com/Devhawk')  
for item in rss.channel.item:  
  print item.title

You see? No screwing around with childNodes or getAttribute here.

The basic processing loop of xml2py creates a new instance of a new type when it encounters a start element tag. It then collects all the attributes and children of that element, and adds them as attributes on the element object, using the name of the type as name of the attribute. If there are multiple children with the same type name, xml2py converts that attribute to a list of values. For example, in an RSS feed, there will be likely be many rss.channel.item elements. In xml2py, the item attribute of the channel object will be a list of item objects.

Since attributes and child elements are getting slotted together, I added a _nodetype attribute on each so I can later tell (if I care) if the value was originally an attribute or element. I haven’t written py2xml yet, but that might be important then.

I do one optimization for simple string elements like <foo>bar</foo>. In this case, I create a type that inherits from string (hence the need for the parent parameter in the create_type function above) and contains the string text. It still has the xmlns and _nodetype attributes, so I could write item.title.xmlns (which is empty since RSS is in the default namespace) or item.title._nodetype (which would be XmlNodeType.Element)

It’s not much code – about 100 lines of code split evenly between the xml2py function and the _type_factory object. Given that you usually see the same element in an XML stream over an over, I didn’t want to create multiple types for the same element. So _type_factory caches types in a dictionary so I can reuse them. One of the cool things is that it’s a callable type (i.e. it implements __call__ so I can use the instance like a function. I started by defining a xtype function that didn’t cache anything, but then later switched xtype to be a _type_factory instance, but none of my code that called xtype had to change!

One other quick note. If you put xml2py.py and ipypylldom.py in a folder, you can experiment with them by launching ipy -i xml2py. This runs xml2py.py as a script, but dumps you into the interactive console when you’re thru. It will run the little snippet of code above which runs xml2py on my FeedBurner feed, but then you can play around with the rss object and see what it contains. Be sure to check out the xmlns attribute for each object in the rss.channel.link list.

When it comes to processing XML, there are two basic approaches – load it all into memory at once or process it a node at a time. In the .NET world where I have spent most of the past ten years, those two models are represented by XmlDocument and XmlReader. There are alternatives to XmlDocument, such as XDocument and XPathDocument, but you get the idea.

Out in non-MSFT land, the same two basic models exist, however the de facto standard for stream based processing is SAX, the Simple API for XML. SAX is supported by many languages, including Python.

Personally, I’ve never been a fan of SAX’s event-driven approach. Pushing events makes total sense for a human driven UI, but I never understood why anyone thought that was a good idea for stream processing XML. I like XmlReader’s pull model much better. When you’re ready for the next node, just call Read() – no mucking about setting content handlers or handling node processing events.

Luckily, the Python standard library supports both approaches. It provides both a SAX based parser as well as a pull based parser called pulldom. Pulldom doc’s are fairly sparse, but Paul Prescod wrote a nice introduction. Here’s an example from Paul’s site (slightly modified):

from xml.dom import pulldom
nodes = pulldom.parse( "file.xml" )  
for (event,node) in nodes:  
    if event=="START_ELEMENT" and node.tagName=="table":  
        nodes.expandNode( node )

Actually, I like this better than XmlReader, since it provides the nodes in a list-like construct that appeals to the functional programmer in me. I’d like it even more if Python had a native pattern matching syntax – you know, like F# – but you can get similar results by chaining together conditionals with elif.

However, IronPython doesn’t support any of the XML parsing modules from Python’s standard library. They’re all based on a C-based python module called pyexpat which IronPython can’t load. ¹ I wanted a pulldom type model, so I decided to wrap XmlReader to provide a similar API and lets me write code like this:

import ipypulldom  
nodes = ipypulldom.parse( "sample.xml" )
for node in nodes:
  if node.nodeType==XmlNodeType.Element:
    print node.xname

There are a few differences from pulldom, but it’s basically the same model. I’m using the native .NET type XmlNodeType rather than a string to indicate the node type. Furthermore, I made the node type a property of the node, rather than a separate variable. I also didn’t implement expandNode, though doing so would be a fairly straightforward combination of XmlReader.ReadSubtree and XmlDocument.Load.

I stuck the code for ipypulldom up in a new folder on my Skydrive: IronPython Stuff. It’s fairly short – only about 45 lines of code. Feel free to use it if you need it.

An email conversation between two members of the IPy team that you, dear reader, might find amusing.

From:Srivatsn Narayanan
Sent: Friday, May 02, 2008 4:00 PM
Subject: You dont need pinky fingers? 😄

http://twitter.com/DevHawk/statuses/802239164

---

From:Dino Viehland
Sent: Friday, May 02, 2008 11:30 PM

Wow, am I really that loud? 😄

This is a response to the idea that no design is complete until you can’t remove anything else without sacrificing the overall goals of the end result.  Is the pinky really necessary?  We can “cut that feature”, right? 😄

---

From: Srivatsn Narayanan
Sent: Friday, May 02, 2008 11:49 PM

If you cut off the pinkie, then the ring finger becomes the pinkie and so iteratively you will have to lose all of them 😃. If I remember right in Europe, you hold up the pinkie to order beer. So it’s quite important 😄

---

From: Dino Viehland
Sent: Friday, May 02, 2008 11:59 PM

Well I find it hard to argue with ordering beer, but I doubt the pinkie’s original use was to order beer.  Therefore I’d describe this as an abuse of an existing API that might be better served by a new API specifically designed for this purpose.  But I question even if a new API is necessary.

Drinking beer by definition involves a container in which the beer is consumed from.  Therefore the container it’s self (or the lack thereof) might be able to serve the dual role for both consuming the beer and indicating the need for a refill.  For example when drinking sake I’ve been told it is customary to tip the bottle on its side to indicate the desire for a refill.  Are we really to expect that the Human API exposes every need in a direct fashion?  For example should I have another appendage for “I need more bread sticks” and yet another for “Can I get a refill or on the water?”.  I’d propose the “2 creams no sugar” appendage because that’s the way I take my coffee.

On the other hand the iterative problem does seem to be more relevant – but it presupposes a purpose for the pinky in the first place!  But alas, I admit I’ve just returned from drinking beer, so this may not be the best argument 😃.

---

From: Srivatsn Narayanan
Sent: Saturday, May 03, 2008 12:21 AM

The pinkie is like a reserved field/custom field in a API. Every user community can come up with its own use for it but they have to devise the protocol themselves. Apparently different civilizations have. Wikipedia as usual has a list of those. So do you get your “2 creams no sugar” appendage – sure if u can agree on the semantics with the waiter. Seems like a PM call to me 😄

PS: What beer did u have? I should try that – seems effective 😄

---

From: Dino Viehland
Sent: Saturday, May 03, 2008 12:26 AM

I hadn’t quite considered it like this but I’d propose an alternate encoding – binary.  With 4 fingers one can communicate up to 16 different which seems to cover most of the listed issues.  Yes 5 gives you up to 32 messages but 16 should be enough for everyone 😃.

And they were various Hale’s Ales fresh from the brewery.

---

From: Srivatsn Narayanan
Sent: Saturday, May 03, 2008 12:31 AM

I wouldn’t want to hold up 0010 really (although it won’t be the middle finger anymore) 😄

---

From: Dino Viehland
Sent: Saturday, May 03, 2008 12:40 AM

Ahh, but let’s just call that one reserved – that leaves the end-user with 15 other messages!  Personally I’d suggest we reserve the entire bit – that way we could use other codes like 1010 for future expansion of various standard messages.  Urination seems popular on the pinky list so maybe we’d want to standardize that in the future – we of course need to look at uses of the other fingers before we come to this conclusion.  Users are now left with 8 user-defined messages but that seems plenty.  Also they can then use 1000 or 0001 (depending their endianness, a.k.a. left-handed or right-handed) for the now deprecated pinky-messages.

---

From: Srivatsn Narayanan
Sent: Saturday, May 03, 2008 12:55 AM

1000, 1100, 1110 and 1111 are actually used for counting numbers 1,2,3,4 😄 They are too obvious to be non-standardized. So they are out as well.

That cuts the user messages to 5 (third bit is reserved anyway). Some of the ones left are hard to make as well – try 0101 😄