IronPython and IronRuby CTPs for .NET 4.0 Beta 2

In case you’ve been hiding under a rock (or maybe just aren’t tracking developments in the .NET community outside of IronPython), Microsoft released Visual Studio 2010 beta 2 this week. Of course for me personally, the most important feature in Visual Studio 2010 is C# 4.0 new dynamic type (also available in Visual Basic, but since VB already supported some level of late binding it’s not exactly “new” to VB).

For those of you who want to experiment with this cool new feature, may I present IronPython 2.6 CTP for .NET 4.0 Beta 2. If you can’t think of any cool things to try with this new feature, the VB team blog has some scenarios to get your started.

Also available: IronRuby CTP for .NET 4.0 Beta 2 if you’re more into gemstones than snakes.

These are preview releases, which means they’ve gone thru basic testing. If you find any bugs, PLEASE report them via the usual channel. I wrote in my Post 2.6 Roadmap post, “we are committed to shipping the RTM of our .NET 4.0 version the day that Visual Studio 2010 is publicly available” but that means shaking out the bugs between now and then. We need your help so we’re ready to go by Visual Studio 2010 launch – March 22, 2010 as per Soma’s blog.

BTW, Alcides Fonseca suggested we call this release “IronPython 2.6 N4” since it’s designed to run on .NET Framework 4.0. I like that. What do you think?

IronPython and CodeDOM: Dynamically Compiling C# Files

As part of my series on using IronPython with WPF 1, I built an extension method in C# that does dynamic member resolution on WPF FrameworkElements. The upshot of this code is that I can write win1.listbox1 instead of win1.FindName('listbox1') when using WPF objects from Python or any DLR language. Convenient, right?

The problem with this approach is that the C# extension method gets compiled into an assembly that’s bound to a specific version of the DLR. I recently started experimenting with a more recent build of IronPython and I couldn’t load the extension method assembly due to a conflict between the different versions of Microsoft.Scripting.dll. Of course, I could have simply re-compiled the assembly against the new bits, but that would mean every time I moved to a new version of IronPython, I’d have to recompile. Worse, it would limit my ability to run multiple versions of IronPython on my machine at once. I currently have three – count ‘em, three – copies of IronPython installed: 2.0 RTM, nightly build version 46242, and an internal version without the mangled namespaces of our public CodePlex releases. Having to manage multiple copies of my extension assembly would get annoying very quickly.

Instead of adding a reference to the compiled assembly, what if I could add a reference to a C# file directly? Kinda like how adding references to Python files works, but for statically compiled C#. That would let me write code like the following, which falls back to adding a reference to the C# file directly if adding a reference to the compiled assembly fails.

try:
  clr.AddReference('Microsoft.Scripting.Extension.Wpf.dll')
except:
  import codedom
  codedom.add_reference_cs_file('FrameworkElementExtension.cs',
    ['System', 'WindowsBase', 'PresentationFramework',
     'PresentationCore', 'Microsoft.Scripting'])

Since this technique uses CodeDOM, I decided to encapsulate the code in a Python module named codedom, which is frankly pretty simple. As a shout-out to my pals on the VB team, I broke compiling out into it’s own separate function so I could easily support adding VB as well as C# files.

def compile(prov, file, references):
  cp = CompilerParameters()
  cp.GenerateInMemory = True
  for ref in references:
    a = Assembly.LoadWithPartialName(ref)
    cp.ReferencedAssemblies.Add(a.Location)
  cr = prov.CompileAssemblyFromFile(cp, file)
  if cr.Errors.Count > 0:
    raise Exception(cr.Errors)
  return cr.CompiledAssembly

def add_reference_cs_file(file, references):
  clr.AddReference(compile(CSharpCodeProvider(), file, references))

def add_reference_vb_file(file, references):
  clr.AddReference(compile(VBCodeProvider(), file, references))

The compile function uses a CodeDOM provider, which provides a convenient function to compile an assembly from a single file. The only tricky part was adding the references correctly. Of the five references in this example, the only one CodeDOM can locate automatically is System.dll. For the others, it appears that CodeDOM needs the full path to the assembly in question.

Of course, hard-coding the assembly paths in my script would be too fragile, so instead I use partial names. I load each referenced assembly via Assembly.LoadWithPartialName then pass it’s Location to the CodeDOM provider via the CompilerParameters object. I realize that loading an assembly just to find its location it kind of overkill but a) I couldn’t find another mechanism to locate an assemblies location given only a partial name and b) I’m going to be loading the referenced assemblies when I load the generated assembly anyway, so I figured it loading them to find their location wasn’t a big deal. Note, that typically you’re used to passing a string to clr.AddReference, but it also can accept an assembly object directly.

Of course, this approach isn’t what you would call “fast”. Loading the pre-compiled assembly is much, much faster than compiling the C# file on the fly. But I figure slow code is better than code that doesn’t work at all. Besides, the way the code is written, I only take the extra compile hit if the pre-compiled assembly won’t load.

I stuck my codedom.py file up on my SkyDrive. Feel free to leverage as you need.


  1. I had to put that series on the back burner in part because the December update to Windows Live totally broke my WPF photo viewing app. I’ve got a new WPF app I’m working on, but I’m not quite ready to blog about it yet.

IronPython and LiveFX: Raw HTTP Access

One of the cool things about the Live Framework is that while there’s a convenient .NET library available, you can use the raw HTTP interface from any platform. LiveFX data is served up over HTTP and is available in ATOM, RSS, JSON or POX formats. As I’ve already shown, you can easily use the .NET library from IronPython, but I wanted to try working with the raw HTTP interface to get a feel for that as well.

Unfortunately, it was harder than I expected it to be. The big issue is that the documentation on how to LiveFX authorization tokens via raw HTTP is fairly sparse and occasionally contradictory. For example, there’s a whole section on Authentication and Live Framework, but it doesn’t cover this scenario. Luckily, I was able to figure it out with the help of AtomPub Project Manager LiveFX Sample, a post on Alex Feinman’s blog, a post on Emmanuel Mesas’ blog and a little groveling around with Reflector. It does appear that the auth docs are in flux –Emmanuel refers to this MSDN article as being about RPS Soap requests, but it’s actually about delegated authority. (Is MSDN reusing URLs? Bad idea.) Also, the sample code has a comment that reads “to be replaced by delegated authorization” so it looks like changes are coming. In other words, no promises on how long this code will work!

If you look at the AtomPub Project Manager sample, there’s a WindowsLiveIdentity.cs file that implements static GetTicket method that looks similar to both the code on Alex’s blog as well as the implementation of GetWindowsLiveAuthenticationToken. The upshot is that there’s a WS-Trust endpoint for Windows Live at https://dev.login.live.com/wstlogin.srf. You send it a RequestSecurityToken (aka RST) message (with a couple of extra WL specific extensions) and it responds with the security token you’ll need for accessing the LiveFx HTTP endpoints.

I ported the GetTicket function over to IronPython. I’m using .NET classes like WebRequest and XmlReader, but there’s nothing fancy here so I would expect it to be easy enough to port over to the standard Python library.

def get_WL_ticket(username, password, compactTicket):
    req = WebRequest.Create(_LoginEndPoint)
    req.Method = "POST"
    req.ContentType = "application/soap+xml; charset=UTF-8"
    req.Timeout = 30 * 10000

    rst = get_RST_message(username, password, compactTicket)
    rstbytes = Encoding.UTF8.GetBytes(rst)
    with req.GetRequestStream() as reqstm:
      reqstm.Write(rstbytes, 0, rstbytes.Length)

    with req.GetResponse() as resp:
      with resp.GetResponseStream() as respstm:
        with XmlReader.Create(respstm) as reader:
          if compactTicket:
            name = "BinarySecurityToken"
            namespace = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
          else:
            name = "RequestedSecurityToken"
            namespace = "http://schemas.xmlsoap.org/ws/2005/02/trust"

          if not reader.ReadToDescendant(name, namespace):
            raise "couldn't find security token element"

          reader.ReadStartElement(name, namespace)
          token = reader.ReadContentAsString()
          reader.ReadEndElement()

          return Convert.ToBase64String(Encoding.UTF8.GetBytes(token))

This code simply uses a WebRequest object to post the RST message to the WS-Trust enpoint then parses the result to find the token. get_RST_message uses standard Python string formatting to generate the RST message that gets posted to the WS-Trust endpoint. I’m not exactly sure why you need to convert the token value to a byte array and then Base64 encode it, but that’s what the sample code does so I did it to.

Once you have the authentication ticket, you need to download root service endpoint document in order to get the base URL and the profiles link. Then you can download all the profiles or you can download a specific one if you know it’s leet-speak identifier. LiveFX data can be downloaded in a variety of formats: ATOM, JSON, RSS or POX. You choose your format by setting the Accept and Content-Type headers.

I wrote the following functions, the generic boilerplate download function as well a specific versions for downloading JSON and POX:

def download(url, contentType, authToken):
  req = WebRequest.Create(url)
  req.Accept = contentType
  req.ContentType = contentType
  req.Headers.Add(HttpRequestHeader.Authorization, authToken)

  return req.GetResponse()  

def download_json(url, authToken):
  resp = download(url, 'application/json', authToken)
  with StreamReader(resp.GetResponseStream()) as reader:  
      data = reader.ReadToEnd()
      return eval(data)

def download_pox(url, authToken):
  resp = download(url, 'text/xml', authToken)
  return XmlReader.Create(resp.GetResponseStream())

Using JSON in Python is really easy, since I can simply eval the returned string and get back Python dictionary objects, similar to what you can do in Javascript.

Here’s some code that uses the get_WL_ticket and download_json functions above to retrieve the the user’s Personal Status Message

#Get user's WL ticket

uid = raw_input("enter WL ID: ")
pwd = raw_input("enter password: ")

authToken = livefx_http.get_WL_ticket(uid, pwd, True)

#download root service document

service = livefx_http.download_json(_LiveFxUri, authToken)

#download general profile document

url = service['BaseUri'] + service['ProfilesLink'] + "/G3N3RaL"

genprofile = livefx_http.download_json(url, authToken)
print genprofile['ProfileBase']['PersonalStatusMessage']

POX is also fairly easy, though a bit more verbose than JSON. The sample code, which I have stuck on my SkyDrive, includes both POX and JSON code, so you can compare and contrast the differences.

IronPython and LiveFX: Ori’s LiveOE.py

Ori Amiga is a Group Program Manager over in the Live Framework team whom you might have seen at PDC08 delivering the Lap Around LiveFX & Mesh Services and LiveFX Programming Model Architecture and Insights talks. And apparently, he’s an IronPython fan as posted a small LiveFX Python module to his blog. It’s pretty simple – it only wraps Connect and ConnectLocal – but it does cut about ten lines of path appending, reference adding and module importing code into a single import statement. Here’s the profile access script from my last post rewritten to use Ori’s LiveOE module.

import LiveOE     
from devhawk import linq

uid = raw_input("Enter Windows Live ID: ")
pwd = raw_input("Enter Password: ")

loe = LiveOE.Connect(uid, pwd)

general = linq.Single(loe.Profiles.Entries,  
  lambda e: e.Resource.Type == LiveOE.ProfileResource.ProfileType.General)

print loe.Mesh.ProvisionedUser.Name
print loe.Mesh.ProvisionedUser.Email
print general.Resource.ProfileInfo.PersonalStatusMessage
print linq.Count(loe.Contacts.Entries)

FYI, make sure you update the sdkLibsPath in LiveOE.py – I’m not sure where Ori has installed the LiveFX SDK, but it’s *not* in the location suggested by the read me file.

BTW, it turns out the WL Profile information is read only which answers a question I had. However, reading the thread it sounds like they will eventually get around to making it read-write at some point.

IronPython and LiveFX: Accessing Profiles

I recently got access to both the Windows Azure and Live Framework CTP programs. Frankly, I’m very interested in Live Mesh, so I decided to start with a simple LiveFX program. Scott (aka ScottIsAFool) at LiveSide posted a “quick and dirty” console app that pulls info from a user’s profile via LiveFx. It’s not Mesh per se, but it does use the same framework and resource model so I decided to port it to IronPython. FYI, this app won’t run unless you’ve been received a LiveFx CTP token and provisioned yourself.

#Add LiveFX References

import sys
sys.path.append('C:\Program Files\Microsoft SDKs\Live Framework SDK\v0.9\Libraries\.Net Library')

import clr
clr.AddReference('Microsoft.LiveFX.Client')
clr.AddReference('Microsoft.LiveFX.ResourceModel')

from Microsoft.LiveFX.Client import LiveOperatingEnvironment
from Microsoft.LiveFX.ResourceModel.ProfileResource import ProfileType
from System.Net import NetworkCredential

from devhawk import linq

#get username and password from the user

uid = raw_input("Enter Windows Live ID: ")
pwd = raw_input("Enter Password: ")
creds = NetworkCredential(uid, pwd, "https://user-ctp.windows.net")

#print out user's info

loe = LiveOperatingEnvironment()
loe.Connect(creds)

general = linq.Single(loe.Profiles.Entries,  
  lambda e: e.Resource.Type == ProfileType.General)

print loe.Mesh.ProvisionedUser.Name     
print loe.Mesh.ProvisionedUser.Email
print general.Resource.ProfileInfo.PersonalStatusMessage
print linq.Count(loe.Contacts.Entries)

I did modify the app slightly, reading the WLID and password off the console – I was sure I would accidently post my personal credentials if I left them embedded in the app. Otherwise, it’s a straight port. First, I add references the LiveFX dlls. Since they’re not local to my script, I add the directory where they’re installed to sys.path, which lets me call clr.AddReference directly. Then I retrieve the user’s ID and password using raw_input (Python’s equivalent to Console.ReadLine). Finally, I connect to the user’s LiveOperatingEnvironment and pull their name, email address, personal status message and the number of contacts they have.

As per the original app, I use LINQ to find the right profile as well as count the number of contacts. I was able to reuse the linq.py file I wrote for my Rock Band song list screen scraper (though I did have to add the Count function since I hadn’t needed it previously). I’ve posted this script on my SkyDrive, and it includes my most recent linq.py file.

BTW, it doesn’t appear that you can set the PersonalStatusMessage programmatically, at least not currently. I was thinking it would be cool to build an app that sets your PSM via Twitter, but the set method of PersonalStatusMessage is marked internal. In fact, all the set methods of all the profile properties I looked at are marked internal. If someone knows how to update LiveFX resource objects in the current CTP, I’d appreciate it if you dropped me a line or left me a comment.