As part of my series on using IronPython with WPF ¹, I built an extension method in C# that does dynamic member resolution on WPF FrameworkElements. The upshot of this code is that I can write win1.listbox1 instead of win1.FindName('listbox1') when using WPF objects from Python or any DLR language. Convenient, right?

The problem with this approach is that the C# extension method gets compiled into an assembly that’s bound to a specific version of the DLR. I recently started experimenting with a more recent build of IronPython and I couldn’t load the extension method assembly due to a conflict between the different versions of Microsoft.Scripting.dll. Of course, I could have simply re-compiled the assembly against the new bits, but that would mean every time I moved to a new version of IronPython, I’d have to recompile. Worse, it would limit my ability to run multiple versions of IronPython on my machine at once. I currently have three – count ‘em, three – copies of IronPython installed: 2.0 RTM, nightly build version 46242, and an internal version without the mangled namespaces of our public CodePlex releases. Having to manage multiple copies of my extension assembly would get annoying very quickly.

Instead of adding a reference to the compiled assembly, what if I could add a reference to a C# file directly? Kinda like how adding references to Python files works, but for statically compiled C#. That would let me write code like the following, which falls back to adding a reference to the C# file directly if adding a reference to the compiled assembly fails.

try:
  clr.AddReference('Microsoft.Scripting.Extension.Wpf.dll')
except:
  import codedom
  codedom.add_reference_cs_file('FrameworkElementExtension.cs',
    ['System', 'WindowsBase', 'PresentationFramework',
     'PresentationCore', 'Microsoft.Scripting'])

Since this technique uses CodeDOM, I decided to encapsulate the code in a Python module named codedom, which is frankly pretty simple. As a shout-out to my pals on the VB team, I broke compiling out into it’s own separate function so I could easily support adding VB as well as C# files.

def compile(prov, file, references):
  cp = CompilerParameters()
  cp.GenerateInMemory = True
  for ref in references:
    a = Assembly.LoadWithPartialName(ref)
    cp.ReferencedAssemblies.Add(a.Location)
  cr = prov.CompileAssemblyFromFile(cp, file)
  if cr.Errors.Count > 0:
    raise Exception(cr.Errors)
  return cr.CompiledAssembly

def add_reference_cs_file(file, references):
  clr.AddReference(compile(CSharpCodeProvider(), file, references))

def add_reference_vb_file(file, references):
  clr.AddReference(compile(VBCodeProvider(), file, references))

The compile function uses a CodeDOM provider, which provides a convenient function to compile an assembly from a single file. The only tricky part was adding the references correctly. Of the five references in this example, the only one CodeDOM can locate automatically is System.dll. For the others, it appears that CodeDOM needs the full path to the assembly in question.

Of course, hard-coding the assembly paths in my script would be too fragile, so instead I use partial names. I load each referenced assembly via Assembly.LoadWithPartialName then pass it’s Location to the CodeDOM provider via the CompilerParameters object. I realize that loading an assembly just to find its location it kind of overkill but a) I couldn’t find another mechanism to locate an assemblies location given only a partial name and b) I’m going to be loading the referenced assemblies when I load the generated assembly anyway, so I figured it loading them to find their location wasn’t a big deal. Note, that typically you’re used to passing a string to clr.AddReference, but it also can accept an assembly object directly.

Of course, this approach isn’t what you would call “fast”. Loading the pre-compiled assembly is much, much faster than compiling the C# file on the fly. But I figure slow code is better than code that doesn’t work at all. Besides, the way the code is written, I only take the extra compile hit if the pre-compiled assembly won’t load.

I stuck my codedom.py file up on my SkyDrive. Feel free to leverage as you need.

Here are some technical details on my Nightly Builds solution. I broke them into a separate post because I figured most people are more interested in the actual service than how it’s built.

As you might expect, I built most of the solution in IronPython. All of the download, build, compress and Azure upload code was written in IPy. The one part I didn’t write in IPy was the Azure cloud web app, which I wrote in C#. Jon Udell’s been investigating getting IPy to run in Azure, but I just wanted something quick and dirty (as you can see from the utter lack of formatting) so I decided to use C# instead. Man, were my ASP.NET skills rusty.

As for the IronPython parts, for the most part I’m using external tools for downloading, building and compressing. I use the Source Control RSS Feed to discover recent source code changesets, CodePlex Client to download source from CodePlex, MSBuild to build the binaries, 7-zip to compress the binaries and the StorageClient library sample to upload the compressed binaries up to Azure blob storage.

For building and compressing, I’m literally shelling out to MSBuild and 7-Zip via os.system. I looked at programmatically building via the MSBuild API, but I ran into an assembly binding bug that I wasn’t motivated enough to work around. As for creating zip files programmatically, IronPython doesn’t have a zlib module implementation yet so I just used 7-Zip’s command line utility instead.

For downloading form CodePlex, I originally started by shelling out to CodePlex Client. However, I wanted the ability to cloak folders – for example Tutorial and SrcTests – that weren’t required to build. CodePlex Client has a very useful TFS library embedded in it – the build process combines all the libraries into a single executable via ILMerge. I could have compiled my own version of the TFS library, but instead I just load cpc.exe as an assembly reference via clr.AddReferenceToFileAndPath. It’s a nifty trick Jim Hugunin showed me once.

Uploading to Azure was very straightforward because of the StorageClient library. Here’s the code to create a blob container object (creating the actual blob container if it doesn’t already exist) and to upload a file to a container.

def get_blob_container(prj):
  azure_account = StorageAccountInfo(endpoint, None, azure_name, azure_key)
  storage = BlobStorage.Create(azure_account)
  container = storage.GetBlobContainer(prj.lower())
  if not container.DoesContainerExist():
    print "Creating", prj, "Azure Blob Storage Container"
    container.CreateContainer(None, ContainerAccessControl.Public)
  return container

def upload_to_azure(container, upload_filepath, azure_filename, metadata):
    print "Uploading", azure_filename, "to Azure"
    prop = BlobProperties(azure_filename)
    nv = NameValueCollection()
    for key in metadata:
      nv[key] = metadata[key]
    prop.Metadata = nv

    with File.OpenRead(upload_filepath) as stream:
      contents = BlobContents(stream)
      if not container.CreateBlob(prop, contents, True):
        raise "Uploading " + azure_filename + " to Azure failed"

I’ve been working on some pure IronPython code to access the blob storage REST API directly, but that’s primarily to familiarize myself with the service. At some point, I’m going to want to leverage Table Storage but my brief experimentation with the StorageClient Table Storage interface makes me think that it depends on static typing too much to be useful for IPy. If that turns out to be true, the Table Storage REST API will be my only option.

As you can see in the code above, these Azure blob containers are set to be publically accessible (via ContainerAccessControl.Public argument passed to CreateContainer). So for my C# app, I’m simply using calling XDocument.Load with the List Blobs operation url, shaping the results via LINQ to XML and binding them to nested ASP.NET Repeater controls.

Assuming people find this useful, I’m thinking of some additional improvements, in order of what I’m likely to get to first:

• Caching Project Info in the cloud app
  Currently, I’m hitting getting and processing the list of binary releases on every request. I’m sure caching that data to make it more efficient.
• Virtual Build Environment
  Currently, I’m just building on my laptop. It would be nice to have a clean environment dedicated to running the build script.
• Auto-Build
  My script uses the RSS feed to find the recent checkins, but I have to manually kick off the process. I’d like it to set it up as a service that periodically checks the source code RSS feed automatically and downloads and builds any new releases that it finds.
• Table Storage for Build Metadata
  Today, I am simply grabbing the list of all uploaded compressed binaries for a given project, parsing their names, and displaying that as a hierarchical list on the project page. If I used Table Storage, I could add additional metadata including social software features like ratings and comments.
• Amazon EC2 Virtual Build Environment
  If I’m creating a virtual machine for my build environment, I could look at hosting it on Amazon EC2. They support Windows now after all. Ideally, I’d use an Azure worker role for compiling and compressing builds, but our build tools need access to the file system.

IronPython 2.0 shipped about a month ago, but we’re still chugging along with our post 2.0 work. We’ve shipped seven source code releases since we shipped 2.0 and we should be back to our normal schedule of updating the source 2-3 times a week schedule by next week. Given how often we ship source, we’re thinking of extending the the time between binary drops. Binary releases have to be signed and there’s a fairly arduous process we have to go thru in order to get each binary release out the door.

However, there’s something nice and convenient about downloading a pre-compiled binary release. So I spent my Christmas vacation building a script to download and build IronPython nightly builds. Once built, I compress the binaries and upload them to Azure blob storage. Finally, I built a very simple cloud app for users to view and download available nightly builds. As an extra benefit, I’m also providing nightly builds of the DLR.

Please note, these are *NOT* official Microsoft releases of IronPython and/or DLR. They aren’t signed and they haven’t gone through the aforementioned release process. I’m just downloading the public source, building it with the publicly available tools, then making them available on a a publicly accessible website.

The website for the IronPython (and DLR) nightly builds is http://nightlybuilds.cloudapp.net.

As usual, I welcome any feedback. Is having prebuilt unsigned binaries of IPy releases useful? Do you want IronRuby binaries as well? What about social features (rating releases, comments, etc)? Please let me know what you think.

One of the cool things about the Live Framework is that while there’s a convenient .NET library available, you can use the raw HTTP interface from any platform. LiveFX data is served up over HTTP and is available in ATOM, RSS, JSON or POX formats. As I’ve already shown, you can easily use the .NET library from IronPython, but I wanted to try working with the raw HTTP interface to get a feel for that as well.

Unfortunately, it was harder than I expected it to be. The big issue is that the documentation on how to LiveFX authorization tokens via raw HTTP is fairly sparse and occasionally contradictory. For example, there’s a whole section on Authentication and Live Framework, but it doesn’t cover this scenario. Luckily, I was able to figure it out with the help of AtomPub Project Manager LiveFX Sample, a post on Alex Feinman’s blog, a post on Emmanuel Mesas’ blog and a little groveling around with Reflector. It does appear that the auth docs are in flux –Emmanuel refers to this MSDN article as being about RPS Soap requests, but it’s actually about delegated authority. (Is MSDN reusing URLs? Bad idea.) Also, the sample code has a comment that reads “to be replaced by delegated authorization” so it looks like changes are coming. In other words, no promises on how long this code will work!

If you look at the AtomPub Project Manager sample, there’s a WindowsLiveIdentity.cs file that implements static GetTicket method that looks similar to both the code on Alex’s blog as well as the implementation of GetWindowsLiveAuthenticationToken. The upshot is that there’s a WS-Trust endpoint for Windows Live at https://dev.login.live.com/wstlogin.srf. You send it a RequestSecurityToken (aka RST) message (with a couple of extra WL specific extensions) and it responds with the security token you’ll need for accessing the LiveFx HTTP endpoints.

I ported the GetTicket function over to IronPython. I’m using .NET classes like WebRequest and XmlReader, but there’s nothing fancy here so I would expect it to be easy enough to port over to the standard Python library.

def get_WL_ticket(username, password, compactTicket):
    req = WebRequest.Create(_LoginEndPoint)
    req.Method = "POST"
    req.ContentType = "application/soap+xml; charset=UTF-8"
    req.Timeout = 30 * 10000

    rst = get_RST_message(username, password, compactTicket)
    rstbytes = Encoding.UTF8.GetBytes(rst)
    with req.GetRequestStream() as reqstm:
      reqstm.Write(rstbytes, 0, rstbytes.Length)

    with req.GetResponse() as resp:
      with resp.GetResponseStream() as respstm:
        with XmlReader.Create(respstm) as reader:
          if compactTicket:
            name = "BinarySecurityToken"
            namespace = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
          else:
            name = "RequestedSecurityToken"
            namespace = "http://schemas.xmlsoap.org/ws/2005/02/trust"

          if not reader.ReadToDescendant(name, namespace):
            raise "couldn't find security token element"

          reader.ReadStartElement(name, namespace)
          token = reader.ReadContentAsString()
          reader.ReadEndElement()

          return Convert.ToBase64String(Encoding.UTF8.GetBytes(token))

This code simply uses a WebRequest object to post the RST message to the WS-Trust enpoint then parses the result to find the token. get_RST_message uses standard Python string formatting to generate the RST message that gets posted to the WS-Trust endpoint. I’m not exactly sure why you need to convert the token value to a byte array and then Base64 encode it, but that’s what the sample code does so I did it to.

Once you have the authentication ticket, you need to download root service endpoint document in order to get the base URL and the profiles link. Then you can download all the profiles or you can download a specific one if you know it’s leet-speak identifier. LiveFX data can be downloaded in a variety of formats: ATOM, JSON, RSS or POX. You choose your format by setting the Accept and Content-Type headers.

I wrote the following functions, the generic boilerplate download function as well a specific versions for downloading JSON and POX:

def download(url, contentType, authToken):
  req = WebRequest.Create(url)
  req.Accept = contentType
  req.ContentType = contentType
  req.Headers.Add(HttpRequestHeader.Authorization, authToken)

  return req.GetResponse()  

def download_json(url, authToken):
  resp = download(url, 'application/json', authToken)
  with StreamReader(resp.GetResponseStream()) as reader:  
      data = reader.ReadToEnd()
      return eval(data)

def download_pox(url, authToken):
  resp = download(url, 'text/xml', authToken)
  return XmlReader.Create(resp.GetResponseStream())

Using JSON in Python is really easy, since I can simply eval the returned string and get back Python dictionary objects, similar to what you can do in Javascript.

Here’s some code that uses the get_WL_ticket and download_json functions above to retrieve the the user’s Personal Status Message

#Get user's WL ticket

uid = raw_input("enter WL ID: ")
pwd = raw_input("enter password: ")

authToken = livefx_http.get_WL_ticket(uid, pwd, True)

#download root service document

service = livefx_http.download_json(_LiveFxUri, authToken)

#download general profile document

url = service['BaseUri'] + service['ProfilesLink'] + "/G3N3RaL"

genprofile = livefx_http.download_json(url, authToken)
print genprofile['ProfileBase']['PersonalStatusMessage']

POX is also fairly easy, though a bit more verbose than JSON. The sample code, which I have stuck on my SkyDrive, includes both POX and JSON code, so you can compare and contrast the differences.

Ori Amiga is a Group Program Manager over in the Live Framework team whom you might have seen at PDC08 delivering the Lap Around LiveFX & Mesh Services and LiveFX Programming Model Architecture and Insights talks. And apparently, he’s an IronPython fan as posted a small LiveFX Python module to his blog. It’s pretty simple – it only wraps Connect and ConnectLocal – but it does cut about ten lines of path appending, reference adding and module importing code into a single import statement. Here’s the profile access script from my last post rewritten to use Ori’s LiveOE module.

import LiveOE     
from devhawk import linq

uid = raw_input("Enter Windows Live ID: ")
pwd = raw_input("Enter Password: ")

loe = LiveOE.Connect(uid, pwd)

general = linq.Single(loe.Profiles.Entries,  
  lambda e: e.Resource.Type == LiveOE.ProfileResource.ProfileType.General)

print loe.Mesh.ProvisionedUser.Name
print loe.Mesh.ProvisionedUser.Email
print general.Resource.ProfileInfo.PersonalStatusMessage
print linq.Count(loe.Contacts.Entries)

FYI, make sure you update the sdkLibsPath in LiveOE.py – I’m not sure where Ori has installed the LiveFX SDK, but it’s *not* in the location suggested by the read me file.

BTW, it turns out the WL Profile information is read only which answers a question I had. However, reading the thread it sounds like they will eventually get around to making it read-write at some point.