Here are some technical details on my Nightly Builds solution. I broke them into a separate post because I figured most people are more interested in the actual service than how it’s built.
As you might expect, I built most of the solution in IronPython. All of the download, build, compress and Azure upload code was written in IPy. The one part I didn’t write in IPy was the Azure cloud web app, which I wrote in C#. Jon Udell’s been investigating getting IPy to run in Azure, but I just wanted something quick and dirty (as you can see from the utter lack of formatting) so I decided to use C# instead. Man, were my ASP.NET skills rusty.
As for the IronPython parts, for the most part I’m using external tools for downloading, building and compressing. I use the Source Control RSS Feed to discover recent source code changesets, CodePlex Client to download source from CodePlex, MSBuild to build the binaries, 7-zip to compress the binaries and the StorageClient library sample to upload the compressed binaries up to Azure blob storage.
For building and compressing, I’m literally shelling out to MSBuild and 7-Zip via os.system. I looked at programmatically building via the MSBuild API, but I ran into an assembly binding bug that I wasn’t motivated enough to work around. As for creating zip files programmatically, IronPython doesn’t have a zlib module implementation yet so I just used 7-Zip’s command line utility instead.
For downloading form CodePlex, I originally started by shelling out to CodePlex Client. However, I wanted the ability to cloak folders – for example \Tutorial and \Src\Tests – that weren’t required to build. CodePlex Client has a very useful TFS library embedded in it – the build process combines all the libraries into a single executable via ILMerge. I could have compiled my own version of the TFS library, but instead I just load cpc.exe as an assembly reference via clr.AddReferenceToFileAndPath. It’s a nifty trick Jim Hugunin showed me once.
Uploading to Azure was very straightforward because of the StorageClient library. Here’s the code to create a blob container object (creating the actual blob container if it doesn’t already exist) and to upload a file to a container.
def get_blob_container(prj):
azure_account = StorageAccountInfo(endpoint, None, azure_name, azure_key)
storage = BlobStorage.Create(azure_account)
container = storage.GetBlobContainer(prj.lower())
if not container.DoesContainerExist():
print "Creating", prj, "Azure Blob Storage Container"
container.CreateContainer(None, ContainerAccessControl.Public)
return container
def upload_to_azure(container, upload_filepath, azure_filename, metadata):
print "Uploading", azure_filename, "to Azure"
prop = BlobProperties(azure_filename)
nv = NameValueCollection()
for key in metadata:
nv[key] = metadata[key]
prop.Metadata = nv
with File.OpenRead(upload_filepath) as stream:
contents = BlobContents(stream)
if not container.CreateBlob(prop, contents, True):
raise "Uploading " + azure_filename + " to Azure failed"
I’ve been working on some pure IronPython code to access the blob storage REST API directly, but that’s primarily to familiarize myself with the service. At some point, I’m going to want to leverage Table Storage but my brief experimentation with the StorageClient Table Storage interface makes me think that it depends on static typing too much to be useful for IPy. If that turns out to be true, the Table Storage REST API will be my only option.
As you can see in the code above, these Azure blob containers are set to be publically accessible (via ContainerAccessControl.Public argument passed to CreateContainer). So for my C# app, I’m simply using calling XDocument.Load with the List Blobs operation url, shaping the results via LINQ to XML and binding them to nested ASP.NET Repeater controls.
Assuming people find this useful, I’m thinking of some additional improvements, in order of what I’m likely to get to first:
- Caching Project Info in the cloud app
Currently, I’m hitting getting and processing the list of binary releases on every request. I’m sure caching that data to make it more efficient. - Virtual Build Environment
Currently, I’m just building on my laptop. It would be nice to have a clean environment dedicated to running the build script. - Auto-Build
My script uses the RSS feed to find the recent checkins, but I have to manually kick off the process. I’d like it to set it up as a service that periodically checks the source code RSS feed automatically and downloads and builds any new releases that it finds. - Table Storage for Build Metadata
Today, I am simply grabbing the list of all uploaded compressed binaries for a given project, parsing their names, and displaying that as a hierarchical list on the project page. If I used Table Storage, I could add additional metadata including social software features like ratings and comments. - Amazon EC2 Virtual Build Environment
If I’m creating a virtual machine for my build environment, I could look at hosting it on Amazon EC2. They support Windows now after all. Ideally, I’d use an Azure worker role for compiling and compressing builds, but our build tools need access to the file system.
IronPython 2.0 shipped about a month ago, but we’re still chugging along with our post 2.0 work. We’ve shipped seven source code releases since we shipped 2.0 and we should be back to our normal schedule of updating the source 2-3 times a week schedule by next week. Given how often we ship source, we’re thinking of extending the the time between binary drops. Binary releases have to be signed and there’s a fairly arduous process we have to go thru in order to get each binary release out the door.
However, there’s something nice and convenient about downloading a pre-compiled binary release. So I spent my Christmas vacation building a script to download and build IronPython nightly builds. Once built, I compress the binaries and upload them to Azure blob storage. Finally, I built a *very* simple cloud app for users to view and download available nightly builds. As an extra benefit, I’m also providing nightly builds of the DLR.
Please note, these are *NOT* official Microsoft releases of IronPython and/or DLR. They aren’t signed and they haven’t gone through the aforementioned release process. I’m just downloading the public source, building it with the publicly available tools, then making them available on a a publicly accessible website.
The website for the IronPython (and DLR) nightly builds is http://nightlybuilds.cloudapp.net.
As usual, I welcome any feedback. Is having prebuilt unsigned binaries of IPy releases useful? Do you want IronRuby binaries as well? What about social features (rating releases, comments, etc)? Please let me know what you think.