Category Archives : IronPython


Hybrid App Debugging – TracebackDelegate and SetTrace

Now that I’ve introduced my simple hybrid GetThings app, we need to set about adding support for debugging just the IronPython part of the app via the new lightweight debugging functionality we’re introducing in 2.6. Note, the code is up on github, but isn’t going to exactly match what I show on the blog. Also, I have a post RC1 daily build of IronPython in the Externals folder since I discovered a few issues while building this sample that Dino had to fix after RC1. Those assemblies will be updated as needed as the sample progresses.

We saw last time how how easy it is to execute a Python script to configure a C# app – only four lines of code. If we want to support debugging, we need to add a fifth:

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    ScriptEngine engine = Python.CreateEngine();
    engine.SetTrace(this.OnTraceback);

    ScriptScope s = engine.CreateScope();
    s.SetVariable("items", lbThings.Items);
    engine.ExecuteFile("getthings.py", s);
}

You’ll notice the one new line – the call to engine.SetTrace. This is actually an extension method – ScriptEngine is a DLR hosting API class and but SetTrace is IronPython specific functionality [1]. If you look at the source of Python.SetTrace, you’ll see that it’s just a wrapper around SysModule.settrace, but it avoids needing to get the engine’s shared PythonContext yourself.

SetTrace takes a TracebackDelegate as a parameter. That delegate gets registered as the global traceback handler for the Python engine (on that thread, but we’ll ignore threading for now). Whenever that engine enters a new scope (i.e. a new function), the IronPython runtime calls into the global traceback handler. While the traceback handler runs, execution of the python code in that engine is paused. When the traceback handler returns, the engine resumes executing python code.

In addition to the global traceback handler, each scope has a local traceback handler as well. The TracebackDelegate type returns a TracebackDelegate which is used as the local traceback handler for the next traceback event within that scope. Traceback handlers can return themselves, some other TracebackDelegate, or null if they don’t want any more traceback events for the current scope. It’s kinda confusing, so here’s a picture:

image

You’ll notice three different traceback event types in the picture above: call, line and return. Call indicates the start of a scope, and is always invoked on the global traceback handler (i.e. the traceback passed to SetTrace). Line indicates the Python engine is about to execute a line of code and return indicates the end of a scopes execution. As you can see, the runtime uses the return value of the traceback for the next tracing call until the end of the scope. The return value from the “return” event handler is ignored.

So now that we know the basics of traceback handlers, here’s a simple TracebackDelegate that simply returns itself. The “Hello, world!” of traceback debugging if you will.

private TracebackDelegate OnTraceback
    (TraceBackFrame frame, string result, object payload)
{
    return this.OnTraceback;
}

If you run this code, there will be no functional difference from the code before you added the SetTrace call. That’s because we’re not doing anything in the traceback handler. But if you run this in the debugger with a breakpoint on this function, you’ll see that it gets called a bunch of times. In the python code from the last post, there are three scopes – module scope, download_stuff function scope and the get_nodes function scope. Each of those function scopes will have a call and return event, plus a bunch of line events in between.

The parameters for TracebackDelegate are described in the Python docs. The frame parameter is the current stack frame – it has information about the local and global variables, the code object currently executing, the line number being executed and a pointer to the previous stack frame if there is one. More information on code and frame objects is available in the python data model (look for “internal types”). Result is the reason why the traceback function is being called (in Python docs, it’s called “event” but that’s a keyword in C#). IronPython supports four traceback results: “call”, “line” and “return” as described above plus “exception” when an exception is thrown. Finally, the payload value’s meaning depends on the traceback result. For call and line, payload is null. For return, payload is the value being returned from the function. For exception, the payload is information about the exception and where it was thrown.

As I mentioned above, python code execution is paused while the traceback handler executes and then continues when the traceback handler returns. That means you need to block in that function if you want to let the user interact with the debugger. For a console app like PDB, you can do that with a single thread of execution easily enough. For a GUI app like GetThings, that means running the debugger and debugee windows on separate threads. And as I alluded to, tracing for Python script engines is per thread. So next time, we’ll look deeper into how to use multiple threads for lightweight debugging a hybrid app.


[1] Eventually, I’d like to see IronRuby support lightweight debugging as well. However, there’s no built in mechanism for Ruby debugging the way there is for Python, so it’s less clear how we should expose debugging to the Ruby developer. We’d also want to build a language neutral DLR Hosting API mechanism for lightweight debugging as well at that point. But honestly, we have higher priorities at this point.


Lightweight Debugging for Hybrid C#/IronPython Apps

designallCAZM93SM

One of the IronPython scenarios that I’m hearing more and more about recently is for polyglot programs. In these scenarios, part of the application is built in IronPython other parts are build in compiled, statically typed languages like C# or Visual Basic. Sometimes, programs are written this way to allow the C# app to access a Python library, like my Pygments for WL Writer plugin. Other programs want to be customizable by the end user, like Intellipad. Whatever the reason, I think that the number of these hybrid polyglot programs is going up, which partially explains why the C# team added the new dynamic type to C# 4.0.

(FYI: the You had me at “dynamic” shirt above is available for sale in my Zazzle store along with my Architecture Help 5¢ shirt)

The thing is that if you’re going to build polyglot apps, you’re probably going to want the ability to debug polyglot apps as well. I’ve written extensively about building a debugger for IronPython. However, ipydbg uses the CLR debugger under the hood which means you have to have the debugger and the code it’s debugging in separate processes. That’s a huge design burden for building a debuggable polyglot application. Luckily, as of IronPython 2.6, we support Python’s built-in trace debugging capability (aka sys.settrace). While you can use this in pure Python apps (like PDB), you can also use it polyglot C# (or VB)/IronPython apps as well. If only someone were to take the time to build a sample and document what he did along the way…

Hey, that sounds like PM work!

Seriously, let me introduce you to the worlds simplest Twitter application: GetThings. The app downloads a list of my tweets via the Twitter API and displays them in a list box. The UI is written in C# while the tweet download code is written in Python. Clearly, this is a pretty brain dead app – but the point isn’t to build a great Twitter app but rather to show how to use the settrace API from C#.

I’ve stuck the code up on GitHub. If you want to see the basic app in action sans debugging, start with the initial checkin. As you can see here, basic C# / IronPython integration is pretty trivial. I’m simply creating an engine and a scope, adding the list boxes’ Items property to the scope, and executing the getthings.py file from the disk.

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    ScriptEngine engine = Python.CreateEngine();
    ScriptScope  scope = engine.CreateScope();
    scope.SetVariable("items", lbThings.Items);
    engine.ExecuteFile("getthings.py", scope);
}

Since GetThings.py is just a text file, the user can modify it to get a list of anything they want – some other user’s timeline, the public timeline, or even – gasp! – something not from Twitter! In fact, as you see below, I’ve actually modified it to pull the tweets from a file on disk so I can avoid hitting the network on every run.

import clr
clr.AddReference("System.Xml")
from System.Xml import XmlDocument

def get_nodes(xml):
    return xml.SelectNodes("statuses/status/text")

def download_stuff():
    x = XmlDocument()

    #load from disk to save time in development
    #x.Load("http://twitter.com/statuses/user_timeline/devhawk.xml")
    x.Load("devhawk.xml")

    for n in get_nodes(x):
        txt = n.InnerText
        items.Add(txt)

download_stuff()

OK, so that’s the basics of the world’s simplest hybrid C#/IronPython Twitter application. Next up, I’ll add the settrace basics.


IronPython Post 2.6 Roadmap

It’s not quite out the door yet, but things have gotten quieter around here since we shipped the Release Candidate of 2.6. But there’s no rest for the dynamic, so we’ve already started thinking about what we do next.

Since we shipped 2.0 last December, we’ve shipped two service releases: 2.0.1 two months later in February and 2.0.2 four months after that in June. We weren’t planning on doing a 2.0.3 release, but then we discovered the CLR folks made a breaking change to partial trust in Windows 7. David recently emailed the IronPython mailing list looking for feedback on other must-fix bugs we can get to for 2.0.3. If you’ve got an opinion on 2.0.3 must-fix bugs, please respond to that thread.

For 2.6 service releases, we’re looking to tighten up the timeframe a bit from last time. We’re planning to ship service releases of 2.6 in December and February. However, since we don’t have a major release ship vehicle in the 2.x branch until Python 2.7 next summer, these service releases may contain new functionality in addition to bug fixes. In particular, we will look to include any missing modules work that I discussed in my RC announcement post in these service releases.

In addition to the IronPython 2.6 service releases, we’ll also continue to track the Visual Studio 2010 beta cycle as it heads to RTM. I can’t comment on VS2010 dates, but I can say that we will ship a CTP of IronPython 2.6 for .NET Framework 4.0 for Visual Studio 2010 Beta 2 and that we are committed to shipping the RTM of our .NET 4.0 version the day that Visual Studio 2010 is publicly available. Given that dynamic the big new feature of C# 4.0, we want to make sure IronPython is ready to go as soon as C# 4.0 is available.

BTW, we are well aware that “IronPython 2.6 for .NET Framework 4.0” is a long ugly name. Any suggestions on a different one? We’ve been thinking “IronPython 2.6 R2” since the “R2” nomenclature is bigaroundhere. But “R2” doesn’t really capture the essence of the .NET 4.0 compatible release. Besides, when has the dynamic languages team at Microsoft ever done ANYTHING because it was popular with the Microsoft marketing folks? :)

At this point, we’ve got the next few months mapped out, but not much more beyond that. Specifically, we have two gaping holes in the roadmap:

  • Visual Studio Integration
  • IronPython 3.x

For now, I’m going to leave these holes unfilled. Currently, the rest of my VS Languages teammates (along with the rest of DevDiv) are heads down driving towards beta 2 of Visual Studio 2010. Once they reach that milestone, planning on Visual Studio v.next will begin. Those plans have the potential for impacting how the IronPython team proceeds going forward.

Frankly, several members of the dynamic languages team have been pitching in with the “drive to beta 2” effort so we’re a little more short handed than usual. If we get those people back, then we’re in a much better position to execute on both VS Integration and IronPython 3.x. If we don’t, then we have to make some hard choices about how to proceed. I’m guessing most would agree that VS integration is more important IronPython 3.x support. However, those aren’t equal efforts by any stretch of the imagination. How valuable is VS integration if, for example, we don’t have the bandwidth to build decent intellisense? As I said, hard choices.

If you want to make your voice heard on Microsoft’s level of commitment to IronPython, make sure you go rate the IronPython integration issue on Microsoft Connect. (while you’re there, rate the IronRuby integration issue too.) The IronPython integration is currently the top rated open issue on the VS Connect site and is the second highest vote getter out of all the Connect issues (active or otherwise) logged since VS 2010 shipped it’s first CTP! [1]


[1] Seriously, the next most recent Connect issue with more votes than IronPython integration is HttpRuntime.ProcessRequest() does not work in IIS7 Integrated Pipeline Mode from November 2007. The next one before that was Create Service Pack for Visual Studio 2005 from November 2005.


IronPython 2.6 Release Candidate

image

We released the IronPython 2.6 release candidate on CodePlex yesterday. If all goes well, this will be the ONLY RC and we’ll republish it as the RTW build in about a month. So if you’ve been holding off on experimenting with the 2.6 release, now’s the time to jump in with both feet.

As I’ve written before on this blog, this is a HUGE release for us:

Anyway, with 2.6 winding down, the IronPython team finds ourselves in a unique position that we’ve never been in before: caught up. As far as I can tell, most of the Python community hasn’t made the move to Python 3.1 and Python 2.7 is looking like it will be released next summer. So IronPython is caught up with the latest version of Python most of the Python community appears to be using.

So that begs the question: what do we do now?

Of course, we want to hear from you regarding our next steps, but some things we are looking at include:

  • .NET Framework 4.0
    We’ve shipped CTP releases of IronPython for each public beta of Visual Studio 2010 and  .NET Framework 4.0. You can expect that to continue as Visual Studio 2010 winds down and ships.
  • Fixing Bugs
    417 bugs fixed is good, but there are still 839 active work items in our issue tracker. In previous releases, we’ve done minor bug fix releases every few months so we’ll probably keep up that cadence. Make sure you go vote for bugs you think are important.
  • App Compatibility
    One thing we can do is take some of the more popular Python apps such as Django and Mercurial and make sure they run well on IronPython. In some cases, there may need to be changes to the Python apps to get them to run on IronPython (for example, see Jeff Hardy’s patch for running Django on IronPython) which assuredly means more time with lawyers for me.
  • Missing Modules
    While IronPython 2.6 is our implementation of Python 2.6, there are binary modules we haven’t implemented yet like _csv, _ast and pyexpat. You could consider this App Compatibility work, but we have a different internal process for bug fixing and implementing new features so I broke this out separately.
  • Documentation
    Invariably an area for improvement in all software, our doc story today is pretty much “go look at docs.python.org”. Of course, that doesn’t cover any IronPython specific functionality. Of course, what would be great would be to combine existing Python docs with IronPython specific docs in a single reference, which also assuredly means more time with lawyers for me.
  • Tutorials
    If you haven’t seen Michael Foord’s Try Python, it’s awesome. However, it was recently pointed out to me that it doesn’t include any IronPython specific behavior (importing and interoperating with .NET types for example). The IronRuby Tutorial has specific IronRuby features and it would be awesome to do the same for IronPython. Of course, if we wrote new tutorials in reStructured Text, then I’m guessing it would be easy for Michael to include it in Try Python via his rst2xaml tool.
  • “Just Text” in Silverlight
    If you haven’t seen MIX Online Labs Gestalt project, browse to one of the samples and View Source. Python as “just text” in the browser. Cool, eh? Jimmy is working on implementing the “just text” model for the Silverlight version of IronPython (and IronRuby).
  • Visual Studio
    There’s no work item for VS Integration in our issue tracker, but there have been 112 votes for IronPython integration (as well as 79 votes for IronRuby integration) in the VS 2010 connect bug database. No promises here, but we are acutely aware of how popular this suggestion is.
  • CodePlex Foundation
    The CodePlex Foundation is a new non-profit foundation sponsored by Microsoft with the explicit mission “to enable the exchange of code and understanding among software companies and open source communities.” As one of the oldest open source projects at Microsoft, we are very interested in the CodePlex Foundation as you might imagine. However, CodePlex Foundation is VERY new – everything is tagged “interim” for the first 100 days. Once the Foundation elects non-interim board members and establishes things like a charter and by-laws, you can be sure we’ll be investigating it more thoroughly. In other words, definitely more time with lawyers for me.

In case I haven’t said it lately, it’s great working on the IronPython team with Dino Viehland, Dave Fugate and David DiCato (The IronRuby guys aren’t bad either!). Also, I may whine about the amount of time I spend with lawyers, but honestly Yong, Kathryn and Kevin – our main LCA contacts – do a great job helping us figure out how and where to push the envelope so thanks to them as well.


Danish University Tour Trip Report

I’ve been back from Denmark for a week – it took me that long to get back on Pacific time zone and dig out from the mountain of email that collected while I was gone. But I got word from my esteemed host Martin Esmann that the video of my TechTalk had been posted to Channel 9, so it seems as good a time as any for my trip report.

At each of these universities, I did two talks. The first was Pumping Iron: The State of Dynamic Languages on the .NET Framework. That’s the one in the Channel 9 video. The other talk was Developing with the DLR, which I’ve posted to my Skydrive. That talk was more focused on the CLR and DLR as a platform for language development. If there’s interest (leave a comment), I’ll record audio for that presentation and post it up on SlideShare or something like that.

Aalborg University

IronPython_DK_trip_on_a_map

First stop of the University Tour was Aalborg University – about a four hour drive from Copenhagen (or as the Danish spell it København). As you can see on the map, we started with the furthest university away then worked our way back to Copenhagen. Martin picked me up at the airport and we hit the road. I was horribly jet lagged, but we spent most of the drive geeking out about programming languages (when I wasn’t napping).

Each of the universities had a different focus that I was interested in. At Aalborg, it was in teaching programming. Given the popularity of Python in education, we had lots to talk about. My host at Aalborg was Bent Thomsen who has done significant research on this topic, as has his wife Lone. One of the areas we discussing in particular was about teaching classes vs. objects first. Apparently, these days there’s significant momentum behind teaching class first, but the folks at Aalborg – as I understand it – have seen that approach has been ineffective. That squared with a talk I saw on teaching OOP with Python that I saw last year at PyCon by Dr. Goldwasser.

The other area we discussed was teaching compiler design. Clearly, this is an area I thought the DLR could be a big help in. Bent primarily uses Programming Language Processors in Java to teach this course, but he finds that it’s a little out of date + he’s much more interested in the compiler aspect than the interpreter aspect of that book. The idea of generating DLR Expression Trees which can be compiled into IL with a simple call to LambdaExpression.Compile and can then run directly on the CLR seemed to sound appealing to him.

Aarhus University

Next up was Aarhus University. I didn’t get to spend as much time with the folks from Aarhus, though our host Olivier Danvy did help me find a “uniquely Danish” gift for my wife (though I also brought her Danish bread & chocolate). We spent some time talking about F# and compiler design, and Olivier made this great comment over lunch that “OCaml is a domain specific language for compiler writers”. That is very true.

Olivier had to leave for a previous engagement, so Jan Midtgaard – who teaches Aarhus’ Compilation class – hosted me for the actual talks. In that class, they implement a subset of Java known as Joos in AspectJ. One of the cool things about this class is how they grade the compilers – you upload your complier code to a web site and they run it against an existing test framework. They couldn’t give the students the test framework directly because it would be too easy to reverse engineer the tests from the compiled Java code (I assume the Java world has the equivalent of Reflector?) Like Aalborg, I spent significant time discussing the idea of targeting the DLR in their compiler class.

University of Southern Denmark

Of the four universities I visited, this was probably the poorest fit for the content I was delivering. On the other hand, it was awesome – USD has a strong focus on Robotics. They even have a RoboDays Robot Festival in Odense. Unfortunately, the festival was the weekend after I visited, so many of the robots were out in preparation for the festival. However, our host Ulrik Schultz did bring us to to visit the Modular Robotics Research Lab so I got to see these guys in action:

Real transforming robots! Not quite Optimus Prime mind you, but you gotta start somewhere. These robots are call ATRON and they’re pretty impressive to see in action. However, they are kinda weak computationally – they only have 128k of flash memory + 4k of ROM. 4K! However, the next generation of these bots is supposed to have like 64MB of memory and a FPGA – more than enough power to use the the Micro Framework.

University of Copenhagen

The final stop on my university tour was University of Copenhagen, where I was hosted by Fritz Henglein of the Algorithms and Programming Languages Group. He’s done some research on dynamic languages, but these days he and his team are doing some fascinating research on the intersection of business and programming languages. He leads a project called 3gERP in partnership with Microsoft Development Center Copenhagen to “develop a standardized, yet highly configurable and flexible, global ERP system for SME’s based on fundamentally new software architecture.” That topic isn’t as interesting to me in my current role in the VS Languages team, but was very relevant to my background in services architecture and MSIT.

Copenhagen .NET Users Group

While I was in Denmark, I got to speak to the local .NET users group about IronPython. These are always fun since they are less formal. Also, this was a much more .NET knowledgeable crowd than I had had all week so I got some deeper questions about the .NET stack. Among the members of that user group is Mads Kristensen, founder of the BlogEngine.NET project. So I put together a special demo “integrating” IronPython into BlogEngine.NET. It was kinda silly – I wrote an extension that would execute any blog post as python that was in the RunPython category. But it was also mind-numbingly simple:

[Extension("Extend .NET Blog Engine with IronPython")]
public class IPyExtension
{
    static ScriptEngine _engine;

    static IPyExtension()
    {
        _engine = IronPython.Hosting.Python.CreateEngine();
        Post.Serving += new EventHandler<ServingEventArgs>(Post_Serving);
    }


    static void Post_Serving(object sender, ServingEventArgs e)
    {
        Post p = (Post)sender;
        bool runpython = p.Categories.Exists(delegate(Category cat) { 
            return string.Compare(cat.Title, "RunPython", true) == 0; });
        if (runpython)
        {
            e.Body = _engine.Execute<string>(e.Body);
        }
    }
}

I’d love to see REAL IronPython integration into BlogEngine.NET, but I needed something I could do in a couple of hours late at night in a code base I wasn’t familiar with.

Pumping Iron TechTalk

IMG_2297 Finally, on Friday I delivered my final talk of the week, which is available to watch and download from Channel 9. It was a packed house and I was a little nervous about having the talk recorded, but I think it went well. Certainly, it seemed well received by the audience.

Get Microsoft Silverlight

I ended the work part of my trip to Denmark with an interview with a Danish IT newspaper (which I’ll post a link to when it gets published) and a sit down with the Dyanmics:NAV team. My wife always tells me that she’d love it if I got a 6-12 month assignment to work overseas, so I tried to convince the NAV team that they REALLY want to integrate IronPython like the Dynamics AX team is doing. If you see a post here about me moving to Copenhagen, you’ll know I was successful! :)

Final Thoughts

IMG_2276Other than the jet lag, which seemed tougher on this trip than when I’ve gone westward to New Zealand, Australia or Malaysia, I had a blast. It was a real treat seeing so much of Denmark and getting to talk to so many interesting people. I even got to do some sight seeing in Copenhagen and Odense. Hans Christian Andersen was born in Odense and so I got to see the house he was born in as well as get my picture taken with this statue of him right outside our hotel. Major thanks to Martin Esmann for inviting me, driving me around – we drove a megameter, aka 1000km, over the course of the trip – and being an all around amazing host.