Passion * Technology * Ruthless Competence

Monday, October 26, 2009

Joining Windows

Big news in DevHawk land: I’ll be taking on a new role in the Windows division after I come back from TechEd Europe (plus a week vacation visiting my bro-in-law in Germany after TechEd).

You guys have heard of Windows, right? They just released a new version recently you might be aware of…Actually, I hear it was your idea!

Seriously, I am stoked to be joining the Windows team. I can’t say much about the new job beyond a) it’s in Windows Client (as opposed to Windows Server) and b) I’m working on a team that’s focused on the Windows developer experience. The Windows team is deep in what you might call “building on teh awesome that is Windows 7” but that they simply call “planning”, so sorry if specifics are kinda sparse. I’ll be back working for Mahesh Prakriya, who originally hired me into my current role on the IronPython team. Someday I might tell you the Mahesh PyCon Lego Story, but for now I’ll just say I was great working for Mahesh the first time and I think this time is going to be even better.

Working on Windows…focused on developer experience…for Mahesh - It’s like the perfect storm of work geekdom for me.

Of course, starting a new job means my time on the IronPython team is coming to an end. As excited as I am about this new opportunity in the Windows division, I’m a little sad to be leaving Developer Division and the IronPython team. I’ve joked with audiences that I care about Python because Microsoft pays me to care about Python, but that’s not completely true. Python is a fantastic language and IronPython’s combination of Python + .NET is hard to beat in my opinion. IronPython has made significant progress while I’ve been here the last eighteen months – two major releases (well, 2.6 is almost done), redisting the Python standard library, Visual Studio 2010 compat - but there’s still much for IronPython to accomplish. And of course, leaving behind such great teammates like Dino, Dave, Jim and Jimmy is ALWAYS hard.

The Windows team is somewhat tighter lipped than the totally transparent approach we use in IronPython. Not to worry, my evangelism skills were part of the reason I got the job so you’ll be hearing plenty from me soon enough. However, my posting here will be kinda sparse until I get my bearings over there. Until then, I’m sure that you will be absolutely fascinated by non-work-related-but-still-sometimes-geeky minutia I post on Twitter.

Posted By Harry Pierson at 4:22 PM Pacific Standard Time

Wednesday, October 21, 2009

IronPython and IronRuby CTPs for .NET 4.0 Beta 2

VS_v_rgb In case you’ve been hiding under a rock (or maybe just aren’t tracking developments in the .NET community outside of IronPython), Microsoft released Visual Studio 2010 beta 2 this week. Of course for me personally, the most important feature in Visual Studio 2010 is C# 4.0 new dynamic type (also available in Visual Basic, but since VB already supported some level of late binding it’s not exactly “new” to VB).

For those of you who want to experiment with this cool new feature, may I present IronPython 2.6 CTP for .NET 4.0 Beta 2. If you can’t think of any cool things to try with this new feature, the VB team blog has some scenarios to get your started.

Also available: IronRuby CTP for .NET 4.0 Beta 2 if you’re more into gemstones than snakes.

These are preview releases, which means they’ve gone thru basic testing. If you find any bugs, PLEASE report them via the usual channel. I wrote in my Post 2.6 Roadmap post, “we are committed to shipping the RTM of our .NET 4.0 version the day that Visual Studio 2010 is publicly available” but that means shaking out the bugs between now and then. We need your help so we’re ready to go by Visual Studio 2010 launch - March 22, 2010 as per Soma’s blog.

BTW, Alcides Fonseca suggested we call this release “IronPython 2.6 N4” since it’s designed to run on .NET Framework 4.0. I like that. What do you think?

Posted By Harry Pierson at 3:27 PM Pacific Daylight Time

Friday, October 09, 2009

Hybrid App Debugging – The Debug Window

In my last installment, I added support for a separate debug window on a separate thread from the main window thread. That way, I can pause the execution of the main window while the debug window stays responsive to user input. Now, let’s add some functionality to the debug window. I’m going to start by showing the source code of the python file being executed.

private void OnTraceback(TraceBackFrame frame, string result, object payload)
{
    FunctionCode code = (FunctionCode)frame.f_code;
    if (_curCode == null || _curCode.co_filename != code.co_filename)
    {
        _source.Inlines.Clear();
        foreach (var line in System.IO.File.ReadAllLines(code.co_filename))
        {
            _source.Inlines.Add(new Run(line + "\r\n"));
        }
    }

The TraceBackFrame instance has a property f_code that represents the FunctionCode object being executed in this frame. We have to explicitly cast to FunctionCode type because currently we’re exposing all properties that hang off TraceBackFrame as object type. Since Python is a dynamic language, we’re going to use reflection against the instance itself anyway so it doesn’t really matter what the return type is. However, I’ve asked Dino to change the TraceBackFrame type to use explicit types in order to make it easier to use SetTrace from statically typed languages like C#. Look for that in RC2.

After we cast the code object so it can be used from C#, we check to see if the currently loaded file matches the file currently loaded into the UI. I’ve ranted recently about the limitations of WPF’s TextBox but I didn’t want to get hung up syntax highlighting for this sample so I just went ahead and used the RichTextBox. In the DebugWindow Loaded event handler, I create _source as a WPF Paragraph and then wrap it in a FlowDocument and use it as the RichTextBox’s Document. I set the FlowDocument to be extremely wide, so as to avoid word wrapping. Then when I need to load a new source file, I clear _source of it’s current contents and add a single line run for every line of code in the file. This convention becomes useful later when I go to highlight the current line of code.

Once I load the current file, I save the current frame, code, result and payload in instance fields and then switch on result to determine what to do next. Currently, I’m just highlighting the relevant line of code and setting a TextBlock control in the menu bar.

private void TracebackCall()
{
    dbgStatus.Text = string.Format("Call {0}", _curCode.co_name);
    HighlightLine((int)_curFrame.f_lineno, 
        Brushes.LightGreen, Brushes.Black);
}

private void TracebackReturn()
{
    dbgStatus.Text = string.Format("Return {0}", _curCode.co_name);
    HighlightLine(_curCode.co_firstlineno, 
        Brushes.LightPink, Brushes.Black);
}

private void TracebackLine()
{
    dbgStatus.Text = string.Format("Line {0}", _curFrame.f_lineno);
    HighlightLine((int)_curFrame.f_lineno, 
        Brushes.Yellow, Brushes.Black);
}

In Visual Studio, we typically highlight the current line of code in yellow. However, that doesn’t work as great in a language like Python that delineates code blocks with whitespace. In ipydbg, I indicated function return with three carets. But I didn’t want to be modifying the text in the RichTextBox here so instead I used different colors for the different traceback event types: light green for call, light pink for return and yellow for line. The frame object contains the current line number, which I use for call and line, while the code object has the first line of the current code object, which I use for return. HighlightLine highlights the line in question with the colors provided and also scrolls that line into view if it isn’t already visible.

So now when a traceback is handled, it shows the text for the file being executed and highlights the appropriate line, based on the type of traceback event that happened. Now all we need is to have some way be able to continue execution. In the code, you’ll see I’ve defined a series of RoutedUICommands for common debugger commands. I’ve got the StepIn command wired up in the DebugWindow XAML to a menu item and the “S” keystroke. All that remains is to define StepInExecuted.

private void StepInExecuted(object sender, ExecutedRoutedEventArgs e)
{
    dbgStatus.Text = "Running";

    foreach (var i in _source.Inlines)
    {
        i.Background = rtbSource.Background;
        i.Foreground = rtbSource.Foreground;
    }

    _dbgContinue.Set();
}

This function does three basic things: changes the dbgStatus text, resets all the text in the RichTextBox back to the default coloring, and sets the _dbgContinue AutoResetEvent which signals the main window thread that’s been blocked in OnTracebackReceived to continue.

With this post, I’m about even with the code that’s up on GitHub. That code has a few other capabilities – notably it will stop tracing if you close the debug window and it supports StepOut command which disables traceback for the current scope by returning null in OnTracebackReceived. But I haven’t implemented things like:

  • Set Next Statement
  • Viewing and changing variables
  • Debugger REPL
  • Breakpoint Management

Any suggestions on which of those would you like to see next?

Posted By Harry Pierson at 11:18 AM Pacific Daylight Time

Thursday, October 08, 2009

Hybrid App Debugging – Threading

I added traceback to my GetThings app in just two lines of code, but so far it doesn’t actually do anything that you would expect a debugger to do. But before we get to that, we need understand a little about how threading works for traceback debugging.

As I mentioned last time, the traceback debugger works by calling into the registered traceback handler at various times (entering/exiting a function, before executing a line of code and on exceptions). Execution of the Python code continues when the traceback function exits. That means that you have to block the execution thread while you let the user poke around with the debugger UI. For a console based app, that’s easy. For a GUI app, not so much.

At a minimum, you need to run your debugger UI on a separate thread from your main app window. If you want your main app window to be responsive while you debug, you’ll need to pump messages at a minimum (DoEvents in Windows Forms, similar approaches are available for WPF) or preferably run your python scripts on a background thread separate from either the main window UI thread or the debugger UI thread. To keep things simple, I’m going to simply block the main window thread while the debugger is active.

Since I’m going to have to setup a new thread for the debugger window, I decided to use a static constructor to centralize creating the thread, creating the window and registering the traceback handler all in one place.

static Thread _debugThread;
static DebugWindow _debugWindow;
static ManualResetEvent _debugWindowReady = new ManualResetEvent(false);

public static void InitDebugWindow(ScriptEngine engine)
{
    _debugThread = new Thread(() =>
    {
        _debugWindow = new DebugWindow(engine);
        _debugWindow.Show();
        Dispatcher.Run();
    });
    _debugThread.SetApartmentState(ApartmentState.STA);
    _debugThread.Start();

    _debugWindowReady.WaitOne();
    engine.SetTrace(_debugWindow.OnTracebackReceived);
}

As you can see, InitDebugWindow spins up a new thread and creates the debug window on that thread. Since it’s not the main WPF application thread, you have to explicitly call Dispatcher.Run to get the event queue pumping. You also have to explicitly set the apartment state to be single threaded for any threads creating WPF objects. Finally, I wait for the window to signal that it’s ready (it set’s the _debugWindowReady AutoResetEvent in the Window Loaded event) and then call SetTrace, passing in the debug window’s OnTracebackReceived event, on the thread that called InitDebugWindow.

It’s critical that you call SetTrace – and thus InitDebugWindow – on the thread that’s going to execute the Python code. Debugging in Python is per thread. Even if you execute the same code in the same ScriptScope with the same ScriptEngine but on a different thread, the traceback handler calls won’t fire. The way DebugWindow is written, it will only support debugging a single thread, but it would be pretty straightforward to support multiple threads by changing the way OnTracebackReceived gets signaled to continue.

Speaking of OnTracebackReceived, this was my initial basic implementation of it:

private TracebackDelegate OnTracebackReceived
    (TraceBackFrame frame, string result, object payload)
{
    Action<TraceBackFrame, stringobject> tbAction = this.OnTraceback;
    this.Dispatcher.BeginInvoke(tbAction, frame, result, payload);
    _dbgContinue.WaitOne();
    return this.OnTracebackReceived;
}

As we saw, the DebugWindow is running on a different thread than the traceback handler call will come in on. So OnTracebackReceived needs to invoke a new call on the correct thread by using Dispatcher.BeginInvoke. Even though OnTracebackReceived is always called on the main window thread, it still has access to the properties of the debug window thread like its Dispatcher. I used BeginInvoke to invoke OnTraceback asynchronously – OnTraceback isn’t going to return anything interesting and we’re going to wait on an AutoResetEvent before continuing anyway so I didn’t see any reason to use a synchronous call.

We’ll discuss OnTraceback more next post, but basically it will configure the UI for the traceback event that happened. Then DebugWindow will wait for user input. When the user indicates they want to resume execution, the command handler in question will set _dbgContinue and the original traceback will return so execution can continue.

Posted By Harry Pierson at 6:42 PM Pacific Daylight Time

Wednesday, October 07, 2009

Hybrid App Debugging Aside - The DLR Hosting API

In my series on Hybrid App Debugging, I showed the following code for executing a Python file in a hybrid C#/IronPython app.

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    ScriptEngine engine = Python.CreateEngine();
    ScriptScope  scope = engine.CreateScope();
    scope.SetVariable("items", lbThings.Items);
    engine.ExecuteFile("getthings.py", scope);
}

The DLR Hosting API has three distinct levels of functionality. As simple as this is, technically it’s level 2 since it’s using a ScriptEngine directly. If you wanted to use the simplest level 1 hosting API, you could use runtimes instead of engines and save a line of code.

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    runtime.Globals.SetVariable("items", lbThings.Items);
    runtime.ExecuteFile("getthings.py");
}

The ScriptRuntime version of ExecuteFile doesn’t include an overload that takes a ScriptScope like ScriptEngine does, so instead you add the items variable to the globals scope. However, this doesn’t automatically add the items object to every child scope – you have to explicitly import items into the local scope if you want to use it. So for Python, that means you need to add “import items” to the top of the GetThings.py script. Nothing else changes.

Personally, I find DLR Hosting API Level 2 to be straightforward and easy enough to understand, so I tend to code to that level by default. I actually had to go read the doc to discover the ScriptRuntime.Globals property and talk to Dino about importing those variables into a local scope. However, I wanted to point out that nothing in my Hybrid App Debugging sample so far is really dependent on the level 2 API. If you just want to execute some Python files in the context of your C# application, you can stick with the simpler level 1 API if you want. You can even use lightweight debugging with the level 1 API – there’s an overload of the SetTrace extension method for ScriptRuntimes just as there is for ScriptEngines. Just something to keep in mind.

Posted By Harry Pierson at 3:58 PM Pacific Daylight Time
Change Congress
Recent Bookmarks
Tags .NET Framework (2) __clrtype__ (9) ADO.NET (5) Agile (7) AJAX (3) Architecture (288) Guidance (6) Interop (2) Modelling (61) Patterns (7) Process (4) SOA (94) Web Services (5) ASP.NET (25) Async Messaging (2) Azure (1) Battlestar Galactica (3) BI (2) BizTalk (4) Blogging (117) dasBlog (11) Podcasting (4) BPM (1) C# (11) C++ (4) Capitals (5) CardSpace (3) CLR (2) CodePlex (1) College Football (10) Comedy Central (1) Community (81) Concurrency (6) Consumer Electronics (1) Database (13) Debugger (23) Dependency Injection (2) Development (122) C Plus Plus (1) Embedded (5) Lanugages (42) Media (2) P2P (11) Rotor (1) SharePoint (6) SOP (3) DIY (1) DLR (25) Domain Specific Languages (15) Durable Messaging (5) Dynamic Languages (12) Dynamic Silverlight (1) Education (3) Enterprise 2.0 (1) Entertainment (14) ETech (15) F# (51) Functional Programming (17) Game Development (2) Guidance Automation (3) Hardware (8) HawkCodeBox (1) HawkEye (3) Health (1) Hockey (31) Home Electronics (1) Home Network (5) Hosting API (1) Humor (5) IASA (1) Idempotence (3) infrastructure (5) Instrumentation (4) Integration (2) IronPython (112) IronRuby (16) Java (2) Job (3) Kodu (1) LangNET (2) Lightweight Debugger (5) LINQ (23) Live Framework (3) Live Mesh (2) Lost (1) Master Data Management (1) Media 2.0 (6) Microsoft (31) MIX06 (2) Mobile Phone (1) Monads (5) Morning Coffee (172) Object Oriented (4) Office (5) Open Source (8) Open Space (2) Operations (3) Other (135) Art (1) Books (1) Family (33) Games (18) General Geekery (27) Home Theater (1) Movies (23) Music (20) Politics (3) Society (1) Sports (37) Working at MSFT (19) Parallel Programming (3) Parsing Expression Grammar (16) patterns & practices (2) PDC08 (5) Politics (48) Polyglot (3) PowerPoint (2) PowerShell (39) Presentation (7) Projects (1) HawkWiki (1) Pygments (5) Python (6) Quote of the Day (4) Refactoring (1) Research (2) REST (18) Reuse (5) Robotics (2) Rock Band (4) Rome (5) Ruby (23) Ruby on Rails (1) Sci-Fi (2) Scripting (4) Security (3) Service Broker (14) SharePoint (2) Silverlight (20) Social Software (1) Software + Services (2) Software Design (2) Software Engineering (1) Software Factories (11) Software Industry (1) Space Elevator (1) Spark (1) SQL Server (2) Stephen Colbert (1) TechEd (7) TechEd06 (1) TechRec League (1) Television (6) Travel (7) Unified Client (1) Unit Testing (4) USC (1) UX (1) Virtual PC (2) Visual Basic (3) Visual Studio (20) Volta (2) Washington Capitals (37) WCF (31) Web 2.0 (67) Web Services (7) WF (21) Windows (3) Windows Live (29) Windows Live Writer (3) WPF (8) Xbox (1) Xbox 360 (54) XML (11) XNA (15) Zune (4)
Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion.