The Lounge Survey

I just joined The Lounge advertising network so I wanted to pass along an opportunity to win a bunch of great technical books.

The Lounge is asking the readers of the blogs in their network to fill out a survey in order for them to improve how they target their advertising. It’s pretty much what you would expect from an advertising network focused on the .NET development platform: what language(s) do you use, what framework(s), what testing tool(s), etc, etc, etc. Takes like three minutes to fill out at most.

We all know that filling out surveys isn’t what most people consider “exciting” or “fun”. In order to incent you, dear Reader, to take a few minutes of your valuable time to fill out the survey, The Lounge is giving away all forty one of Manning’s “In Action” books, including IronPython In Action. Even if you don’t win, you still get 40% discount off any purchase from Manning.

So it’s up to you, a scant few minutes of your time in exchange for a chance to win enough technical books to keep you busy for months.

Introducing __clrtype__ Metaclasses

Everyone knows Anders announced at PDC08 that C# 4.0 will include new features (aka the dynamic keyword + the DLR) that makes it much easier for C# to call into dynamically typed code. What you probably don’t know is that IronPython 2.6 includes a new feature that makes it easier for IronPython code to be called by statically typed code.

While the vast majority of .NET is available to IronPython, there are certain APIs that just don’t work with dynamic code. In particular, any code that uses Reflection over an object’s CLR type metadata won’t work with IronPython. For example, while WPF supportsICustomTypeDescriptor, Silverlight only supports data binding against reflectable properties. Furthermore, any code that uses custom attributes inherently uses Reflection. For example, Darrel Hawley recently blogged a WCF host he wrote in IronPython, but he wrote the WCF service in C#. You can’t write WCF services in IronPython because WCF expects service classes to be adorned with ServiceContract and OperationContract attributes (among many others). IronPython users want access to use these APIs. Support for custom attributes is one of the most common requests we get – it’s currently the 5th highest vote getter among open issues.

In IronPython 2.6, we’re adding the ability to customize the CLR type of Python classes. This means you can add custom attributes, emit properties, whatever you want. For those of you who’ve been dreaming of implementing WCF services or databinding in Silverlight purely in IronPython, then this is the feature for you.

In a nutshell, IronPython 2.6 extends Python’s metaclass feature that lets you to customize the creation of classes. In the metaclass, you can implement an IronPython-specific method __clrtype__ which returns a custom System.Type of your own creation that IronPython will then use as the underlying CLR type of the Python class. Implementing __clrtype__ gives you the chance to implement whatever reflectable metadata you need: constructors, fields, properties, methods, events, custom attributes, nested classes, whatever.

Over a series of posts, I’ll be demonstrating this new feature and implement some common scenario requests – including Silverlight databinding and WCF services – purely in Python. Quick warning: __clrtype__ uses low level features like Python metaclasses, Reflection.Emit and DLR Binders so these posts will be deeper technically than usual. Don’t worry – this isn’t the API interface we expect everyone to use. Eventually, we want to have an easy to use API that will sit on top of the low-level __clrtype__ hook.

Issue Tracking for my GitHub Projects

FYI, anyone using ipydbg, devhawk_ipy or Pygments for WL Writer can now submit issues, bugs and feature requests other the associated GitHub project page. Head to the associated GitHub project page and click the “Issues” tab at the top. There’s a short video of introducing the GitHub Issues feature you can check out.

BTW, thanks to defunkt @ GitHub for the ipydbg shoutout.

Joining the Lounge Advertising Network

For those of you who read this blog primarily via my RSS feed, I made a change to my homepage over the weekend. Goodbye adSense, hello The Lounge. The Lounge is an ad network run by James Avery’s company InfoZerk. I’ve known James for a while – he included my now-obsolete SccSwitch tool in his book Visual Studio Hacks.

From a financial perspective, I’m not really sure how much of a difference this will make. I guess I’ll see when I get my first check. Given how little I was making with adSense, I’ve got nowhere to go but up. Regardless, I feel much better working with a smaller ad network run by someone I respect and that is focused on the .NET platform.

Thanks for having me in the Lounge ad network, James.

Writing an IronPython Debugger: Breakpoint Management

Setting a breakpoint was the second feature I implemented in ipydbg. While setting a breakpoint on the first line of the Python file being run is convenient, it was obviously necessary to provide the user a mechanism to create their own breakpoints, as well as enable and disable existing breakpoints.

First thing I had to do was to refactor the create_breakpoint method. Originally, I was searching thru the symbol documents looking for the one that matched the filename in OnUpdateModuleSymbols. However, since I wanted to specify by new breakpoints via the same filename/line number combination, it made more sense to move symbol document logic into create_breakpoint:

def create_breakpoint(module, filename, linenum):
    reader = module.SymbolReader
    if reader == None:
      return None

    # currently, I'm only comparing filenames. This algorithm may need

    # to get more sophisticated to support differntiating files with the

    # same name in different paths

    filename = Path.GetFileName(filename)
    for doc in reader.GetDocuments():
      if str.Compare(filename, Path.GetFileName(doc.URL), True) == 0:
        linenum = doc.FindClosestLine(linenum)
        method = reader.GetMethodFromDocumentPosition(doc, linenum, 0)
        function = module.GetFunctionFromToken(method.Token.GetToken())

        for sp in get_sequence_points(method):
          if sp.doc.URL == doc.URL and sp.start_line == linenum:
            return function.ILCode.CreateBreakpoint(sp.offset)

        return function.CreateBreakpoint()

The new version isn’t much different than the old. It loops thru the symbol documents looking for one that matches the filename argument. Then it creates the breakpoint the same way it did before. Eventually, I’m going to need a better algorithm than “only compare filenames”, but it works for now.

Once I made this change, it was trivial to implement a breakpoint add command. What was harder was deciding on the right user experience for this. I decided that breakpoint management was going to be the first multi-key command in ipydbg. so all the debug commands are prefixed with a “b”. I use the same command routing decorator I used for input commands. As you can see, my breakpoint command looks a lot like my top level input method – read a key from the console then dispatch it via a commands dictionary that gets populated by @inputcmd decorators.

@inputcmd(_inputcmds, ConsoleKey.B)
def _input_breakpoint(self, keyinfo):
    keyinfo2 = Console.ReadKey()
    if keyinfo2.Key in IPyDebugProcess._breakpointcmds:
        return IPyDebugProcess._breakpointcmds[keyinfo2.Key](self, keyinfo2)
    else:
        print "nInvalid breakpoint command", str(keyinfo2.Key)
        return False

Currently, there are four breakpoint commands: “a” for add, “l” for list, “e” for enable and “d” for disable. List is by far the simplest.

@inputcmd(_breakpointcmds, ConsoleKey.L)
def _bp_list(self, keyinfo):
  print "nList Breakpoints"
  for i, bp in enumerate(self.breakpoints):
    sp = get_location(bp.Function, bp.Offset)
    state = "Active" if bp.IsActive else "Inactive"
    print "  %d. %s:%d %s" % (i+1, sp.doc.URL, sp.start_line, state)
  return False

As you can see, I’m keeping a list of breakpoints in my IPyDebugProcess class. Originally, I used AppDomain.Breakpoints list, but that only returns enabled breakpoints so I was forced to store my own list. Note also that I’m using the enumerate function, which returns a tuple of the collection count and item. I do this so I can refer to breakpoints by number when enabling or disabling them:

@inputcmd(_breakpointcmds, ConsoleKey.E)
def _bp_enable(self, keyinfo):
  self._set_bp_status(True)

@inputcmd(_breakpointcmds, ConsoleKey.D)
def _bp_disable(self, keyinfo):
  self._set_bp_status(False)

def _set_bp_status(self, activate):
  stat = "Enable" if activate else "Disable"
  try:
    bp_num = int(Console.ReadLine())
    for i, bp in enumerate(self.breakpoints):
      if i+1 == bp_num:
        bp.Activate(activate)
        print "nBreakpoint %d %sd" % (bp_num, stat)
        return False
    raise Exception, "Breakpoint %d not found" % bp_num

  except Exception, msg:
    with CC.Red: print "&s breakpoint Failed %s" % (stat, msg)

Since the code was identical, except for the value passed to bp.Activate, I factored the code into a separate _set_bp_status method. After the user presses ‘b’ and then either ‘e’ or ‘d’, they then type the number of the breakpoint provided by the breakpoint list command. _set_bp_status then simply iterates thru the list until it finds the matching breakpoint and calls Activate. Note that since it’s possible to have 10 or more breakpoints, I’m using ReadLine instead of ReadKey, meaning you have to hit return after you type in the breakpoint number.

Finally, I need a way to create new breakpoints. With the refactoring of create_breakpoint, this is pretty straightforward

@inputcmd(_breakpointcmds, ConsoleKey.A)
def _bp_add(self, keyinfo):
  try:
    args = Console.ReadLine().Trim().split(':')
    if len(args) != 2: raise Exception, "Only pass two arguments"  
    linenum = int(args[1])

    for assm in self.active_appdomain.Assemblies:
      for mod in assm.Modules:
          bp = create_breakpoint(mod, args[0], linenum)
          if bp != None:
            self.breakpoints.append(bp)
            bp.Activate(True)
            Console.WriteLine( "Breakpoint set")
            return False
    raise Exception, "Couldn't find %s:%d" % (args[0], linenum)

  except Exception, msg:
    with CC.Red:
      print "Add breakpoint failed", msg

Most of _bp_add is processing the input arguments, looping through the modules and then storing the breakpoint that gets returned. When I set the initial breakpoint inside OnUpdateModuleSymbols, I have the module with updated symbols as an event argument. However, in the more general case we’ve got no way of knowing which module of the current app domain contains the filename in question. So we loop thru all the modules, calling create_breakpoint on each until one returns a non-null value. Of course, “all the modules” will include the IronPython implementation, but assuming you’re running against released bits the call to create_breakpoint will return right away if debug symbols aren’t available.

As usual, the latest version is up on GitHub. This will be the latest update to ipydbg for a little while. I worked on it quite a bit while I was at PyCon and have been busy with other things since I got home. Don’t worry, I’ll come back to it soon enough. As I mentioned Monday, I want to get function evaluation working so I can have a REPL console running in the target process instead of the one I’ve got currently running in the debugger process.