Hawk Notes, Volume 1

This is the first in a series of blog posts about Hawk, the engine that powers this site. My plan is to make a post like this for every significant update to the site. We'll see well that plan works.

  • I just pushed out a new version of Hawk on my website. The primary feature of this release is support for ASP.NET 5 Beta 7. I also published the source code up on GitHub. Feedback welcome!
  • As I mentioned in my post on Edge.js, the publishing tools for Hawk is little more than duct tape and bailing wire at this point. Eventually, I'd like to have a dedicated tool, but for now it's a manual three step process:
    1. Run the PublishDraft to publish a post from my draft directory to a local git repo of all my content. As part of this, I update some of the metadata and render the markdown to HTML.
    2. Run my WritePostsToAzure Custom Command to publish posts from my local git repo to Azure. I have a blog post on my custom command infrastructure in the works.
    3. Trigger a content refresh via an unpublished URL.
  • I need to trigger a content refresh because Hawk loads all of the post metadata from Azure on startup. The combined metadata for all my posts is pretty small - about 2/3 of a megabyte stored on disk as JSON. Having the data in memory makes it easy to query as well as support multiple post repositories ( Azure storage and the file system).
  • I felt comfortable publishing the Hawk source code now because I added a secret key to the data refresh URL. Previously, the refresh URL was unsecured. I didn't think giving random, anonymous people on the Interet the ability to kick off a data refresh was a good idea, so I held off publishing source until I secured that endpoint.
  • Hawk caches blog post content and legacy comments in memory. This release also adds cache invalidation logic so that everything gets reloaded from storage on data refresh, not just the blog post metadata.
  • I don't understand what the ASP.NET team is doing with the BufferedHtmlContent class. In beta 7 it's been moved to the Common repo and published as source. However, I couldn't get it to compile because it depends on an internal [NotNull] attribute. I decided to scrap my use of BufferedHtmlContent and built out several classes that implement IHtmlContent directly instead. For example, the links at the bottom of my master layout are rendered by the SocialLink class. Frankly, I'm not sure if rolling your own IHtmlContent class for snippet of HTML code you want to automate is a best practice. It seems like it's harder than it should be. It feels like ASP.NET needs a built-in class like BufferedHtmlContent, so I'm not sure why it's been removed.

The Brilliant Magic of Edge.js

In my post relaunching DevHawk, I mentioned that the site is written entirely in C# except for about 30 lines of JavaScript. Like many modern web content systems, Hawk uses Markdown. I write blog posts in Markdown and then my publishing "tool" (frankly little more than duct tape and bailing wire at this point) coverts the Markdown to HTML and uploads it to Azure.

However, as I went thru and converted all my old content to Markdown, I discovered that I needed some features that aren't supported by either the original implementation or the new CommonMark project. Luckily, I discovered the markdown-it project which implements the CommonMark spec but also supports syntax extensions. Markdown-it already had extensions for all of the extra features I needed - things like syntax highlighting, footnotes and custom containers.

The only problem with using markdown-it in Hawk is that it's written in JavaScript. JavaScript is a fine language has lots of great libraries, but I find it a chore to write significant amounts of code in JavaScript - especially async code. I did try and rewrite my blog post upload tool in JavaScript. It was much more difficult than the equivalent C# code. Maybe once promises become more widely used and async/await is available, JavaScript will feel like it has a reasonable developer experience to me. Until then, C# remains my weapon of choice.

I wasn't willing to use JavaScript for the entire publishing tool, but I still needed to use markdown-it 1. So I started looking for a way to integrate the small amount of JavaScript code that renders Markdown into HTML in with the rest of my C# code base. I was expecting to have to setup some kind of local web service with Node.js to host the markdown-it code in and call out to it from C# with HttpClient.

But then I discovered Edge.js. Holy frak, Edge.js blew my mind.

Edge.js provides nearly seamless interop between .NET and Node.js. I was able to drop the 30 lines of JavaScript code into my C# app and call it directly. It took all of about 15 minutes to prototype and it's less than 5 lines of C# code.

Seriously, I think Tomasz Janczuk must be some kind of a wizard.

To demonstrate how simple Edge.js is to use, let me show you how I integrated markdown-it into my publishing tool. Here is a somewhat simplified version of the JavaScript code I use to render markdown in my tool using markdown-it, including syntax highlighting and some other extensions.

// highlight.js integration lifted unchanged from 
// https://github.com/markdown-it/markdown-it#syntax-highlighting
var hljs  = require('highlight.js');
var md = require('markdown-it')({
  highlight: function (str, lang) {
    if (lang && hljs.getLanguage(lang)) {
      try { 
        return hljs.highlight(lang, str).value;
      } catch (__) {}
    }

    try {
      return hljs.highlightAuto(str).value;
    } catch (__) {}

    return ''; 
  }
});

// I use a few more extensions in my publishing tool, but you get the idea
md.use(require('markdown-it-footnote'));
md.use(require('markdown-it-sup'));

var html = return md.render(markdown);

As you can see, most of the code is just setting up markdown-it and its extensions. Actually rendering the markdown is just a single line of code.

In order to call this code from C#, we need to wrap the call to md.render with a JavaScript function that follows the Node.js callback style. We pass this wrapper function back to Edge.js by returning it from the JavaScript code.

// Ain't first order functions grand? 
return function (markdown, callback) {
    var html = md.render(markdown);
    callback(null, html);
}

Note, I have to use the callback style in this case even though my code is syncronous. I suspect I'm the outlier here. There's a lot more async Node.js code out in the wild than syncronous.

To make this code available to C#, all you have to do is pass the JavaScript code into the Edge.js Func function. Edge.js includes a embedded copy of Node.js as a DLL. The Func function executes the JavaScript and wraps the returned Node.js callback function in a .NET async delegate. The .NET delegate takes an object input parameter and returns a Task<object>. The delegate input parameter is passed in as the first parameter to the JavaScript function. The second parameter passed to the callback function becomes the return value from the delegate (wrapped in a Task of course). I haven't tested, but I assume Edge.js will convert the callback function's first parameter to a C# exception if you pass a value other than null.

It sounds complex, but it's a trivial amount of code:

// markdown-it setup code omitted for brevity
Func<object, Task<object>> _markdownItFunc = EdgeJs.Edge.Func(@"
var md = require('markdown-it')() 

return function (markdown, callback) {
    var html = md.render(markdown);
    callback(null, html);
}");
  
async Task<string> MarkdownItAsync(string markdown)
{
    return (string)await _markdownItFunc(markdown);
}

To make it easier to use from the rest of my C# code, I wrapped the Edge.js delegate with a statically typed C# function. This handles type checking and casting as well as provides intellisense for the rest of my app.

The only remotely negative thing I can say about Edge.js is that it doesn't support .NET Core yet. I had to build my markdown rendering tool as a "traditional" C# console app instead of a DNX Custom Command like the rest of Hawk's command line utilities. However, Luke Stratman is working on .NET Core support for Edge.js. So maybe I'll be able to migrate my markdown rendering tool to DNX sooner rather than later.

Rarely have I ever discovered such an elegant solution to a problem I was having. Edge.js simply rocks. As I said on Twitter, I owe Tomasz a beer or five. Drop me a line Tomasz and let me know when you want to collect.


  1. I also investigated what it would take to update an existing .NET Markdown implementation like CommonMark.NET or F# Formatting to support custom syntax extensions. That would have been dramatically more code than simply biting the bullet and rewriting the post upload tool in JavaScript.