Morning Coffee 135

• Bill Gates does his last CES Keynote, and we announce a PC that looks like a purse?
• News that Warner Brothers is going exclusively Blu-Ray is disappointing. However, I'm convinced that neither side will win this format war but that online downloads will trump both. Obviously, XBLM is a significant player in this space, but the market is crowding up quickly. Netflix apparently will unveil a new set-top box @ CES to let you watch HD movies via the Internet.
• Don Syme has a roundup of posts by John Liao about F#. Mostly, WPF + F# with a couple of ASP.NET 2.0 posts and one on XML .
• Speaking of F#, Stephan Tolksdorf has been working on an F# port of MS Research's Parsec library called FParsec. Parsec is a "monadic parser combinator library", something I have little experience with, so I've gone back to some source research on the topic, which I hope to blog at length about soon.
• Steve Vinoski talks about serendipitous reuse in his latest Internet Computing article. I'm not a believer in reuse in the enterprise, serendipitous or otherwise, but I liked the conclusion to Steve's article when he wrote "It's highly ironic that many enterprise architects seek to impose centralized control over their distributed organizations. In many cases, such centralization is a sure recipe for failure." Also, his point that "control without controlling" works sounds vaguely familiar.
• Update - This is really Morning Coffee 136, but I don't want to change the title since it's part of the URL

Morning Coffee 134

• Bill de Hora responds to a few of my Durable and RESTful ideas. He points out that relying on a client-generated ID can be troublesome, and recommends using multiple identifiers - one created by the sender, one by the receiver and one representing the message exchange itself. However, the sender ID is vulnerable to client bugs & tampering as Bill points out, and neither the receiver ID nor the exchange ID can be used to determine if a given message is a duplicate. If you don't trust the sender, is it even possible to determine if a given message is a duplicate?
• Pablo Castro confirms that there are "practical limits" to what ADO.NET Data Services can do with respect to idempotence. Nothing in his post was surprising, though I hope it will be more explicitly called out in the final docs. Developers used to the comforting protection of a transaction may be in for a rude awakening.
• Dare Obasanjo has a great post comparing the new features in C# 3.0 to dynamic languages like IronPython. I believe many of the productivity aspects of dynamic languages have little to do with being dynamic.
• Pat Helland noodles on durability and messaging, two topics near and dear to my heart (probably from working with him for a couple of years). I'm not sure where he's going with this - his conclusion that "Basically, big, complex, and distributed system are big, complex, and distributed" isn't exactly ground-breaking. But his point that "durable" isn't a binary concept is worth more consideration. Also, his description of IMS only looking at the effects of a committed transaction is very similar to how web sites work, though obviously HTTP isn't durable so you can't make event horizon optimizations like IMS did.
• Tangentially related, Werner Vogels discusses the idea of eventually consistent distributed databases. Today, that's a problem mostly only Internet-scale sites like Amazon deal with. In the near future of continued data explosion + manycore, we'll all have to deal with it.
• Nick Malik argues that categorizing enterprise applications by lifecycle is much less useful than categorization based on organizational impact. He might also need a new chair.
• Jesus Rodriguez digs into one of SSB's new features in SQL 2008: conversation priorities.
• Arnon Rotem-Gal-Oz and Sam Gentile are mixing it up over the definition of SOA. Sam thinks SOA has to include business drivers and Arnon doesn't. I'm with Sam on this, defining "SOA" independently from "Applying SOA" seems pointless. Then again, rigorously defining SOA - much less arguing about said definition - seems like a waste of time in the first place IMHO.
• Wow, this guy Zed is mad at the Ruby community.
• Andrew Baron has 8 Reasons Why The TV Studios Will Die. Personally, I think reason #2 - Expendable Middle-Person - is the most important. If content producers can reach consumers directly, what value-add will the networks provide? (via United Hollywood)

Durable and RESTful

A while back I wondered if it's still REST if you don't use HTTP. The reason I wondered that is because like many I've become disillusioned with the WS-* stack over time and see REST as a viable alternative to all that spec-driven complexity. However, just because I'm looking to REST means I'm willing to give up on durable messaging. So I shouldn't be asking "can I do REST without HTTP?" I should be asking "what protocol can I use to do durable messaging with REST?"

It turns out HTTP is just fine for RESTful durable messaging, if you take the time to make your POSTs idempotent. There's even a IETF RFC that builds on HTTP and specifies a mechanism to do it.

As I wrote last month, idempotence is critically important to ensuring "things" happen exactly once when connecting disparate systems together. At the end of that post, I asked you, dear reader, to contemplate just how durable messaging systems ensures exactly once delivery. They do it by assigning messages to be delivered a unique identifier. Any non-idempotent operations can be made idempotent with unique identifiers and a message ID log.

"Not Idempotent:
     Withdrawing $1 Billion.
Idempotent:
     If Haven’t Yet Done Withdrawal #XYZ for $1 Billion,
     Then Withdraw $1 Billion and Label as #XYZ"
Pat Helland

For example, when you send a message in MSMQ, it's assigned a 20 byte identifier which is "unique within your enterprise." [1] If the destination system receives multiple messages with the same message ID, it knows they are duplicates and can safely toss all but one of the messages with the same ID. Exactly once, no transactions.

While many operations in REST are naturally idempotent, using REST doesn't magically make all your operations idempotent, contrary to popular belief. Have you ever seen a message like "please don't press submit order twice" on the checkout page of an e-commerce website? It's there because POST is not naturally idempotent and the site hasn't taken any extra steps to identify duplicate POSTs. If the site embedded a unique ID in a hidden form field, it could use that to identify duplicate orders.

If you're a RESTifarian, haven't you seen this approach somewhere before?

Given that POST isn't naturally idempotent, I think it's kinda surprising that new resources are created in AtomPub by POSTing them to a collection rather than PUTting them to a specific URL. RESTful Web Services specifically points out that PUT is idempotent, so I wonder why AtomPub uses POST. I'd guess most AtomPub implementations (aka blogs) aren't much concerned about ensuring Exactly Once. If an blog entry gets posted twice, you delete one and go on with your life.

However, if you wanted to use AtomPub and ensure Exactly Once, you can use the fact that Atom entries must contain exactly one ID element which as per the spec must be universally unique. From reading the Atom spec, the ID element seems primarily designed for Atom feed consumers, but AtomPub servers could also use it as an "idempotence identifier", similar to how MSMQ uses the message ID. If you end up with multiple entries with the same entry ID, discard all but one.

So by creating a unique identifier on the client side and logging that identifier on the server side, we can make any REST service idempotent. We can make it a durable service if we write the outgoing message - with the message ID we generate - to a durable store before trying to send it. If you write it to a durable store within the scope of a local transaction, you're even closer to duplicating MSMQ's functionality, yet the only protocol requirement beyond vanilla HTTP is having a unique message ID.

The one problem with the Atom entity ID approach is that it requires cracking the message in order to see if we should process it. For REST services, I would think we'd want to stick the idempotence identifier in an HTTP header. We already headers to implement conditional GET, why not a header for what amounts to conditional POST?

Turns out such a header exists in the AS2 spec, i.e. "MIME-Based Secure Peer-to-Peer Business Data Interchange Using HTTP". AS2 defines a Message-Id HTTP header which "SHOULD be globally unique". In the case of an HTTP error, AS2 specifies the "POST operation with identical content, including same Message-ID, SHOULD be repeated" and that "Servers SHOULD be prepared to receive a POST with a repeated Message-ID." I assume this implies a server shouldn't process a message with the same ID twice.

So what would a durable REST service look like? I think like this:

1. Sending system records the intent to send a message by saving it to a local durable store, potentially in the scope of a local transaction. As part of saving the message, a unique message id is generated (I'd use a GUID, but as long as it's unique it doesn't matter.)
2. A background thread in the sending system monitors the durable message store. When a new to-be-sent message arrives, the thread POSTs it to the destination, setting the Message-Id HTTP header to the unique identifier generated in step 1.
3. The receiving system stores the Message-Id header value in a log table and processes the received message, potentially in the scope of a local transaction. Optionally, it can store the return message (if there is one) in the durable store as well.
4. If the sending system doesn't receive a 2xx status code, it rePOSTs the message to the receiving system until it does.
5. If the receiving system receives a message that's already listed in the log table, it ignores it and returns a success status code. Optionally, if the return message has been saved, the receiving system can resend the return message as long as it doesn't redo the work.

This seems like a better approach than my original direction of doing REST over a durable protocol like MSMQ or SSB. What do you think?

UPDATE - Erik Johnson points out that an HTTP POST's idempotency is "left unsaid". So my statement that "POST isn't idempotent" isn't quite correct. POST isn't naturally idempotent. I've updated the post accordingly.

[1] Technically, the MSMQ message ID isn't universally unique as it is a 16 byte GUID representing the source system + a 4 byte sequence number. The sequence number can rollover, after sending 2^32 messages. In practice, rolling over the message ID after 4 billion messages is rarely an issue.

Morning Coffee 124

• While my blog was down last week, I finally finished Gears of War. I played thru on hardcore, but had to throttle back to casual to beat the last boss. I'd like to try and finish on hardcore, but I've moved on to Dead Rising - another game from last year I never had time to finish. I'm almost done the main play mode, though I understand there are other play modes that get unlocked when you finish it.
• I'm forbidden from buying any new games before Christmas, so Mass Effect, Assassin's Creed and The Orange Box will have to wait. My next game will either be Blue Dragon, which a friend let me borrow, or R6:Vegas, yet another (but the last) game from last year I never got time to play.
• I'll skip the "giving thanks" jokes and point out that Visual Studio 2008 and .NET FX 3.5 have shipped.  Soma has the announcement and both Scott Guthrie and Sam Gentile summarize what's new. The Express editions are available from the new Express Developer Center. The VS SDK doesn't appear to be released yet, but I'm sure it will be along in due course.
• Speaking of VS SDK, CoDe Magazine did an entire issue on VS Extensibility which you can read online or download as PDF.
• Nick Malik took a bunch of heat back in June for what some thought was a redefinition of Mort, one of the Developer Division personas. Now Paul Vick thinks it's time to retire the Mort persona, primarily because of the negative connotation the name carries. His suggestion for a replacement is Ben (as in Franklin). And did you notice how similar Paul's description of Mort is to what Nick described? I'd say some folks owe Nick an apology.
• I said Friday I was going to take a closer look @ OpenID and OAuth. There's an intro to OpenID on their wiki and Sam Ruby's OpenID for non-SuperUsers seems to be the canonical source on implementing OpenID on your own blog. Frankly, reading the OpenID intro reminded me a lot of WS-Federation Passive Requestor Profile. Does OpenID have the equivalent of an "active" mode?
• Likewise, the Beginner’s Guide to OAuth series of posts by Eran Hammer-Lahav is a good intro to OAuth. The phrase "Jane notices she is now at a Faji page by looking at the browser URL" from the protocol walkthru makes me worry that OAuth is vulnerable to phishing. Having one of the OAuth authors call phishing victims careless and wishing for Karl Rove to "scare people into being more careful and smarter about what they do online" makes me think my fears are well grounded. I'm thinking maybe OAuth and OpenID aren't quite ready to nail down WS-*'s coffin.
• In researching OpenID, I came across this presentation hosted on SlideShare. I had never seen SlideShare before - it's kinda like YouTube for presentations. Sharing basic presentations is kinda lame - there doesn't appear to be any animation support, so the slides are basically pictures. However, they also support "slidecasting" where you sync slides to an audio file hosted elsewhere. That I like. I have a bunch of old decks + audio, maybe I'll stick them up there.

Afternoon Coffee 123

• Morning Coffee is late this morning because we went for our Christmas portrait this morning and it took forever. The pictures turned out great though.
• Nick Malik finishes up his series on business operation models by covering the diversification model. Also, Nick's points about the synergy between a diversified model and the coordinated model are spot on. I happen to be a big fan of those models (aka the models with low standardization) which probably drives some of the  more my "unique" perspectives on SOA.
• Scott Guthrie starts out a new series and future technology, this time it's ASP.NET MVC Framework that gets the series treatment. The first entry in the series is a general overview. I wonder why there's no cool code name for the MVC framework? Whatever it's named, I like the auto routing and action rules - it seems very Rails-inspired.
• Over the weekend, Don Box points out that the REST authentication story "blows chunks". I've recently given up on the reliable part of the original "Secure, Reliable, Transacted Web Services" vision - and I never believed the transacted part. Security, on the other hand, is the one part of that original vision that has worked out IMO. My experience with the WS-* security stack has been pretty good, though Dare Obasanjo thinks that OpenID and OAuth are the final nail in the WS-* coffin.
• Speaking of Dare, he goes on to say WS-* is to REST as Theory is to Practice. He makes the point that "The only times I encounter someone with good things to say about WS-* is if it is their job to pimp these technologies or they have already “invested” in WS-* and want to defend that investment." I gave up pimping evangelizing technology a while back and I don't want to be in the position of defending a bad investment, so I'm spending lots of time looking at REST.
• Jesus Rodriguez takes a look at the Managed Services Engine and comes away excited. Jesus is a self-described "strong believer" in SOA governance. I'm a self-described strong disbeliever in SOA governance, so MSE sounds like more of the Worst of Both Worlds to me.
• A little light reading: I pulled Applied Cryptography and A New Kind of Science out of my garage last weekend. Plus my copies of RESTful Web Services and Programming Erlang just arrived yesterday.

Morning Coffee 122

• Sorry for the posting lag. Had a few technical difficulties around here. In the process of moving hosts, so expect more glitches.
• My talk at the p&p Summit on Monday went really well. At least, it felt good and the applause at the end felt genuine. I recorded the audio on my laptop, so I'll be posting a Silverlight version as soon as I figure out how to adjust the levels so their somewhat consistent. Paraesthesia and #2872 have reactions.
• Speaking of the p&p Summit, Scott Hanselman posted his ASP.NET MVC demo from his talk. Said ASP.NET MVC bits aren't available yet, so you can't, you know, run the demo for yourself. But at least you can review what the ASP.NET MVC code will look like.
• I stopped by the SOA/BPM conference last week and saw Jon, Sam and Jesus among others. Spent quite a bit of time talking to Sam and his Neudesic colleagues about this "physically distributed/logically centralized" approach that I think is hogwash. It sounds to me like Neudesic approach is really federated not centralized, though I'm not sure David Pallmann would agree. Federated makes much more sense to me than centralized.
• Nick Malik continues his series on SOA Business Operations Model. I especially like his point that this isn't a series of choices, you need to "look at your company and try to understand which model the business has selected. "
• The first CTP of PowerShell 2.0 is out! Check out what's new on the PowerShell team blog and Jeffrey Snover's TechEd Presentation. (via Sam Gentile)
• Soma announced updates to VC++ coming next year, including TR1 support and a "major" MFC upgrade to support creating native apps that look like Office, IE or VS. I get supporting TR1, but the idea that people are clamoring for MFC updates is kinda surprising. Many years ago when I first came to MSFT, a friend asked "But don't you hate Microsoft?" to which I responded "No, I just hate MFC". Obviously, not everyone agrees with that sentiment.
• Steve Vinoski thinks there's no hope for IT. Funny, I keep agreeing with Steve's overall point but disagreeing with his reasoning. I still don't buy the serendipity argument. I like compiled languages. And I think he's overstating the amount of "real, useful guidance" for REST floating around. Basically, there's "the book".
• In widely reported news, Windows Live launched their next generation services. Don't bother with the press release, just go to the new WL home page.
• Speaking of WL, Dare Obasanjo points to the Live Data Interactive SDK page where you can experiment with the WL Contacts REST API. It gives you a good sense of how the Web3S protocol works. Pretty well, IMO. However, how come WL Contacts Schema doesn't include some type of update timestamp for sync purposes? If you wanted to build say a Outlook <--> WL Contacts sync engine, you'd have to download the entire address book and grovel thru it for changes every sync.
• Speaking of Web3S, I'd love to see some info on how one might implement a service using Web3S. Yaron Goland positions Web3S as an alternative to APP that WL developed because they "couldn't make APP work in any sane way for our scenarios". I'm sure other folks have similar scenarios.

Morning Coffee 121

• My daughter had her tonsils & adenoids out on yesterday. It was a routine procedure and it went by-the-numbers, but any parent will tell you it's hard to see your kid in a hospital bed.
• Given the previous bullet, I'm not at the SOA/BPM conference for the big announcement. Don't worry, there's lots of other folks covering the news.
• It was a crappy sports weekend in the Pierson house. Va Tech snatched defeat from the jaws of victory, Southern Cal never led at Oregon, the Capitals lost twice, and the Redskins got blown out by the Pats. At least the Caps won big yesterday in Toronto.
• Speaking of the Capitals, Peter Bondra retired Monday. I still think it's a travesty that he didn't spend his whole career in DC, but I've made my peace with it.
• Nick Malik has a great series on business operations models and how they apply to SOA. Regular readers should be unsurprised that I favor low standardization, though I can see the value of high integration. That makes the Coordinated Operating Model my fav, though I can see the benefit of the Diversified Model as well. I can't wait to read what Nick has to say on changing models.
• Speaking of Nick, I'm doing a roundtable with him on "Making SOA Work in the Enterprise" @ the Strategic Architect Forum. Should be fun. Sorry for the lack of linkage on this, but it's an invite-only event.
• Jezz Santos has a new series of white papers on building software factories. First up "Packaging with Visual Studio 2005"
• Aaron Skonnard has a new whitepaper on using the WCF LOB Adapter SDK with BTS 2006 R2. I've been building one of these things recently, so I'm looking forward to checking that out. (via Sam Gentile)
• Tim Ewald looks at Resource Oriented Architecture (when did ROA become a TLA?) and wonders "what if your problem domain is more focused on processes than data?" I wonder that all the time. (via Jesus Rodriguez)
• It's not just durable messaging - Libor Soucek also disagrees with my opinions on centralized control. I agree 100% with Libor that centralized management would make operation's lives "much, MUCH easier" as he puts it. However, that doesn't make it feasible at any significant scale. Furthermore, I wouldn't describe an approach that requires that "all services adopt [the] same common management interface" as "pragmatic". Frankly, just the opposite.

Throwing Gasoline on the Fire

Steve Vinoski has raised a bit of a flame war by admitting he has lost the ESB religion. Given that I've never been a fan of ESB's anyway, there's a lot there that I agree with. In particular I liked the description of "magical framework" middleware, blaming enterprise architects for driving ESB's as the "single integration architecture" even though a single *anything* in the enterprise is untenable and his point that flexibility means you don't do any one thing particularly well.

However, Steve goes on to bash compiled languages and WS-* while suggesting the One True Integration Strategy™ is REST + <insert your favorite dynamic language here>, then acts surprised that the conversation denigrates into "us vs. them". When you start by saying that compiled language proponents "natter on pointlessly", I think you lose your right to later lament the depreciating level of conversation .

All programming languages provide their own unique model of the execution environment.  Dynamic languages have a very different model than compiled languages. Arguing that this or that model is better for everyone, everywhere, in all circumstances seems unbelievably naive and arrogant at the same time.

On the other hand, I do agree with Steve's point that most developers only know a single programming language, to their detriment. One language developers often miss a better solution because their language of choice doesn't provide the right semantics to solve the problem at hand. Developers could do a lot worse than learn a new language. And I don't mean a C# developer should learn VB.

The most pressing example of picking the right language for the right problem today is multi-threading. Most languages - including dynamic languages - have shitty concurrency semantics. If you're building an app to take advantage of many-core processing, "mainstream" apps like C#, Java and Ruby won't help you much. But we're starting to see languages with native concurrency semantics like Erlang. Erlang is dynamically typed, but that's not what makes it interesting. It's interesting because of it's native primitives for spawning tasks. I don't see why you couldn't add similar primitives for task spawning to a compiled functional language like F#.

As for REST vs. SOAP/WS-*, I thought it was interesting that Steve provided no rationale whatsoever for why you should avoid them. The more I listen to this pissing match debate, the more I think the various proponents are arguing over unimportant syntactical details when the semantics are basically the same. SOAP is just a way to add metadata to an XML message, much as HTTP headers are. WS-* provides a set of optional message-level capabilities for handling cross-cutting concerns like security. Past that, are the models really that different? Nope.

For system integration scenarios like Steve is talking about, I'm not sure how important any of the WS-* capabilities are. Security? I can get that at the transport layer (aka HTTPS). Reliable Messaging? If I do request/response (which REST excels at), I don't need RM. Transactions? Are you kidding me? Frankly, the only capability you really need in this scenario is idempotence, and neither REST or SOAP provides any standard mechanism to achieve that. (more on that in a later post)

I understand that some vendors are taking the WS-* specs and building out huge centralized infrastructure products and calling them ESBs. I think Steve is primarily raging against that, and on that point I agree 100%. But Steve sounds like he's traded one religion for another - "Born Again REST". For me, picking the right tool for the job implies much less fanaticism than Steve displays in his recent posts.

Where Have All the SOA Mashups Gone?

John Heintz responded to my serendipitous reuse post. Nice to see I misunderstood his opinions about how easy RESTful systems are to integrate:

I didn't mean to imply that building RESTful system would lead to magical integration without any hard work. I can see how that came across in my post, and I guess I got the reaction I asked for ;)

I get the feeling that John would be a good guy to have a beer with.

John spends most of his post writing about the SOA in the Real World book. I've flipped thru it and I'm familiar with the model (it is my old team after all) but I haven't read it so I don't really want to comment about the book specifically. But there were two things John mentioned that I did want to comment on.

First, at the end of his post John writes:

Can some of the constraints of REST be applied to SOA? Absolutely. I think an asynchronous, message-passing architecture with a uniform interface would be astoundingly interesting! I'm not the only one: see MEST, AMPQ, and Erlang.

This goes back to a REST question I asked two months ago: is it still REST if you don't use HTTP? I'm guessing John would say yes.

I might be going out on a limb here, I'll bet the core of John's problem with SOA is how toolkits like WCF all but force you to build RPC style services that can easily be modeled as method calls. That's certainly one of my problems with SOA. Tim Ewald said it best:

It's depressing to think that SOAP started just about 10 years ago and that now that everything is said and done, we built RPC again. I know SOAP is really an XML messaging protocol, you can do oneway async stuff, etc, etc, but let's face it. The tools make the technology and the tools (and the examples and the advice you get) point at RPC. And we know what the problems with RPC are. If you want to build something that is genuinely loosely-coupled, RPC is a pretty hard path to take.

If SOA == RPC and REST == loosely coupled messages, then I'll start growing dreadlocks right now. Frankly, as Tim says, I think it's a problem with the tools (I'm looking at you WCF) and not the underlying architecture, but how many people can distinguish the architecture from the tools? Not many, I'm afraid.

Second, John asks an interesting question:

Where are the SOA mashups?

That's easy! They're inside the firewall where you can't see them! ;)

Seriously, I'm not sure about "SOA" mashups, but I'm working with what you might call a huge "enterprise" mashup system inside Microsoft. Our Enterprise Data Integration Services push around massive amounts of data to downstream systems. There are over fifty datasets in production, each with scores of tables, millions of rows and hundreds of subscribing systems. One example, our Products dataset, has over 100 tables and nearly 300 subscribing systems.

Is it "service oriented"? No, but then again it was originally developed ten years ago on SQL 6.5. But is it a mashup? Is it an "application that combines content from more than one source into an integrated experience"? Yep. Is it easy to work with? No, but guess why I'm involved? We're looking at ways to "modernize" the system. Am I going to build RPC style services as part of this modernization? Hell, no.

So John, am I right or wrong about that beer?

Is Serendipity the Heart of the WS-*/REST Debate?

Thanks to Technorati, I found this post by John Heintz. He's checking out John Evdemon's e-book on SOA and has a problem with this overview:

SOA is an architectural approach to creating systems built from autonomous services. With SOA, integration becomes forethought rather than afterthought. This book introduces a set of architectural capabilities, and explores them in subsequent chapters.

To which John H. responds:

I, for one, would rather build on an architecture that promotes integration as an afterthought, so I don't have to think about it before hand!!!

Yeah, I'd rather not have to think about integration before hand either. On the other hand, I want integration that actually works. It sounds like John H. is suggesting here that REST somehow eliminates the need to consider integration up front. It doesn't. Consider this: if you're building a Web 2.0 site then you are expected to expose everything in your site via APP, RSS and/or RESTful POX services. In other words, the Web 2.0 community expects you to have the forethought to enable integration. If you don't, Marc Canter will call you out in front of Bill Gates and Tim O'Reilly. 

This integration by afterthought approach seems to be big among RESTifarians. John H. links to a REST discussion post by Nick Gall advocating the principle of generality, "unexpected reuse" and "design for serendipity". Money quote:

The Internet and the Web are paradigms of Serendipity-Oriented Architectures. Why? Largely because of their simple generality. It is my belief that generality is one of the major enablers of serendipity. So here I immodestly offer Gall's General Principle of Serendipity: "Just as generality of knowledge is the key to serendipitous discovery, generality of purpose is the key to serendipitous (re)use."

Serendipity means "the accidental discovery of something pleasant, valuable, or useful". "Serendipitous reuse" sounds an awful lot like accidental reuse. Most enterprises have been there, done that and have nothing to show for their efforts or $$$ except the team t-shirt. Yet Tim Berners-Lee believes "Unexpected reuse is the value of the web" and Roy Fielding tells us to "Engineer for serendipity". What gives?

First off, enterprises aren't interested in unexpected or serendipitous reuse. They want their reuse to be systematic and predictable. The likelihood of serendipitous reuse is directly related to the number of potential reusers. But the number of potential reusers inside the enterprise is dramatically smaller than out on the public Internet. That brings the chance for serendipitous reuse inside the enterprise to nearly zero.

Second, enterprise systems aren't exactly known for their "simple generality". If Nick's right that "generality of purpose is the key to serendipitous (re)use", then enterprises might as well give up on serendipitous reuse right now. As I said last year, it's a question of context. Context is specifics, the opposite of generality. Different business units have different business practices, different geographies have different laws, different markets have different competitors, etc. If an enterprise operates in multiple contexts - and most do - enterprise solutions have to take them into account. Those different contexts prevent you from building usable - much less reusable - general solutions.

Finally, I think the amount of serendipitous reuse in REST is overstated. If you build an app on the Facebook Platform, can you use it on MySpace? Nope. If you build an app that uses the Flickr services, will it work with Picasa Web Albums? Nope. Of course, there are exceptions - pretty much everyone supports the MetaWeblog API - but those exceptions seem few and far between to me. Furthermore, the bits that are getting reused - such as identifier, format and protocol - are infrastructure capabilities more suitable to reuse anyway. Serendipitously reusing infrastructure capabilities is much easier than serendipitously reusing business capabilities, REST or not.

The problems that stand in the way of reuse aren't technology ones. Furthermore, the reuse problems face by enterprises are very different than ones faced by Web 2.0 companies. REST is a great approach, but it isn't a one-size-fits-all technology solution that magically relegates integration and reuse to "afterthought" status. Serendipity is nice, when it happens. However, by definition it's not something you can depend on.

Morning Coffee 91

• My wife loves me. I'm a very lucky man.
• I'm starting to really dig Safari Books Online. Having a tablet really helps here, I can sit in bed and read and it's ALMOST like reading a real book. Is there an offline experience? Something like the NYTimes WPF Reader app would be killer.
• I'm not a Twitter guy, but I like the idea of using it to publish CI results. Not quite as cool as using the Ambient Orb, but close. (via DotNetKicks)
• Soma details the dogfood usage of TFS in Developer Division. Sorta interesting if you're into knowing that stuff. Brian Harry apparently has much more.
• I realize that linking to Pat Helland every time he writes something is fairly redundant. If you want his feed, you know where to find it. But he writes great stuff! The latest is Accountants Don't Use Erasers, which talks about append-only computing. His point that the database is a cache of the transaction log is mind blowing, yet makes total sense.
• Bruce Payette blogs a PS DSL for creating XML documents.
• Jesus Rodriguez details WCF's new Durable Service support in .NET 3.5. I get the need for the [DurableServiceBehavior] attribute, but do I really have to adorn each of the service methods with [DurableOperationBehavior] too? That seems redundant. Also, I wonder how this looks at the channel layer?
• Speaking of WCF's channel layer, I recently picked up a copy of Inside Windows Communication Foundation by Justin Smith. This is the first book I've found that has more coverage of the channel layer than the service layer, so I like it.
• Dare writes about Web3S, Windows Live's general purpose REST protocol. Apparently, WL started with Atom Publishing Protocol, but found that it didn't meet their needs around hierarchy and granular updates. David Ing says it's "not that similar" to my concept of REST, but I going to read the spec before I comment.
• Scott Hanselman writes about how he learned to program and some thoughts about teaching his son. Patrick has recently started expressing interest in programming (he want's to do what Daddy does). At four, I'm thinking I'll start him on Scratch (though ToonTalk looks interesting). As he gets older, I was thinking about Squeak, though I'm a smalltalk noob. I really like Scott's idea of creating a connection to the physical world via something like Mindstorms. Patrick loves Lego almost as much as his dad, so that would be cool.

Morning Coffee 90 - REST Response Roundup

Last week, I asked a REST Question: is it still REST if you don't use HTTP? My interest in durable messaging is well documented, so I want is to see a RESTful approach combined with a durable messaging. We all know my durable messaging tool of choice, though I'd trade SSB in a second for something that provided durable duplex messaging in a standard way.

Anyway, there were some fairly interesting responses that I wanted to highlight.

Probably most interesting to the discussion at hand was John Heintz' comment pointing out the existence of "Waka",  a new transfer protocol to replace HTTP from Roy Fielding. The fact that Dr. REST is working on a new protocol that's designed to be more RESTful than HTTP should put to bed any REST "is and only is" HTTP arguments.

Erik Johnson agrees that you can separate REST and HTTP, but he thinks I ought to call it something else. He suggests "resource-oriented" - have we created a new TLA here? Are you down with ROA, ROAD, ROD, ROP and all the other acronyms we could spawn?

Nick Malik breaks out the IFaP acronym - Identifier, Format, Protocol - and points out "Each of the successful Internet standards, from HTTP to SMTP, has an IFaP at the heart of it." But does anyone think SMTP is RESTful? I don't. I think standardization of IFaP's is on par in importance with RESTfulness, but they're orthogonal. That is to say I think Nick's wrong - I'm guessing we'll go a few rounds on this when he gets back from Nashville - or should I say, if he gets back? :)

Arnon Rotem-Gal-Oz has been wondering about REST without HTTP the same way I have, but he doesn't really go into detail as to why. I want durable messaging, Arnon mentions something about topic hierarchies. Couldn't you do that with HTTP, Arnon? He also points out a new DDJ article on REST. It's good, if high-level overview-y.

Pat Helland writes that Every Noun Can be Verbed. It's more related to CRUD is CRAP than REST == HTTP, but it's well worth the read. His point about using filling out a form being CRUD, but then handing the form over to someone else being an invocation of behavior is fairly eye-opening. As long as you "interpret the durn' things with the correct semantics", it doesn't really matter if they're nouns or verbs.

Last but not least, Ted Neward and Adrian Trenaman discuss SOAP vs. POX over on The Server Side. They focus too much on SOAP encoding (isn't that dead yet?), but near the end Ted points out: "Problem is, REST assumes that you want to carry all of the state in the payload itself, and for a modern enterprise system, or, hell, even for a game, that’s not always a safe assumption." Doesn't address my questions about using REST without HTTP, but a very good point nonetheless.

A REST Question

Since there appears to be at least a handful of RESTifarians among my readership, I'm just going to throw this half-formed thought / almost question out there. Maybe it's a FAQ, in which case I'd appreciate a pointer in the right direction.

My observations about REST are:

1. REST is a an "architectural style for distributed hypermedia systems".
2. REST "has been used to guide the design and development" of HTTP and URI.
3. Therefore REST as an architectural style is independent of HTTP and URI.
4. Yet, I get the feeling that the REST community would consider a solution that uses the REST architectural style but not HTTP and/or URI as "not RESTful".

Am I wrong in observation #4 above? If you're addressing resources by resource identifiers [aka URIs] but transferring those resource representations over a durable duplex protocol [aka not HTTP], are you still RESTing?

(Note, such a RESTful durable duplex protocol doesn't exist to my knowledge, though I would be very happy to be wrong about that. SSB does durable duplex, but it doesn't support URI style resource addressing. Granted, if I was going to build a durable duplex RESTful protocol, I would build on SSB - much the same way that HTTP builds on TCP. Though I am a huge fan of SSB, I'm specifically not suggesting that SSB is RESTful.)

Morning Coffee 87

FYI, I'm at TechEd all next week. Given that WiFi access at conferences usually blows, I'm not planning on regular morning coffee posts. I've asked Dale again to keep the lights on around here and he's graciously said yes. Since I'm not on vacation, I'll be lurking around as well, but I'll be in an out. See you in Orlando!

• Jeff Atwood proclaims that developers are their own worst enemy, because they write too much code. Add in a pinch of "not invented here" syndrome and I think you've got it. This is one of the reasons why people think Ruby is the tits.
• Scott Hanselman has already taken advantage of the new WLWriter provider customization API.
• Erik Johnson writes about the thunderous REST bandwagon. He doesn't explicitly say it, but my take is that he thinks this all ends up in some middle ground between REST and WS-*. I hear that, at least, if that's what he's saying. I'm not sold on "HTTP is all you need" - I need durability and async messaging, but I don't see how to get there from here with just HTTP.

Morning Coffee 86

• Google announces Gears, a browser plugin for taking your web application offline. Developer docs are also available. TechMeme has lots more, but obviously this is yet another significant bow shot in the upcoming unified client platform war. By my count, there are four horses in this race: Microsoft with .NET and Silverlight, Adobe with Flash and Apollo, Google with AJAX and Gears and Sun with Java and JavaFX. Did I miss anyone? (via Dare Obasanjo and Scott Hanselman)
• Alex James writes that REST is about intent and shows a pseudo-code sample posting multiple changes to a single endpoint as a way of demonstrating your intent that they be applied atomically. Andres Aguiar left a comment saying that Astoria does something similar. Personally, I like that model for transactions better than the transaction factory approach Jon Udell describes. But either way, you've moved beyond simple CRUD style services and into the world of protocol. Surfacing intent via protocol aligns with what Tim described as making the protocol explicit
• Windows Live posted new beta versions of Writer, Mail and Messenger. I've been on an internal build of the new Writer for a while and I've really been impressed. There's also a new Provider Customization API, so I can't wait to see what the DasBlog folks do with that.
• Scott Guthrie's LINQ series continues, this time covering how to build the LINQ to SQL data model. Looks like they used the DSL toolkit to build the LINQ to SQL data model designer, cool! 
• Martin Fowler digs into racc, a yacc-esque compiler compiler for Ruby. Looks interesting as a internal DSL example (better than the now-canonical rake example). But why is the sexy new language on the block using old school CFG's instead of new hotness PEG's?
• Speaking of Martin, he writes about the opportunity Ruby presents to Microsoft, building on Scott Hanselman's concerns that Microsoft is losing the Alpha Geeks. Sam Gentile also weighs in, suggesting that Microsoft is at the crossroads. Frankly, I don't work in evangelism anymore so I'm going pass these links along without comment except to say that Scott, Martin and Sam are all folks I have much much respect for.

Responses on REST

Here is a roundup of some responses to my REST discussion with David Ing. Mostly, I'm passing them along with minimal comment so I figured I'd group them into one post.

David started by leaving me a comment, but decided instead to post the comments on his blog. His big worry seems to be how well REST as CRUD will interop with REST as protocol. but in general I'm not sure that's a big worry. Bill de Hora's example below about WebDAV transactions activities seems to demonstrate at least some RESTifarians are cool with a protocol view of the world.

Erik Johnson (who's blog I mistakenly purged from my reader at some point, so needless to say I've resubscribed) writes that "real-world experience shows you rarely POST exactly what you GET" and that "even with the flawed cast of characters you see a lot of whining about...the pieces are there to build good systems that also make great constituents in anyone’s SOA." I agree 100%. He goes on to agree with me agreeing with Tim for people "not to limit their comprehension of REST around entities accessed via GET and PUT". Generally speaking, I agree with people when they agree with me, and this is no exception. Erik also has a REST post that predated my dustup with David(*) where he reaches some interesting conclusions about designing REST style systems.

Bill de Hora weighs in, pointing out that "value exchange != transaction". Given that I never suggested they were equal, I'm not sure what his point is (other than to be snarky). He also points to Subversion's use of WebDAV as an example of...well, I'm not sure since it seems to prove my point that a simple CRUD style approach doesn't cut it for many scenarios. According to the page Bill linked to:

"Subversion commits are modeled using the "activity" concept from DeltaV. An activity can be viewed as a transaction for a set of resources." [emphasis mine]

While I'm sure this sounds snarky as well, I really don't get what Bill is getting at here. It sounds like he took my disparaging of CRUD as a disparagement of REST, which is NOT how it was intended. This kind of layering of higher-level protocol concepts (like WebDAV's activity) is exactly what I was thinking of when I wrote "I can spurn CRUD and still embrace REST".

And though not a direct comment on my post, Omri Gazitt writes about REST vs. SOAP and the support for both in the next version of WCF. His main point is that "It seems like we'll continue to live in a world where multiple integration paradigms and protocol choices exist for applications". Of course, since he's with the WCF product group, regardless of your integration paradigm or protocol choice, WCF is the way to build it (at least on Windows). We'll have to wait a while to see if WCF is really "future-proof", but the ability to add significant changes in a .5 release is a fairly good sign.

(*) It's a very polite dustup, characterized mostly by agreement rather than disagreement. Which makes it, as dustups go, very boring.

This Isn't The Droid I'm Looking For

Since David Ing responded to my REST/CRUD/CRAP entry on his blog, I guess that means I respond to his response here, right?

Actually, this is going to be very short(*) because I mostly agree with what David wrote. For example:

If we say stuff like 'REST shall/must replace all foolish WS-* SOAP systems (insert Nelson-style HaHa here)' then we are just repeating the same lessons as before - One size won't fit all and there is no One Ring to Bind Them illusion in REST as well. If you have something complex that fits the WS-* style then maybe REST isn't the droid you are looking for.

Of course, this begs the question "what is complex?" David described complex as "long running transaction updating multiple distributed stores". Personally, I'd replace "stores" with "systems" and I hate term "long running transaction" so I would have written "long running operation" or "business process", but grammatical nitpicking aside that certainly seems like a fair description. Certainly, most of the scenarios I'm looking at in my day job could be described as being long running and updating multiple systems. David says later that he thinks "a lot of apps don’t need to be modeled that way", but I'm not aware of the alternatives so I'll let him expand on this on his own blog. I'll try to get to writing about my thoughts around protocol state next week.

I did disagree with this one point:

We may say REST is really about a protocol state and not CRUD, but unless the rest of the world gravitates to that view then I'm afraid it just won't be. If, say, through some ongoing groundswell of common usage, people start modelling entities as dereferenceable URI's and using POST to do Create and Updates, then REST will be about CRUD by default. This is all very unacademic and unjust, but thems the breaks peachy. The lesson from SOAP is not to fight it by trying to re-educate the masses after they get a perception in their head.

I think it doesn't really matter how the rest of the world gravitates, it only matters how you as the the service provider choose to expose your service. If you're doing something simple like ATOM publishing, you can probably get away with REST as CRUD. (Would that be hi or lo REST?) If you're doing something more complex, either in terms of being long running or needing multiple updates in one atomic operation like Tim's airline example, you'll probably need to gravitate towards REST as protocol state. But can't the two models can co-exist nicely, even in the same app? No re-education required.

UPDATE - From Fielding's Dissertation: "REST relies instead on the author choosing a resource identifier that best fits the nature of the concept being identified". My point exactly.

(*) Wow, this post wasn't that short at all. Can you imagine how long this would have been if I had mostly disagreed with David?

REST is neither CRUD nor CRAP

In the wake of my praise for CRUD is CRAP, David Ing asked “how do you reconcile something like REST (astoria etc) with being CRUD-adverse? Is there a happy place where the two can go for coffee?” Sure there is: I hear Tim Ewald’s XML Nation serves great coffee and the scones are pretty good. (*)

Seriously, the key observation that Tim recently made is that REST != CRUD. Sure, it can be used that way and for simple scenarios it works fine. (I’ll define “simple scenarios” in a second.) But I don’t believe CRUD style REST works in the large. Tim said you can’t build with just CRUD because it’s “to simplistic to be useful”. I’ll go even more fundamental, using REST for CRUD means having giving up transactions entirely. I've already accepted that building loosely coupled services means giving up distributed transactions. But the idea of giving up transactions entirely is just crazy talk.

So when I said "simple scenarios" above, I meant "scenarios that don't need transactions". (I take it as a given that RESTifarians aren't hot for WS-AtomicTransaction.) ATOM Publishing is a simple scenario because the web resource authoring scenario doesn’t need transactions to protect updates to multiple resources at a time. If it did, I don't believe the REST as CRUD approach they use would work.

As you might guess then, I’m not a fan of Astoria. I believe the sweet spot for so called “data services” will be read only (because they don't need transactions, natch). I'm sure there are some read/write scenarios Astoria will be useful for, but I think they will be limited - at least within the enterprise.

If REST != CRUD, then what is it? Let's go back to Tim's post:

Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I'm not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressable by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you're moving to, making that your new state. A state's representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app's protocol can't work that way too. (The ATOM Publishing protocol is the canonical example, though its easy to think that its about entities, not a state machine.)

While I disagree with Tim's disagreement of ATOM (i.e. I believe APP is about entities, but it works because it doesn't need transactions), I agree 100% that REST is about protocol state. Tim lays this very clear in his airline reservation sample. Thus, I can spurn CRUD and still embrace REST if I want to.

Further, Tim's points on the opaque nature of RPC style interactions (which web services appear to have fallen into despite the best of intentions) are spot on. If you're doing simple request/response services, the protocol state is trivial, so that works fine. However, in the scenarios I face, long running services are the norm and managing the protocol state is critical. I've got some ideas on how to do that, but that's a future blog post.

(*) Actually, I have no idea if Tim even likes coffee or scones. FYI, DevHawk Nation would not feature great coffee or pretty good scones. We would, however, have Arrogant Bastard Ale on tap.