The SOA Manifesto – Pointlessness Manifested

You know what the Agile Manifesto doesn’t have? Video of a “very formal ceremony” announcing said manifesto. Instead, we just have artistically rendered picture of what sure looks like Martin Fowler pointing at a white board while some of the other original signatories look on. Sure, it’s a cool picture, but wouldn’t it have been much cooler if they had captured that moment on video instead? Especially if it was video of them all standing around looking vaguely uncomfortable while photographers took their picture and someone gravely read the manifesto to give it artificially inflated importance.

I only watched ten seconds of the SOA Manifesto announcement video before I realized there’s nothing to see here, move along, just a bunch of navel gazing from the usual SOA suspects.

Seriously, if you having a big announcement about how cool, earth shattering, significant or, hell, even interesting your manifesto is, then it’s not any of those things. It’s a waste of my time.

Then I noticed that my previous manager and personal friend John deVadoss is one of the signatories. I have metric tons of respect for John, so I gave the SOA Manifesto a second chance.

It lost me at the second sentence.

Service orientation is a paradigm that frames what you do. Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation. We have been applying service orientation to help organizations consistently deliver sustainable business value, with increased agility and cost effectiveness, in line with changing business needs.

Are you frakking kidding me? “Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation.” Who the hell came up with that? I didn’t realize the International SOA Symposium had a Department of Redundancy Department.

When you define SOA in terms of SO, then you can’t possibly score well on the practicality quotient.

The rest of the manifesto isn’t much better. What made the Agile manifesto great IMO was that it ran counter to “common” beliefs like “changing requirements late in the process is bad” and “shipping software cycles should take years”. Sure, we all realize how right those Agile manifesto guys were now, but at the time it was the next best thing to heresy for many organizations. Those guys were agents of change. And I mean real change, not “this will help me sell books” change.

SOA manifesto on the other hand is basically repackaged common sense. Stuff like “Recognize that SOA ultimately demands change on many levels” and “The scope of SOA adoption can vary. Keep efforts manageable and within meaningful boundaries” Any help figuring out what that change is or what the scope should be? Nope. Thanks for the advice guys, but I had your so called “Guiding Principles” figured out long ago.

But hey, I’m sure the manifesto will help Thomas Erl et. al. sell more SOA books. So, I guess from that perspective it’s mission accomplished.

Nick's Flawed Vision of a Shared Integration Model

Of all the things you might say about Nick Malik, "thinks small" is not one of them. He takes on a significant percentage of the .NET developer community over the definition of Mort. He wants to get IT out of the applications business. He invents his own architecture TLA: SDA (aka Solution Domain Architecture). He's a man on a mission, no doubt. And for the most part, I'm with him 110% on his ideas.

However, when he starts going on about a shared global integration model, I start to wonder if he has both hands on the steering wheel, as it were.

Nick's been talking about this for over a year. As he points out, SaaS integration layer is the new vendor lock-in. One of the attractions of SaaS is that you could - theoretically, anyway - switch SaaS application providers easily which would drive said SaaS companies to constantly innovate. However, if the integration layers aren't compatible, the cost to switch goes up dramatically, leaving the customer locked-in to whatever SaaS company they initially bet on - even if that bet turns out to be bad.

OK, I'm with him so far. Not exactly breaking news here - we've seen the same integration issues inside the enterprise for decades. SaaS adds new wrinkles - if your ERP vendor goes belly-up, they can't take your data with them or worse sell it to your competition - but otherwise it sounds like the same old story to me.

However, where Nick loses me is when he recommends this solution:

"To overcome this conflict, it is imperative that we begin, now, to embark on a new approach. We need a single canonical mechanism for all enterprise app modules to integrate with each other. If done correctly, each enterprise will be able to pick and choose modules from different vendors and the integration will be smooth and relatively painless."

Yeah, and if a frog had wings, it wouldn't bump its ass when it hopped.[1] There are so many things wrong with this approach, I'm not sure I can get them all into a single web post.

First off, it won't, in fact, be done correctly - at least, not the first time. I realize everyone knocks MSFT for never getting an application right before version 3.0, but I believe it's actually systemic to the industry. Whatever you think you know about the problem to be solved, it's at best woefully incomplete and at worst wrong on all counts. So getting it right the first time is simply not possible. Getting it right the second time is very unlikely. It isn't until the third time that you really start to get a handle on the problem you're really trying to solve. Getting an effort like this off the ground in the first place would be a Herculean task. Keeping it together thru a couple of bad spec revisions would be impossible.

Meanwhile, the vendors aren't going to be waiting around twiddling their thumbs waiting for the specs to be done. We've seen efforts to unify multiple completing vendors around a single interoperable specification. By and large, those efforts (UNIX, CORBA, Java) have been failures. The technologies themselves haven't been failures, but the idea that there was going to be "relatively painless" portability or interoperability among different vendors never really materialized. If it didn't work for UNIX, CORBA or Java, what makes Nick think it will work for the significantly more complex concept of a shared global integration model? Not only more complex in terms of spec density, but the mind-boggling number of vendors in this space.

Nick is worried that either "we do this as a community or one vendor will do it and force it on the rest of us." But if you look at how specifications evolve, retroactive realization of defacto standards is the way the best standards get created. For example, I could argue that TCP was forced on us by the US Military, but I don't hear anyone complaining. I realize there's a big difference between having a vendor force a spec down our throat vs. a single big customer, but either way it's not designed by committee. Besides, if we do see get an enterprise integration standard forced on us, I don't believe it will be the vendors doing the forcing. If I were a betting man, I'd bet on Wal-Mart. Business leverage trumps IT leverage and Wal-Mart has more business leverage than anyone in this space these days.

BTW, would design-by-committee be an extreme example of BDUF? Do we really want to develop a this critical integration model using the same process that produced the XSD spec?

Finally, Nick thinks that this model will improve innovation, where I think it will have the exact opposite effect. Once you lay a standard in place, the way you innovate is to build proprietary extensions on top of that standard. However, by definition, these extensions aren't going to be interoperable. If someone has a good idea, others will copy it and eventually it will become a defacto standard.

A recent example of the process of defacto standardization is XMLHttpRequest. Microsoft created it in 1999 for IE 5, Mozilla copied it for their browser a couple of years later, followed by the other major browser vendors. Google innovated with it, Jesse James Garrett coined the term AJAX, everyone else started doing it and then finally - nearly a decade later and still counting - a standards body is getting around to putting their stamp of approval on it.

However, if you're worried about painless integration and not having something forced on you by some vendor, then you're not going to embrace these innovations - which means, you won't embrace any innovation. Well, there may be some innovation in the systems themselves that doesn't affect the interface, but once that interface is cast in stone, the amount of innovation will go way down. How do vendors differentiate themselves? There's only two ways: price and innovation. Take away innovation with standardization, and you're left with a race to the rock bottom price with no incentive to actually improve the products.

I get where Nick is going with this. He looks around our enterprise and sees duplication of effort and massive resources being spent on hooking shit together. It sure would be nice to spend those resources on something more useful to the bottom line. But standardizing - or worse legislating - the problem out of existence isn't going to work. What will? IMO, applying Nick's ideas of Free Code to interop code. If we're going to get IT out of the app business, can't we get out of the integration business at the same time?

[1] It's exceedingly rare that you get to quote Wayne's World or Raising Arizona in a discussion about Enterprise Architecture, much less both at the same time. Savor it.

Morning Coffee 134

• Bill de Hora responds to a few of my Durable and RESTful ideas. He points out that relying on a client-generated ID can be troublesome, and recommends using multiple identifiers - one created by the sender, one by the receiver and one representing the message exchange itself. However, the sender ID is vulnerable to client bugs & tampering as Bill points out, and neither the receiver ID nor the exchange ID can be used to determine if a given message is a duplicate. If you don't trust the sender, is it even possible to determine if a given message is a duplicate?
• Pablo Castro confirms that there are "practical limits" to what ADO.NET Data Services can do with respect to idempotence. Nothing in his post was surprising, though I hope it will be more explicitly called out in the final docs. Developers used to the comforting protection of a transaction may be in for a rude awakening.
• Dare Obasanjo has a great post comparing the new features in C# 3.0 to dynamic languages like IronPython. I believe many of the productivity aspects of dynamic languages have little to do with being dynamic.
• Pat Helland noodles on durability and messaging, two topics near and dear to my heart (probably from working with him for a couple of years). I'm not sure where he's going with this - his conclusion that "Basically, big, complex, and distributed system are big, complex, and distributed" isn't exactly ground-breaking. But his point that "durable" isn't a binary concept is worth more consideration. Also, his description of IMS only looking at the effects of a committed transaction is very similar to how web sites work, though obviously HTTP isn't durable so you can't make event horizon optimizations like IMS did.
• Tangentially related, Werner Vogels discusses the idea of eventually consistent distributed databases. Today, that's a problem mostly only Internet-scale sites like Amazon deal with. In the near future of continued data explosion + manycore, we'll all have to deal with it.
• Nick Malik argues that categorizing enterprise applications by lifecycle is much less useful than categorization based on organizational impact. He might also need a new chair.
• Jesus Rodriguez digs into one of SSB's new features in SQL 2008: conversation priorities.
• Arnon Rotem-Gal-Oz and Sam Gentile are mixing it up over the definition of SOA. Sam thinks SOA has to include business drivers and Arnon doesn't. I'm with Sam on this, defining "SOA" independently from "Applying SOA" seems pointless. Then again, rigorously defining SOA - much less arguing about said definition - seems like a waste of time in the first place IMHO.
• Wow, this guy Zed is mad at the Ruby community.
• Andrew Baron has 8 Reasons Why The TV Studios Will Die. Personally, I think reason #2 - Expendable Middle-Person - is the most important. If content producers can reach consumers directly, what value-add will the networks provide? (via United Hollywood)

Afternoon Coffee 123

• Morning Coffee is late this morning because we went for our Christmas portrait this morning and it took forever. The pictures turned out great though.
• Nick Malik finishes up his series on business operation models by covering the diversification model. Also, Nick's points about the synergy between a diversified model and the coordinated model are spot on. I happen to be a big fan of those models (aka the models with low standardization) which probably drives some of the  more my "unique" perspectives on SOA.
• Scott Guthrie starts out a new series and future technology, this time it's ASP.NET MVC Framework that gets the series treatment. The first entry in the series is a general overview. I wonder why there's no cool code name for the MVC framework? Whatever it's named, I like the auto routing and action rules - it seems very Rails-inspired.
• Over the weekend, Don Box points out that the REST authentication story "blows chunks". I've recently given up on the reliable part of the original "Secure, Reliable, Transacted Web Services" vision - and I never believed the transacted part. Security, on the other hand, is the one part of that original vision that has worked out IMO. My experience with the WS-* security stack has been pretty good, though Dare Obasanjo thinks that OpenID and OAuth are the final nail in the WS-* coffin.
• Speaking of Dare, he goes on to say WS-* is to REST as Theory is to Practice. He makes the point that "The only times I encounter someone with good things to say about WS-* is if it is their job to pimp these technologies or they have already “invested” in WS-* and want to defend that investment." I gave up pimping evangelizing technology a while back and I don't want to be in the position of defending a bad investment, so I'm spending lots of time looking at REST.
• Jesus Rodriguez takes a look at the Managed Services Engine and comes away excited. Jesus is a self-described "strong believer" in SOA governance. I'm a self-described strong disbeliever in SOA governance, so MSE sounds like more of the Worst of Both Worlds to me.
• A little light reading: I pulled Applied Cryptography and A New Kind of Science out of my garage last weekend. Plus my copies of RESTful Web Services and Programming Erlang just arrived yesterday.

The Importance of Idempotence

Every organization has some operations or processes that have to happen Exactly Once. Your employer needs to make sure they issue your paycheck exactly once. Your bank needs to make sure that paycheck is deposited in your account exactly once. Exactly Once isn't something that just "traditional" enterprises like banks care about. Google needs to make sure your AdSense check is issued exactly once. Amazon needs to make sure your credit card is charged exactly once. Especially when there's money involved, the company wants to make sure it gets handled correctly - Exactly Once.

In application (aka siloed) development, transactions are often used to ensure stuff happens Exactly Once, to good effect. But how do we guarantee Exactly Once now that we're connecting systems together? Given how well transactions work inside applications, it's not surprising that early attempts to guarantee Exactly Once between systems relied on distributed transactions, this time to not-so-good effect. Pat Helland summarized the problems with distributed transactions this way:

"The two-phase commit protocol will ensure perfect consistency given infinite time.  I say that because it will wait and wait and wait until the transaction is resolved and then provide perfect consistency.   Of course, while partitioned and waiting, arbitrary swaths of the application's database may be locked up rendering the application unusable.  For this reason, I've frequently referred to the two phase commit protocol as the "Anti-Availability Protocol". "
Pat Helland, SOA and Newton's Universe

So now we're faced with a dilemma. Transactions are, for all practical purposes, unusable to ensure Exactly Once processing between connected systems. And yet, the business requirement to ensure Exactly Once hasn't gone away. We need another way.

The first fallacy of distributed computing is that the network is reliable. It's usually works, but usually isn't a guarantee. If I send a message to a remote system but don't get an acknowledgement, which got lost: the original message or the ack? There's no way to know, so I have to send the message again. But if I send it again and it's the ack that got lost, then the target system will receive the message multiple times.

Since the network is not reliable, there's no way to guarantee that a message will be delivered exactly once. The best we can go is ensure a message will be delivered at least once. However, that implies the target system will receive some messages multiple times. If we need to ensure Exactly Once, we need to make sure the target system won't duplicate the work if it receives duplicate messages. In other words, we need the target system to be idempotent. 

"In computer science, the term idempotent is used to describe method or subroutine calls which can safely be called multiple times, as invoking the procedure a single time or multiple times results in the system maintaining the same state i.e. after the method call all variables have the same value as they did before.

Example: Looking up some customer's name and address are typically idempotent, since the system will not change state based on this. However, placing an order for a car for the customer is not, since running the method/call several times will lead to several orders being placed, and therefore the state of the system being changed to reflect this."
Wikipedia, Idempotence (Computer Science)

Or more succinctly:

"Idempotent Means It’s OK to Arrive Multiple Times"
Pat Helland (again)

I can't overstate the importance of designing your cross-system communication to be idempotent. If you care about ensuring Exactly Once, each step of your process has to be either transactional or idempotent, or you'll be screwed. It's interesting to note that you have to be transactional *OR* idempotent, but not both. You can chain together multiple steps in long business process, across multiple disparate systems, but as long as each step is either transactional or idempotent, you can guarantee Exactly Once across the entire process. In other words:

Transactional/Exactly Once == Idempotent/At Least Once

This implies that you can substitute an idempotent operation for a transactional operation, and still ensure Exactly Once.

Let's look at an example. Typically you ensure Exactly Once processing with MSMQ by receiving messages within the scope of a transaction along with whatever other work you're doing. But what if you can't use a transactional receive, say because it's a remote queue? What would an idempotent equivalent for transactional receive look like?

How about:

1. Peek a message from the remote queue
2. Insert the message into the target system database, using the unique MSMQ Message ID as the primary key
3. Remove the message from the queue by ID

Each of those steps is idempotent. Peek is a read, which is naturally idempotent. Inserting the message into the database is idempotent, since we use the message ID as the primary key. As long as that ID is unique, we can never insert it into the database more than once. Finally, removing a message based on it's unique ID is also naturally idempotent. Once the message is in the target system database, we can use traditional transactions to ensure it gets processed Exactly Once.

So we took a single transactional operation and turned it into a series of idempotent steps. Both ensure each message is processed Exactly Once. Given the choice, I'd rather write the transactional operation - it's much less code since we're we can use existing infrastructure - aka the distributed transaction coordinator. But if the transactional infrastructure isn't available, I'd rather write multiple idempotent steps and ensure Exactly Once rather than risk losing or duplicating messages.

I've got more on this topic, but in the meantime think about this: How do you think durable messaging infrastructure like MSMQ ensures exactly once delivery? You can use that pattern, even if you're not using durable messaging infrastructure.

Morning Coffee 122

• Sorry for the posting lag. Had a few technical difficulties around here. In the process of moving hosts, so expect more glitches.
• My talk at the p&p Summit on Monday went really well. At least, it felt good and the applause at the end felt genuine. I recorded the audio on my laptop, so I'll be posting a Silverlight version as soon as I figure out how to adjust the levels so their somewhat consistent. Paraesthesia and #2872 have reactions.
• Speaking of the p&p Summit, Scott Hanselman posted his ASP.NET MVC demo from his talk. Said ASP.NET MVC bits aren't available yet, so you can't, you know, run the demo for yourself. But at least you can review what the ASP.NET MVC code will look like.
• I stopped by the SOA/BPM conference last week and saw Jon, Sam and Jesus among others. Spent quite a bit of time talking to Sam and his Neudesic colleagues about this "physically distributed/logically centralized" approach that I think is hogwash. It sounds to me like Neudesic approach is really federated not centralized, though I'm not sure David Pallmann would agree. Federated makes much more sense to me than centralized.
• Nick Malik continues his series on SOA Business Operations Model. I especially like his point that this isn't a series of choices, you need to "look at your company and try to understand which model the business has selected. "
• The first CTP of PowerShell 2.0 is out! Check out what's new on the PowerShell team blog and Jeffrey Snover's TechEd Presentation. (via Sam Gentile)
• Soma announced updates to VC++ coming next year, including TR1 support and a "major" MFC upgrade to support creating native apps that look like Office, IE or VS. I get supporting TR1, but the idea that people are clamoring for MFC updates is kinda surprising. Many years ago when I first came to MSFT, a friend asked "But don't you hate Microsoft?" to which I responded "No, I just hate MFC". Obviously, not everyone agrees with that sentiment.
• Steve Vinoski thinks there's no hope for IT. Funny, I keep agreeing with Steve's overall point but disagreeing with his reasoning. I still don't buy the serendipity argument. I like compiled languages. And I think he's overstating the amount of "real, useful guidance" for REST floating around. Basically, there's "the book".
• In widely reported news, Windows Live launched their next generation services. Don't bother with the press release, just go to the new WL home page.
• Speaking of WL, Dare Obasanjo points to the Live Data Interactive SDK page where you can experiment with the WL Contacts REST API. It gives you a good sense of how the Web3S protocol works. Pretty well, IMO. However, how come WL Contacts Schema doesn't include some type of update timestamp for sync purposes? If you wanted to build say a Outlook <--> WL Contacts sync engine, you'd have to download the entire address book and grovel thru it for changes every sync.
• Speaking of Web3S, I'd love to see some info on how one might implement a service using Web3S. Yaron Goland positions Web3S as an alternative to APP that WL developed because they "couldn't make APP work in any sane way for our scenarios". I'm sure other folks have similar scenarios.

Morning Coffee 121

• My daughter had her tonsils & adenoids out on yesterday. It was a routine procedure and it went by-the-numbers, but any parent will tell you it's hard to see your kid in a hospital bed.
• Given the previous bullet, I'm not at the SOA/BPM conference for the big announcement. Don't worry, there's lots of other folks covering the news.
• It was a crappy sports weekend in the Pierson house. Va Tech snatched defeat from the jaws of victory, Southern Cal never led at Oregon, the Capitals lost twice, and the Redskins got blown out by the Pats. At least the Caps won big yesterday in Toronto.
• Speaking of the Capitals, Peter Bondra retired Monday. I still think it's a travesty that he didn't spend his whole career in DC, but I've made my peace with it.
• Nick Malik has a great series on business operations models and how they apply to SOA. Regular readers should be unsurprised that I favor low standardization, though I can see the value of high integration. That makes the Coordinated Operating Model my fav, though I can see the benefit of the Diversified Model as well. I can't wait to read what Nick has to say on changing models.
• Speaking of Nick, I'm doing a roundtable with him on "Making SOA Work in the Enterprise" @ the Strategic Architect Forum. Should be fun. Sorry for the lack of linkage on this, but it's an invite-only event.
• Jezz Santos has a new series of white papers on building software factories. First up "Packaging with Visual Studio 2005"
• Aaron Skonnard has a new whitepaper on using the WCF LOB Adapter SDK with BTS 2006 R2. I've been building one of these things recently, so I'm looking forward to checking that out. (via Sam Gentile)
• Tim Ewald looks at Resource Oriented Architecture (when did ROA become a TLA?) and wonders "what if your problem domain is more focused on processes than data?" I wonder that all the time. (via Jesus Rodriguez)
• It's not just durable messaging - Libor Soucek also disagrees with my opinions on centralized control. I agree 100% with Libor that centralized management would make operation's lives "much, MUCH easier" as he puts it. However, that doesn't make it feasible at any significant scale. Furthermore, I wouldn't describe an approach that requires that "all services adopt [the] same common management interface" as "pragmatic". Frankly, just the opposite.

The Worst of Both Worlds

David Pallmann of Neudesic responded to my comment that "Physically distributed but logically centralized" didn't make any sense to me at all:

What exactly does this mean? To some this may sound like a contradiction.

This simply means that a bus is physically more like the point-to-point architecture (spread out, no hub) but functionally more like the hub-and-spoke architecture (pub-sub messaging, centralized configuration and activity tracking, easy change management).

Unfortunately, I wasn't confused about the seeming contradictory nature of these concepts. In other words, I understand the "what" and "how" of David's physically distributed/logically centralized approach.

I don't understand the "why". As in, "why would you want to do this?" or "why do you think this would work at any significant scale?".

If we check out Neudesic's page on their ESB product (which David pointed me to) we find the following blurb:

Centralized Management
The distributed nature of service oriented programming can create a management nightmare. Neuron·ESB supports this distributed architecture while simultaneously centralizing monitoring and configuration.

SOA's "distributed nature" is it's primary strength. SOA's not primarily about standards or ease-of-connectivity - though those obviously play a role. It's about enabling decentralized decision making. Since you can't be both centralized and decentralized, enforcing centralized management basically negates SOA's primary strength. This seems like the worst of both worlds to me. All the hassle of distributed decision making combined with all the hassle of centralized management.

Yes, decentralized decision making can create a management nightmare. Personally, a management nightmare is much more attractive anything centralized approaches have ever delivered in the IT industry.

Dare Obasanjo recently wrote "If You Fight the Web, You Will Lose". He was talking about the Web as a Platform, but it's good general advice. Can you imagine applying the marketing blurb above to the Internet at large?

Centralized Management
The distributed nature of service oriented programming the Internet can create a management nightmare. Neuron·ESB supports this distributed architecture while simultaneously centralizing monitoring and configuration.

If the Internet can somehow get by without centralized management, why can't you?

Throwing Gasoline on the Fire

Steve Vinoski has raised a bit of a flame war by admitting he has lost the ESB religion. Given that I've never been a fan of ESB's anyway, there's a lot there that I agree with. In particular I liked the description of "magical framework" middleware, blaming enterprise architects for driving ESB's as the "single integration architecture" even though a single *anything* in the enterprise is untenable and his point that flexibility means you don't do any one thing particularly well.

However, Steve goes on to bash compiled languages and WS-* while suggesting the One True Integration Strategy™ is REST + <insert your favorite dynamic language here>, then acts surprised that the conversation denigrates into "us vs. them". When you start by saying that compiled language proponents "natter on pointlessly", I think you lose your right to later lament the depreciating level of conversation .

All programming languages provide their own unique model of the execution environment.  Dynamic languages have a very different model than compiled languages. Arguing that this or that model is better for everyone, everywhere, in all circumstances seems unbelievably naive and arrogant at the same time.

On the other hand, I do agree with Steve's point that most developers only know a single programming language, to their detriment. One language developers often miss a better solution because their language of choice doesn't provide the right semantics to solve the problem at hand. Developers could do a lot worse than learn a new language. And I don't mean a C# developer should learn VB.

The most pressing example of picking the right language for the right problem today is multi-threading. Most languages - including dynamic languages - have shitty concurrency semantics. If you're building an app to take advantage of many-core processing, "mainstream" apps like C#, Java and Ruby won't help you much. But we're starting to see languages with native concurrency semantics like Erlang. Erlang is dynamically typed, but that's not what makes it interesting. It's interesting because of it's native primitives for spawning tasks. I don't see why you couldn't add similar primitives for task spawning to a compiled functional language like F#.

As for REST vs. SOAP/WS-*, I thought it was interesting that Steve provided no rationale whatsoever for why you should avoid them. The more I listen to this pissing match debate, the more I think the various proponents are arguing over unimportant syntactical details when the semantics are basically the same. SOAP is just a way to add metadata to an XML message, much as HTTP headers are. WS-* provides a set of optional message-level capabilities for handling cross-cutting concerns like security. Past that, are the models really that different? Nope.

For system integration scenarios like Steve is talking about, I'm not sure how important any of the WS-* capabilities are. Security? I can get that at the transport layer (aka HTTPS). Reliable Messaging? If I do request/response (which REST excels at), I don't need RM. Transactions? Are you kidding me? Frankly, the only capability you really need in this scenario is idempotence, and neither REST or SOAP provides any standard mechanism to achieve that. (more on that in a later post)

I understand that some vendors are taking the WS-* specs and building out huge centralized infrastructure products and calling them ESBs. I think Steve is primarily raging against that, and on that point I agree 100%. But Steve sounds like he's traded one religion for another - "Born Again REST". For me, picking the right tool for the job implies much less fanaticism than Steve displays in his recent posts.

The One Business Case for Integration

Nick Malik lays out what he thinks are the four business cases for integration:

Assume we succeeded, and our systems are all optimally integrated.  What has changed? 

1. We have better business intelligence.  We have better understanding of our customers, our partners, our products, and our business.  And from that understanding, we make better decisions.  Those decisions are made in a federated manner using self-apparent information.
2. We have end-to-end business processes that cross multiple systems, multiple roles, multiple geographies, and multiple data stores, all aware of and supporting the needs of the customer.
3. We have end-to-end awareness of the metrics that drive both dissatisfaction and cost, and we can take that knowledge and apply it to making our business better.
4. We have a more efficient enterprise, more able to grow to a larger size, at an accelerated rate, and still respond with agility to changing business opportunities.

I put to you that, in fact, we only have one business case for integration: better business intelligence. The other reasons Nick lists are either redundant or not as important to the business - at least in the general case - as you might think.

First off, #3 from Nick's list sounds suspiciously like #1. If there's a difference between "better understanding driving better decisions" and "applying awareness of metrics to making our business better", I don't know what it is. We'll send one of them off to the Dept. of Redundancy Dept. and be done with it.

Second, I don't think the business cares that IT has multiple systems or multiple data stores. If the business could run on one big centralized system that could meet the needs of the customer (aka the ERP fantasy), they'd be fine with that. The fact that realities of modern enterprise IT require splitting up capabilities across many systems is an implementation detail that frankly isn't a concern of the business.

Besides, what's the business benefit here? News flash: the enterprise already has end-to-end business processes that cross multiple systems, multiple roles, blah blah blah. They're just not automated end-to-end. Does the business care that their not automated? Not a bit. Sure, they care about processes are slow, costly and error-prone, which manual processes tend to be. But it's those negative characteristics that the business cares about, not integration. Besides, making processes quick, cheap and error-free sounds a lot like making them efficient. In other words, more work for the Dept. of Redundancy Dept.

Finally, I don't think efficiency and agility is as important to the enterprise as Nick makes it out to be. I mean, the enterprise will say it cares about efficiency - especially in front of the stock holders. But when it comes to putting it's money where it's mouth is, the enterprise doesn't, more often than not. Think about how success is measured in the typical IT project. Is efficiency one of the criteria for judging success? Not really. Will your project stakeholders let you run over budget and ship a few months late, just to improve efficiency? Probably not, unless that efficiency gain is both demonstrable and dramatic.

Of course, there are certainly specific examples where a automation or efficiency business case for integration can be made. For example, if replacing a specific manual process with an automated one has a large and measurable ROI, the business will likely be interested in making that investment. If you have a certain process that you do over and over that's core to the business, the business will probably be interested in optimizing the frak out of it. For example, I would guess a delivery company like UPS or FedEx has spent a lot of time and money on optimizing their delivery processes.

But what it sounds like Nick's talking about here is making a general case for making all our systems "optimally integrated". Given that our current systems aren't, this would take significant time and money. Yet the tangible benefit to the business is at best nebulous. Nick thinks improved integration will allow the business to "respond with agility to changing business opportunities." He's probably right. But how do you quantify this agility? How much will we save in the future for what we're spending today? For the general case, the answer is "it depends". It's really hard to fund a project when it's projected ROI is "it depends" .

However, business intelligence is a no brainer for the enterprise to invest in. Giving decision makers better and more up-to-date information, that's a tangible benefit that the organization can quantify now. If you can quantify the value of a project, you've got the start of a budget. Of course, all that juicy data is smeared across a variety of systems, which means integration. Again, the enterprise doesn't really care about said multiple systems or integration, but they care about the outcome.

Nick recommends to SOA folks that "if you aren't already working with your BI team, pick up the phone. Their mature processes and practices are able to address many of your issues, and the natural synergy between BI and SOA can make them a strong ally in the fight for a better, faster, cheaper, and more intelligent enterprise." Good advice. Otherwise, selling integration to the business isn't much different than selling them SOA. In other words, don't sell it - just do it.

Morning Coffee 113

• I'm in Chicago today and tomorrow for a reunion of sorts. In my last job, I managed a group of external architects called the Microsoft Architecture Advisory Board (aka the MAAB). We discontinued the program a while back, but the core of the group found the program valuable enough they have continued to meet anyway. I found the MAAB meetings incredibly valuable and insightful, so I'm really excited to be invited to continue my involvement with the group.
• I picked up Bioshock Tuesday (Circuit City had it on sale) on my way to my bi-weekly campus excursion. My meetings were over around 2pm so I headed home early, expecting to surprise the kids. But Jules had decided to skip naps and go shopping with them. Her cell phone was dead, so I ended up at home with a couple of hours to myself and a brand new copy of Bioshock. Wow, is that a good game. Certainly deserving of the amazingly good reviews it's garnered.
• Speaking of reviews, this transparently biased review of Bioshock over at Sony Defense Farce Force is frakking hilarious. Somehow, I doubt their dubious review will stem the tidal wave of Bioshock's well-deserved hype. Can't wait to read their Halo 3 review.
• Pat Helland writes at length on master-master replication. I reformated it into PDF so I could read it - the large images were messing up the text flow on my system. As usual for Pat, there's gold in that thar post. His thoughts on DAGs of versions and vector clocks as identifiers are very exciting. However, I think he glosses over the importance of declarative merging. I would think programmatic merge would likely be non-deterministic across nodes. If so, wouldn't you end up with two documents with the same vector-clock identifier by different data?
• Joe McKendrick points to a few people who predict the term "service-oriented" will eventually be subsumed under the general heading of "architecture". Not to brag, but I made that exact same prediction almost three years ago.
• Erik Johnson thinks that SOA 2.0 centers on transformational patterns. The idea (I think) is that if systems "understand each other more deeply", then we can build a "smarter stack" and design apps via new constructs to promote agility and simplicity. Personally, I'm skeptical that we can define unambiguously system semantics except in the simplest scenarios, but Erik talks about using "graph transformation mathematics" to encode semantics. I don't know anything about graph transformation mathematics, but at least Erik has progressed beyond hand waving to describing the "what". Here's looking forward to the "how".
• New dad Clemens Vasters somehow finds time to implement an XML-RPC binding for WCF 3.5. I was encouraged that it didn't require any custom attributes or extensions at the programmer level. Of course, XML-RPC fits semantically into WCF's interface based service model, so it shouldn't be a huge surprise that it didn't require any custom extensions. But did it need WCF 3.5? Would this work if recompiled against the 3.0 assemblies?
• Phil Haack writes a long post on Duck Typing. VB9 originally supported duck typing - the feature was called Dynamic Interfaces - when it was first announced, but it was subsequently cut. I was really looking forward to that feature. Between it and XML Literals, VB9 was really stepping out of C#'s shadow. I guess it still is, even without dynamic interfaces.
• Since I've been doing some LINQ to XML work lately, I decided to go back and re-write my code in VB9 using XML literals. While XML literals are nice, I don't think they're a must have. First, LINQ to XML has a nice fluent interface, so the literals don't give you that much cleaner code (though you do avoid writing XElement and XAttribute over and over.) Second, I find VB9's template syntax (like ASP <%= expression %>) clunky to work with, especially in nested templates. Finally, I like the namespace support of XNames better. As far as I can tell, VB9 defines namespaces with xmlns attributes just like XML does. So I'm not dying for literal XML support in a future version of C#. How about you?

Morning Coffee 110

• Monday @ Gamefest, the XNA team announced XNA Game Studio 2.0. The two big new things are support for the entire VS product line (1.0 only works on VC# Express) and the addition of networking APIs. Let's Kill Dave has a good wrapup of the announcements from Gamefest Day One.
• Speaking of Xbox 360, I played thru the demos of Stranglehold and Bioshock. Two thumbs up on both. It's gonna be an expensive year for Xbox gamers.
• Mark Cuban noodles on taking your house public. "Why not create a market or exchange where homeowners can sell equity in their homes?" I've thought about this myself from time to time. However, Mark thinks making it happen would "probably take the country's biggest banks working together". I wonder if there's a more Web 2.0 social lending approach that would work better.
• Jeff Atwood calls virtualization as "the next great frontier for computer security". I agree 100%. But I don't think the action is going to be in "full-machine" virtualization like Virtual PC. Rather, it's going to be sandbox virtualization. Jeff mentions GreenBorder (now part of Google) but it's not the only solution. Some time ago, Microsoft acquired SoftGrid which uses sandbox virtualization for application deployment, but using SystemGuard for security sandboxing seems like a logical step.
• The WCF LOB Adapter SDK has released. Sonu Arora has the details. As part of the Integration team @ MSIT, I have a feeling we're going to become fairly familiar with this technology. (via Jesus Rodriguez).
• Speaking of Jesus, he thinks the six new SCA4SOA committees are "going to help". Why? Because inventing technology in committee has turned out so well in the past?
• John deVadoss cements BPM's fad du jour status by contrasting "big" BPM and "little" BPM. It's fairly obvious to me that big *anything* just doesn't work in the enterprise. But I worry that little *anything* doesn't work that well either. So how long until someone (probably Nick) starts arguing for "middle out" BPM?
• David Bressler wonders "What is it about registries that everyone thinks is a panacea for all things SOA?" Amen, Brother! Joe McKendrick claims it's required for governance, but then gets to what I think is the *real* reason for focus on registries: the "registry is a tangible offering" that vendors can sell. Just because it's productizable doesn't mean you need it.
• Hartmut Wilms responds to my retire the tenets post, but he seems to contradict himself. On the one hand, he suggests that "the four tenets just expressed, what “almost” everybody outside the MS world knew already". But then he goes on to dispute that the SO paradigm shift has even occurred! Hartmut, I'll grant you that WCF (among other similar stacks) are way too focused on "you write the classes, we'll handle the contracts and messages". On the other hand, if you don't provide a productive interface that most everyone can pick up and run with, the technology won't get adopted in the first place.

Retire the Tenets

John Heintz and I continue to be in mostly violent agreement. It's kinda like me saying "You da architect! Look at my massive scale EAI Mashup!" and having him respond "No, you da architect! The SOA tenets drive me bonkers!" Makes you wonder what would happen after a few beers. What's the architect version of Tastes Great, Less Filling? [1]

Speaking of the tenets, John gives them a good shredding:

Tenet 1: Boundaries are Explicit
(Sure, but isn't everything? Ok, so SQL based integration strategies don't fall into this category. How do I build a good boundary? What will version better? What has a lower barrier to mashup/integration?)

Tenet 2: Services are Autonomous
(Right. This is a great goal, but provides no guidance or boundaries to achieve it.)

Tenet 3: Services share schema and contract, not class
(So do all of my OO programs with interface and classes. What is different from OO design that makes SOA something else?)

Tenet 4: Service compatibility is based upon policy
(This is a good start: the types and scope of policy can shape an architecture. The policies are the constraints in a system. There not really defined though, just a statement that they should be there.)

Ah, I feel better getting that out.

As John points out, the four tenets aren't particularly useful as guidance. They're too high level (like Mt. Rainier high) to be really actionable. They're like knowing a pattern's name but not understanding how and when to use the actual pattern. However, I don't think the tenets were ever intended to be guidance. Instead, they were used to shift the conversation on how to build distributed applications just as Microsoft was introducing the new distributed application stack @ PDC03.

John's response to the first tenet makes it sound like having explicit boundaries is obvious. And today, maybe it is. But back in 2003, mainstream platforms typically used a distributed object approach to building distributed apps. Distributed objects were widely implemented and fairly well understood. You created an object like normal, but the underlying platform would create the actual object on a remote machine. You'd call functions on your local proxy and the platform would marshal the call across the network to the real object. The network hop would still be there, but the platform abstracted away the mechanics of making it. Examples of distributed object platforms include CORBA via IOR, Java RMI, COM via DCOM and .NET Remoting.

The (now well documented and understood) problem with this approach is that distributed objects can't be designed like other objects. For performance reasons, distributed objects have to have what Martin Fowler called a "coarse-grained interface", a design which sacrifices flexibility and extensibility in return for minimizing the number of cross-network calls. Because the network overhead can't be abstracted away, distributed objects are a very leaky abstraction.

So in 2003, Indigo folks came along and basically said "You know the distributed object paradigm? The one we've been shipping in our platform since 1996? Yeah, turns out we think that's the wrong approach." Go back and check out this interview with Don Box from early 2004. The interviewer asks Don if WCF will "declare the death of distributed objects". Don hems and haws at first, saying "that's probably too strong of a statement" but then later says that the "contract, protocol, messaging oriented style will win out" over distributed objects because of natural selection.

The tenets, IMHO, were really designed to help the Windows developer community wrap their heads around some of the implications of messaging and service orientation. These ideas weren't really new - the four tenets apply to EDI, which has been around for decades. But for a generation of Windows developers who had cut their teeth on DCOM, MTS and VB, it was a significant paradigm shift.

These days, with the tenets going on four years old, the conversation has shifted. Platform vendors are falling over themselves to ship service/messaging stacks like WCF and most developers are looking to these stacks for the next systems they build. Did the tenets do that? In part, I think. Mainstream adoption of RSS was probably the single biggest driver of this paradigm shift, but the tenets certainly helped. Either way, now that service orientation is mainstream, I would say that the tenets' job is done and it's time to retire them. Once you accept the service-oriented paradigm, what further guidance do the tenets provide? Not much, if any.

[1] Not that you would catch me drinking Miller Lite. Ever.

Morning Coffee

• Libor Soucek continues our conversation about durable messaging. We still don't agree, but he says he "fine" with durable messaging. He does go out of his way to differentiate between *enterprise* and *supporting* systems. But when you're building connected systems, does that differentiation still matter?
• After taking a few months off, John deVadoss is back at the blog. Check out his Big SOA/Little SOA post. I especially like his snowball analogy "How do you build a big snowball? You start with a small snowball.") though he's also on this "middle out" bandwagon. Do we really believe "middle out" works, or are we just saying it because we know top down and bottom up don't? And John: You're welcome!
• Anyone coming to the Microsoft SOA & Business Process Conference this fall? Maybe we can have a shindig / blogger dinner / unconference / something?
• Remus Rusanu writes about SSB's dynamic routing. One of the (many) cool things about SSB is that all the addressing is logical, not physical. Routing is what binds logical addresses to physical addresses, and it's extensible.
• Martin Fowler discusses the value of sticking to one language. I agree with his points about large frameworks being as difficult to learn as a new language. I've said for a long time "If you build a framework, build tools to make it easy to use your framework". Language is obviously a core example of a tool. Another interesting point Martin makes is the traditional "intimate relationship" between scripting languages and C, but that the rise of JVM & CLR makes them impossible to ignore. Does the need to play well in a managed environment hinder a C based language like Ruby when compared to a natively managed scripting language like Powershell? Finally, Martin's "jigger of 80 proof ugliness" quote made me laugh.
• Politics 2.0 Watch: EJ Dionne says that DailyKos is doing for Democrats what Rush Limbaugh did for Republicans almost twenty years ago: mobilization. Josh Marshall points out that "what's happening today is vastly more participatory and distributed...than anything happening back then."

Where Have All the SOA Mashups Gone?

John Heintz responded to my serendipitous reuse post. Nice to see I misunderstood his opinions about how easy RESTful systems are to integrate:

I didn't mean to imply that building RESTful system would lead to magical integration without any hard work. I can see how that came across in my post, and I guess I got the reaction I asked for ;)

I get the feeling that John would be a good guy to have a beer with.

John spends most of his post writing about the SOA in the Real World book. I've flipped thru it and I'm familiar with the model (it is my old team after all) but I haven't read it so I don't really want to comment about the book specifically. But there were two things John mentioned that I did want to comment on.

First, at the end of his post John writes:

Can some of the constraints of REST be applied to SOA? Absolutely. I think an asynchronous, message-passing architecture with a uniform interface would be astoundingly interesting! I'm not the only one: see MEST, AMPQ, and Erlang.

This goes back to a REST question I asked two months ago: is it still REST if you don't use HTTP? I'm guessing John would say yes.

I might be going out on a limb here, I'll bet the core of John's problem with SOA is how toolkits like WCF all but force you to build RPC style services that can easily be modeled as method calls. That's certainly one of my problems with SOA. Tim Ewald said it best:

It's depressing to think that SOAP started just about 10 years ago and that now that everything is said and done, we built RPC again. I know SOAP is really an XML messaging protocol, you can do oneway async stuff, etc, etc, but let's face it. The tools make the technology and the tools (and the examples and the advice you get) point at RPC. And we know what the problems with RPC are. If you want to build something that is genuinely loosely-coupled, RPC is a pretty hard path to take.

If SOA == RPC and REST == loosely coupled messages, then I'll start growing dreadlocks right now. Frankly, as Tim says, I think it's a problem with the tools (I'm looking at you WCF) and not the underlying architecture, but how many people can distinguish the architecture from the tools? Not many, I'm afraid.

Second, John asks an interesting question:

Where are the SOA mashups?

That's easy! They're inside the firewall where you can't see them! ;)

Seriously, I'm not sure about "SOA" mashups, but I'm working with what you might call a huge "enterprise" mashup system inside Microsoft. Our Enterprise Data Integration Services push around massive amounts of data to downstream systems. There are over fifty datasets in production, each with scores of tables, millions of rows and hundreds of subscribing systems. One example, our Products dataset, has over 100 tables and nearly 300 subscribing systems.

Is it "service oriented"? No, but then again it was originally developed ten years ago on SQL 6.5. But is it a mashup? Is it an "application that combines content from more than one source into an integrated experience"? Yep. Is it easy to work with? No, but guess why I'm involved? We're looking at ways to "modernize" the system. Am I going to build RPC style services as part of this modernization? Hell, no.

So John, am I right or wrong about that beer?

Is Serendipity the Heart of the WS-*/REST Debate?

Thanks to Technorati, I found this post by John Heintz. He's checking out John Evdemon's e-book on SOA and has a problem with this overview:

SOA is an architectural approach to creating systems built from autonomous services. With SOA, integration becomes forethought rather than afterthought. This book introduces a set of architectural capabilities, and explores them in subsequent chapters.

To which John H. responds:

I, for one, would rather build on an architecture that promotes integration as an afterthought, so I don't have to think about it before hand!!!

Yeah, I'd rather not have to think about integration before hand either. On the other hand, I want integration that actually works. It sounds like John H. is suggesting here that REST somehow eliminates the need to consider integration up front. It doesn't. Consider this: if you're building a Web 2.0 site then you are expected to expose everything in your site via APP, RSS and/or RESTful POX services. In other words, the Web 2.0 community expects you to have the forethought to enable integration. If you don't, Marc Canter will call you out in front of Bill Gates and Tim O'Reilly. 

This integration by afterthought approach seems to be big among RESTifarians. John H. links to a REST discussion post by Nick Gall advocating the principle of generality, "unexpected reuse" and "design for serendipity". Money quote:

The Internet and the Web are paradigms of Serendipity-Oriented Architectures. Why? Largely because of their simple generality. It is my belief that generality is one of the major enablers of serendipity. So here I immodestly offer Gall's General Principle of Serendipity: "Just as generality of knowledge is the key to serendipitous discovery, generality of purpose is the key to serendipitous (re)use."

Serendipity means "the accidental discovery of something pleasant, valuable, or useful". "Serendipitous reuse" sounds an awful lot like accidental reuse. Most enterprises have been there, done that and have nothing to show for their efforts or $$$ except the team t-shirt. Yet Tim Berners-Lee believes "Unexpected reuse is the value of the web" and Roy Fielding tells us to "Engineer for serendipity". What gives?

First off, enterprises aren't interested in unexpected or serendipitous reuse. They want their reuse to be systematic and predictable. The likelihood of serendipitous reuse is directly related to the number of potential reusers. But the number of potential reusers inside the enterprise is dramatically smaller than out on the public Internet. That brings the chance for serendipitous reuse inside the enterprise to nearly zero.

Second, enterprise systems aren't exactly known for their "simple generality". If Nick's right that "generality of purpose is the key to serendipitous (re)use", then enterprises might as well give up on serendipitous reuse right now. As I said last year, it's a question of context. Context is specifics, the opposite of generality. Different business units have different business practices, different geographies have different laws, different markets have different competitors, etc. If an enterprise operates in multiple contexts - and most do - enterprise solutions have to take them into account. Those different contexts prevent you from building usable - much less reusable - general solutions.

Finally, I think the amount of serendipitous reuse in REST is overstated. If you build an app on the Facebook Platform, can you use it on MySpace? Nope. If you build an app that uses the Flickr services, will it work with Picasa Web Albums? Nope. Of course, there are exceptions - pretty much everyone supports the MetaWeblog API - but those exceptions seem few and far between to me. Furthermore, the bits that are getting reused - such as identifier, format and protocol - are infrastructure capabilities more suitable to reuse anyway. Serendipitously reusing infrastructure capabilities is much easier than serendipitously reusing business capabilities, REST or not.

The problems that stand in the way of reuse aren't technology ones. Furthermore, the reuse problems face by enterprises are very different than ones faced by Web 2.0 companies. REST is a great approach, but it isn't a one-size-fits-all technology solution that magically relegates integration and reuse to "afterthought" status. Serendipity is nice, when it happens. However, by definition it's not something you can depend on.

Afternoon Coffee 106

Lots of meetings today, so my coffee post is late...

• The Big News^tm: Visual Studio 2008 and .NET Framework 3.5 Beta 2 is available for download. Soma and Scott have more. Silverlight 1.0 RC and the Silverlight Add-in for VS08 will apparently be available in a couple of days. Finally, there's a go-live license for the framework, so you get a head-start deploying apps before VS08 and NETFX 3.5 RTM. Time to build out a new VPC image.
• Next week, I'm attending the p&p Service Factory v3 Customization Workshop. I'm looking forward to playing with the new Service Factory drop, but I'm really interested in learning more about building factories. I wonder if they're going to discuss their VS08 plans.
• Nick Malik recently wrote about making "middle out SOA" work. I hate that term "middle-out". It feels like we're pinning our hopes on middle-out because we know top-down and bottom-up don't work. My old boss John DeVadoss (who assures me he'll be blogging regularly again "soon") big vs. little SOA, with big SOA being "dead". I like the term "little SOA" better than "middle-out SOA", but just because big SOA is a big failure, doesn't mean little SOA will make any headway.
• There's a new F# drop available. Don Syme has the details. Looks like they've got some interesting new constructs for async and parallel programing.
• ABC announced yesterday that they are streaming HD on their website. So you can check out the season finale of Lost in HD for free. They embed commercials so it's not really "for free", but you don't have to pay $3 an episode like you do on XBLM. I wonder if XBLM might offer this capability in the future? Certainly would increase my use of XBLM. (as would an all-you-can-eat pricing scheme)

Early Afternoon Coffee 105

• My two sessions on Rome went very well. Sort of like what I did @ TechEd last month, but with a bit more kimono opening since it was an internal audience. Best things about doing these types of talks is the questions and post-session conversation. I've missed that since moving over to MSIT.
• Late last week, I got my phone switched over to the new Office Communications Server 2007 beta. In my old office, I used the Office Communicator PBX phone integration features extensively. However, when we moved we got new IP phones that didn't integrate with Communicator. So when a chance to get on the beta came along, I jumped. I'll let you know my impressions after a few weeks, in the meantime you can read about Mark Deakin's experience.
• Matevz Gacnik figures out how to build a transactional web service that interacts with the new transactional file system in Vista and Server 08. Interesting, but personally I don't believe in using transactional web services. The whole point of service orientation is to reduce the coupling between services. Trying two services (technically, a service consumer and provider) together in an atomic transaction seems like going in the wrong direction. Still, good on Matevz for digging into the transactional file system.
• Udi Dahan gives us 6 simple steps to being a "top" IT consultant. I notice that getting well known, speaking and publishing are at the top of the list but actually being good at what you're well known for comes in at #5 on the list. I'm sure Udi thinks that's implicit in becoming a "top" consultant, but I'm not so sure.
• Pat Helland thinks Normalization is for Sissies. Slide #6 has the key take away: "For God's Sake, Don't Normalize Immutable Data".
• Larry O'Brien bashes the new binary efficient XML working group and working draft. I agree 100% w/ Larry. These aren't the droids we're looking for.
• John Evdemon points to a new e-book from my old team called SOA in the Real World. I flipped thru it (figuratively) and it appears to drill into the Foundations of Solution Architecture as well as provide real-world case studdies for each of the pillars recurring logical capabilities. Need to give it a deeper read.

Restating the Concurrency Problem

I'll be honest, I recommended Herb Sutter's concurrency series in today's Morning Coffee because it a series on concurrency by Herb Sutter. In other words, I hadn't actually read it yet, but I know how good Sutter's stuff is. Now I have read it and I wanted to re-issue my recommendation, even more strongly. Go read it.

Interestingly enough, I like the article because it doesn't provide an "answer" to the problem of concurrency. Rather, by providing a mental model, it essentially is a concise and precise restatement of the problem. Often, in the rush to find a solution to a problem, this step is skipped and it isn't until the end that you realize that you misunderstood the original problem and what you built doesn't match what you need.

I've often argued that this is also the key to selling in the enterprise. In my experience, whatever solution you're selling is usually way too complicated to be understood by the people who have the purchasing power to buy it. So explaining how your solution works or how your solution solves the problem isn't going to get you very far. However, the buyer does understand the problem at hand. Being able to demonstrate that you understand the fundamental nature of the problem and can communicate it back to them garners you a great deal of trust in the selling process. If you can show that you understand their problem, then you probably know how to fix it - even if the buyer doesn't understand how your solution works.

One other quick thought on Sutter's article. In discussing the use of concurrency to keep a GUI responsive (aka Pillar 1), he wrote the following:

Today, we typically express Pillar 1 by running the background work on its own thread or as a work item on a thread pool; the foreground task that wants to stay responsive is typically long-running and is usually a thread; and communication happens through message queues and message-like abstractions like futures (Java Future, .NET IAsyncResult). In coming years, we'll get new tools and abstractions in this pillar, where potential candidates include active objects/services (objects that conceptually run on their own thread, and calling a method is an asynchronous message); channels of communication between two or more tasks; and contracts that let us explicitly express, enforce, and validate the expected order of messages. [emphasis added]

If we're going to provide the ability to express, enforce and validate the expected order of messages between concurrent blocks of code, can we also do it for services across the network? WSDL is wholly deficient in this area. SSDL's Communicating Sequential Processes (CSP) and Rules-based Protocol Frameworks are a good start.

Morning Coffee 99

• Mladen Prajdic has a great post on handling a database in your unit tests. He mentions NDbUnit but seems mostly to favor SQL 2005's database snapshot feature. He's got sample code for creating and restoring a snapshot. (via DNK)
• Microsoft Robotics Studio 1.5 released yesterday. Tandy Trower - GM of the Robotics group - has the details on what's new.
• Herb Sutter has a new column in Dr. Dobbs on concurrency. First up, "building a consistent mental model for reasoning about concurrency". Sounds like a must read column. (via LtU)
• Scott Hanselman describes "Sez You Architecture". I wonder, do architecture ninjas get to wear a Shinobi shozoku?
• From the Not Everyone Agrees With DevHawk Dept.: Libor Soucek disagrees with me and thinks that durable messaging should be avoided. I had a hard time following Libor's logic but needless to say, I disagree with his disagreement. He writes that one of the reasons to use DM is for "Cooperating on transaction with external system". While multiple systems may be cooperating on a business transaction, in no way do I believe they are going to cooperate on a database transaction. But since he started talking about the DTC, I suspect we're talking past each other. Libor, drop me a line and we can discuss further.

Morning Coffee 97

• For the first six months of 2007, I posted 158 times in 181 days. I'm obviously off the pace I set in January of averaging a post a day, but I am averaging just under nine tenth of a post per day. Not bad. At this rate, I'll post almost as much this year as I did the last two years combined.
• It was a great family weekend. Saturday, three of my friends helped me move an upright piano that we got used for a great price. Luckily, one of said friends is also a physics teacher, otherwise I don't think we could have gotten that heavy thing in the truck. To say thanks, we BBQed for them Saturday evening. Then yesterday we took the kids to see a Sesame Street Live show. Both days were beautiful, which my wife greatly appreciated.
• The Caps hit the free agent market running yesterday, picking up Tom Poti (four years, $14 million) and Victor Kozlov (two years, $5 million). They weren't the A-list free agents, but they both seem like solid pickups. According to Japer's Rink, the Caps were about $6.5 million under the new cap minimum. These two signings just about close that gap, but it doesn't sound like they're done. That's good news for Caps fans.
• Scott Guthrie continues his series on LINQ to SQL. While I've seen most of this before, the cool thing Scott shows is hovering over the LINQ to SQL result and bringing up the exact SQL statement in a debugger window. That's pretty cool.
• Nick Malik is now "Mr. SOA" inside MSIT. As you might imagine, I'll be working with him fairly closely. Actually, he's late to a meeting with me as I type this.
• John Shewchuk announces a new version of BizTalk Services coming soon. The big new feature is access control for services exposed via the BizTalk Services. If you can't wait, you can try out the new stuff in their pre-production environment right now, before it's live. Is this a beta of a beta?
• Soma announces the MSDN Small Business Developer Center. I took a quick look thru the site. Strangely enough, it doesn't cover Dynamics - Microsoft's business software primarily targeting small and medium size businesses.
• Ted Neward called object/relational mapping the "Vietnam of Computer Science". David Chappell gives us our next war / technology analogy, declaring that the REST vs. WS-* war is over, ending in a truce like the Korean war rather than "crushing victory for one side".
• Like Jeff Atwood, I didn't realize About Face has been updated, twice. I am a huge fan of the first edition, but Jeff calls About Face 3 "the best edition of this classic yet". I just ordered a copy for myself.
• David McGhee transcribed a fantastic session with Dr. Don Ferguson at the Australian Architecture Forum on SOA/ESB integration in the real world. Go read the whole thing. Udi Dahan pulls out the quote "there is no such thing as a centralized ESB." Amen to that. My other favorite quotes from this discussion is "The temptation is often to get everything in a repository, but often you cannot rely on people to put everything in the registry" and "there is sometimes the “Highlander” philosophy of there can be only one service". If you're design depends on centralization and/or significant change in human behavior, it's doomed from the start. Frankly, it's amazing how often that happens.
• In response to my What is the Rails Question post, Hartmut Wilms wonders why "the .NET community (for the most part) ignores Open Source Projects". I wonder the same thing, though I don't think you can lump the whole .NET community together on this. While some parts of the community ignore anything they can't download from MSDN, other parts strongly embrace open source projects.

Morning Coffee 96

• My friend David "LetsKillDave" Weller writes a long post on corporate blogging, responding to comments on the subject from Penny Arcade. Andre "Ozymandias" Vrignaud also responds. David is specifically talking about blogging within the gaming division, but they apply pretty broadly to Microsoft as a whole when it comes to blogging. "I don't want to get fired", "I don't want to do things that needlessly hurt my company" and "We can say things that PR or marketing people can't.  Or won't." all ring true to me.
• Speaking of gaming, there seems to be more that your average cool games coming our for Xbox 360 this summer. I just picked up Forza 2 which rocks with the Racing Wheel. The Darkness looks very cool and I laughed my ass off playing the Overlord demo. Both shipped this week and have gotten good reviews. On their way in August are Bioshock and Blue Dragon. Of course, there are a few other big games coming this holiday. A good, but expensive, year to be a gamer.
• I laughed my ass off reading Larry O'Brien's Top 10 Things To Do With Your Petaflop Supercomputer, esp. #9.
• WSDL 2.0, it's official. Nick Allen has the news. Personally, WSDL seems to be the spec most responsible for driving RPC-style request/response web services, so let's just say that I am not a fan.  
• Joe McKendrick thinks something is "holding back SOA"? I don't think it's any one thing, but certainly the RPC style that most web service toolkits pretty much force down your throat isn't helping.
• Nick Malik thinks Acropolis is promising as a SOA service consumer, but Udi Dalan thinks it doesn't support multi-threading well enough. I lean towards Nick on this one since I see multi-threading as a language problem, which a library like Acropolis can't solve on it's own.
• Jon Flanders has been busy building the BizTalk Server 2006 extensions for Windows Workflow Foundation (June CTP) SDK Sample. I'm not sure why the marketing folks gave this such a long and involved name, but the sample does look pretty cool. Paul Andrews has the project overview and demo video. However, given that the WF workflows are hosted in BTS, is it accurate to say "No Biztalk Experience Required"?
• Speaking of WF, Tomas Restrepo takes a detailed look at the new WF service hosting in .NET FX 3.5. Mostly, he likes what he sees. I have the same problem he does with the message correlation IDs. I'd like to have other options here, including support for what I call "message data correlation" (Tomas describes this as "natural correlating identifiers") and "address correlation" which is basically the REST model.

Morning Coffee 95

• New version of dasBlog is out, the final version on ASP.NET 1.1 (unless this release "kills a kitten" as per Scott Hanselman). I don't have the time (make the time?)to run daily builds, but I do try and upgrade to new major releases in a timely fashion. I'm also moving hosters, so expect a little downtime around here at some point in the near future.
• Matt Winkler is doing a series on alternate WF execution patterns. His first is the N of M pattern. While I can nitpick some things in WF - especially the limitations of transaction flow - WF's support for variability and extensibility of execution patterns is fraking brilliant. (via Sam Gentile)
• Joe McKendrick is all excited about a SOA built without web services! We've been "doing SOA" since the EDI days without web services, so I'm not sure this level of excitement - with an exclamation point and everything - is warranted. But it is good to see people realize web services != SOA. Instead of web services, CERN is using JMS to move messages around. I don't know much about JMS, but I do know it supports async and durable messaging, two things I think are critical for enterprise services.
• I saw on LtU that there's a new paper on Singularity out. For those who don't know, Singularity is a MS Research platform designed for reliability instead of performance. But there's more than just a new paper. According to the project home page, "Singularity Version 1.0 is complete. We've shipped the Singularity Research Development Kit (RDK) to a small number of universities for their research efforts." I wonder if I can get my hands on that RDK?
• Jeff Atwood is starting to show ads on Coding Horror, but he's donating "a significant percentage" of the ad revenue back into the programming community. He's starting with $5,000 and Microsoft is matching for a total of $10,000 to be donated to open source .NET projects. Go tell Jeff which projects you think he should donate to. Castle seems to be an early favorite.
• On Monday, Nick Malik posted what he called the Simple Lifecycle Agility Maturity Model (aka SLAMM) as a way of measuring your "agile factor". Surprisingly, the community response has been zilch. After Nick's comments on Agile last week, I figured someone would have something to say about it, even if only to slam it. (Slam SLAMM, ha ha.) Maybe nobody opened the spreadsheet and saw Mort has an agile factor rating of 71%? Personally, SLAMM seems like a rather coarse tool for measuring how agile you are, but coarse tools are better than no tools at all.

The Myth of the Service Catalog

Normally, I would simply note that Nick Malik had finally moved on from the Great Mort vs. Agile Debate^tm in my next Morning Coffee post and be done with it. But man, his post on the Unimportant SOA Catalog is just too good to leave until then...

Nick has come around to the view that the catalog does "a really good job of solving the wrong problem". I agree 100%. I haven't talked about it much here, but my teammates could tell you I have been on a rampage about this internally. People think a service catalog will create adoption and reuse. That suggests that the big obstacle to service adoption and reuse is simply the issue of finding something to adopt or reuse.

It's not.

The next PM I meet who says "I wish I had a big catalog of services so I can search for something that I can take an external dependency on" will be the first. And they'll probably be wearing a straightjacket, because looking for dependencies on purpose like that is crazy talk. Project managers avoid external dependencies like the plague. So when a PM looks at a service catalog, what they see a big list of stuff they don't want to take a dependency on. Not exactly the mindset to stimulate reuse and adoption, is it?

Nick suggests that a catalog might work if it was small (20-50 services) but not for hundreds or thousands of services. I think it's a culture issue not a scale issue, so I don't think it would work even for a small number of services. But in the end, it's a moot point since he expects a large number of services. Different reasons, same conclusion. 'nuff said.

However, while I rail against wasting time building a service catalog that won't do what everyone thinks it will do, I haven't had a better idea.

Nick does. The Periodic Table of Services. Go read. More of my thoughts on this later.

Morning Doughnuts 11

Harry will be back on Monday so I will returning to blogging on my website, while I will let the expert return to his normal posts here (not that he really took a break). I agree with Harry's post in that I really want to get something built so that we can talk about more than theoretical models. Like last time I appreciate the opportunity to sub for the master this last week. I hope that you found some of my entries interesting.

• Sam Gentile wrote the other day why it's great to be a Microsoft developer. I enjoyed that post because I just celebrated the end of my first year here at Microsoft. At this point I am not sure what I have contributed, but I have learned a great deal and want to apply that knowledge over the next year to help the company to succeed. We really do have great people and great technologies.
• The Seattle/Oklahoma City Sonics hired a GM who is only 30 years old. You know you must be getting old when the people running the sports teams are younger than you. He comes from the Spurs organization though so at least he has a background from a successful franchise.
• Ben Pearce listed out his top 5 questions about PowerShell this week at TechEd. He also recommends the book "PowerShell in Action" by Bruce Payette. I heartedly agree with this endorsement as the book is excellent.
• It looks like there are going to be more family friendly games for the XBOX360. I for one am glad to hear that. The other day as I was trying to find some games my 4 year old with the broken leg could play I realized how many games I have that wouldn't be appropriate for him. This is very good news in my opinion.

Lunchtime Doughnuts 9

• I am a few days behind on this, but Joe McKendrick writes an interesting piece on if businesspeople are begging for SOA. It is fascinating because I believe that SOA should come from the business, not because of the delivery mechanism, but because of the results. If services will truly make a business more adaptable and responsive to change shouldn't all business people desire those results? At the same time they don't care how that end is achieved, just that it is. That's where we in the IT industry need to do a better job of working out the details amongst ourselves and show the business how SOA can benefit them. Once we do that we should see more SOA adoptions go smoother and real ROI can be seen.
• Joel Dehlin has blogged on the myth of youth being the ones that use instant messaging, publish and read blogs, participate in social networks, etc. I agree that the technology has been integrated into every layer of society. If you have ever been at the airport or at a Starbucks you know what I mean. Who is it exactly that has a Crackberry addiction? It seems technology has really become a part of our culture, and that it's not just one age group that is adopting the changes.
• Visual Studio 2008 shell was announced at TechEd yesterday. Even Harry who was on-site missed the release, but it certainly looks cool.
• If you have ever met me you would quickly discover I have quite a background in Unix. That being the case I couldn't ignore the news that Sun is releasing new blades for the desktop. I had a blade on my desk for several years and it was really a nice system to use. For those that would bash me since I work at Microsoft now I will just say that when you support Solaris boxes, having one on your desk is quite helpful. I don't take sides in the Holy War. (via Scoble)

Afternoon Doughnuts 8

Due to a temporary reassignment this morning, I spent my usual blogging time moving all of my computer equipment from one cube to another.

• Sam Gentile writes about the ALT.NET moniker. My favorite of his principles is number 4 where he discusses the importance of tools versus principles and knowledge. I really agree that knowledge and principles are more important because the best tools in the world can't help us if we don't know how and when to use them.
• I find Mark Cuban's ideas (here and here) about how advertising on the Internet is different than traditional media advertising. He points out that the ability of a provider to deliver higher simultaneous views is more important and valuable than delivering views for a longer period of time. I think he right on here, even if I believe his football league is going to fail. (via Blog Maverick)
• Worse Than Failure has been running an unusual contest to get the most interesting, buggy, and unusual way of writing a calculator application. They have 12 finalists for readers to review. I find that the descriptions of how the programs work (or don't) to be hilarious.
• As I have been working on service-oriented management and monitoring I have given a lot of thought to the best way to present the data. Doesn't it make the most sense to primarily display information from the business process point of view? I would be interested in your feedback.

Morning Doughnuts 7

Once again I get the chance to fill in for Harry while he is gone. I will attempt to keep up with the high standard he has set.

• Be careful with trampolines. I have a four year old that was bouncing on a trampoline this weekend. He twisted his knee slightly on landing, and decided that the fastest way to get comfort was to jump off the edge. His knee gave out and he fractured his tibia. I don't think this was how he planned to start his summer. At least he is pretty calm about the whole thing.
• It looks like there is a good product out there named CliSecure to obfuscate .net code. From what I was able to read it looks like a pretty decent product, even hiding the code while its in memory. (via Larkware)
• TechEd started this morning. While I am sure Harry will be giving some on-site reports there is a link to the virtual site here.
• There is a great video showing Gregor Hohpe talking about SOA, and the many unrealistic claims in the industry. If you have read any of Hohpe's work it is clear that he has a great understanding of the topic. (via Nick Malik)
• The NBA Finals will begin this week with the Spurs playing the Cavaliers and King LeBron. I wonder if this will help rescue the NBA from what seems to be a real apathy on the part of the average fan. Other than the series between Dallas and Golden State earlier in the play-offs it really seems there haven't been any compelling stories. The NBA really needs a shot in the arm to become relevant again.

Morning Coffee 87

FYI, I'm at TechEd all next week. Given that WiFi access at conferences usually blows, I'm not planning on regular morning coffee posts. I've asked Dale again to keep the lights on around here and he's graciously said yes. Since I'm not on vacation, I'll be lurking around as well, but I'll be in an out. See you in Orlando!

• Jeff Atwood proclaims that developers are their own worst enemy, because they write too much code. Add in a pinch of "not invented here" syndrome and I think you've got it. This is one of the reasons why people think Ruby is the tits.
• Scott Hanselman has already taken advantage of the new WLWriter provider customization API.
• Erik Johnson writes about the thunderous REST bandwagon. He doesn't explicitly say it, but my take is that he thinks this all ends up in some middle ground between REST and WS-*. I hear that, at least, if that's what he's saying. I'm not sold on "HTTP is all you need" - I need durability and async messaging, but I don't see how to get there from here with just HTTP.

The Case for Durable Messaging in Service Orientation

Nick jumps on the durable messaging bandwagon, pointing out that it’s HUGE. Not just huge, or Huge but HUGE. But for my taste, he could emphasis it even more – HUGE – and still not capture just how important I think durable messaging is. But while he could use more bolding and italics, he certainly explains the problem well:

The reason that [reliability] becomes a problem in SOA is because the basic strength of SOA is the message, and the weakest link is the mechanism used to move the message. If we create a message but we cannot be certain that it gets delivered, then we have created a point of failure that is difficult to surpass.

Durable messaging solves two fundamental reliability issues:

1. Transactional Message Send. I want to send a message to some external service as part of a transaction. That is, I only want to send the message if the transaction commits. If the transaction aborts, I don’t want to send the message. The only way to do this is to durably record the intent to send the message within the transaction and then deliver the message after the transaction successfully commits.
2. External System Unavailable. I’m sending a message to an external service that is unavailable at the moment. Maybe it’s a temporary network condition, maybe it’s scheduled downtime, maybe the data center burned down, I don’t know. But because the message is durably stored, I can retry long after the sending transaction has committed. Furthermore, I can continue to retry (until success of course) even if my sending system reboots, fails over to a hot standby or has to be restored from backup (assuming you backed up after message was sent).

However, Nick points out that reliability has to be considered as part of our design, so do Agility, Flexibility, Scalability, Maintainability, etc. etc. etc. Agility and flexibility require standard transport protocols while scalability and maintainability require intermediation. Unfortunately, at this time there is no standard transport that provides intermediation and durability. Nick says that Microsoft’s “platform is lacking here”, but I’d say it’s an industry wide problem.

Nick mentions least three Microsoft technologies that provide some sort of durable messaging – MSMQ, SSB and BizTalk – but they’re all proprietary. The market leader in this space is MQ Series, which is also proprietary. WS-RM was supposed to be support durable messaging, but doesn’t. There is the Advanced Message Queuing Protocol group, which is defining an open protocol for MQ style systems, but without involvement from any major platform vendors I’m hard pressed to see this go anywhere. Personally, I’d love to see the SSB protocol published, and apparently the SSB wire protocol was designed “to be completely SQL Server agnostic.” Here’s hoping that happens.

Nick goes on to call WCF “immature” because of the lack support for message durability. I think that’s somewhat unfair: I think it’s WS-* that’s immature here, not WCF. It’s easy to confuse the two since they’re so joined at the hip in WCF v1. But WCF’s support for MSMQ shows that it can handle durable messaging, even though there is no usable standard for durable messaging in the WS-* stable. Over time, I think WCF will evolve to support a larger variety of messaging scenarios – WS-*, REST, durable messaging, etc. – out of the box. But for those of us who care deeply about durable messaging, WCF’s current lack of support is pretty frustrating.

Morning Coffee 85

• Microsoft announces Surface Computing. When can you buy one for your house? Probably not anytime soon. TechMeme has lots more.
• The one piece of swag I want more than anything else at TechEd is an Evil Mastermind shirt.
• Nick Allen notes that WSDL 2.0 has reached "proposed recommendation" stage. I guess having a "recommended" version of WSDL is an improvement over the "note" version. But other than having a RESTful HTTP binding in addition to the SOAP binding - and being longer - what's new?
• Speaking of description languages, Don Box writes about the Web Application Description Language which looks very REST-y in that it supports specifying both the URI as well as the payload format. Like Don, I agree with Erik Johnson who commented that "people attracted to REST (in whatever form) are rebelling against interface-based programming more than WS-* itself". I have a longer post on this coming, but suffice to say I'm really souring on interface-based programming.
• Nick Malik writes that WCF is immature because of it's "lack of a routable, intermediable, declared message durability option". Yeah, that's a huge problem in my book too. It also relates to the last bullet - since durable messaging is inherently async, it doesn't fit well into the interface-based programming model.

Responses on REST

Here is a roundup of some responses to my REST discussion with David Ing. Mostly, I'm passing them along with minimal comment so I figured I'd group them into one post.

David started by leaving me a comment, but decided instead to post the comments on his blog. His big worry seems to be how well REST as CRUD will interop with REST as protocol. but in general I'm not sure that's a big worry. Bill de Hora's example below about WebDAV transactions activities seems to demonstrate at least some RESTifarians are cool with a protocol view of the world.

Erik Johnson (who's blog I mistakenly purged from my reader at some point, so needless to say I've resubscribed) writes that "real-world experience shows you rarely POST exactly what you GET" and that "even with the flawed cast of characters you see a lot of whining about...the pieces are there to build good systems that also make great constituents in anyone’s SOA." I agree 100%. He goes on to agree with me agreeing with Tim for people "not to limit their comprehension of REST around entities accessed via GET and PUT". Generally speaking, I agree with people when they agree with me, and this is no exception. Erik also has a REST post that predated my dustup with David(*) where he reaches some interesting conclusions about designing REST style systems.

Bill de Hora weighs in, pointing out that "value exchange != transaction". Given that I never suggested they were equal, I'm not sure what his point is (other than to be snarky). He also points to Subversion's use of WebDAV as an example of...well, I'm not sure since it seems to prove my point that a simple CRUD style approach doesn't cut it for many scenarios. According to the page Bill linked to:

"Subversion commits are modeled using the "activity" concept from DeltaV. An activity can be viewed as a transaction for a set of resources." [emphasis mine]

While I'm sure this sounds snarky as well, I really don't get what Bill is getting at here. It sounds like he took my disparaging of CRUD as a disparagement of REST, which is NOT how it was intended. This kind of layering of higher-level protocol concepts (like WebDAV's activity) is exactly what I was thinking of when I wrote "I can spurn CRUD and still embrace REST".

And though not a direct comment on my post, Omri Gazitt writes about REST vs. SOAP and the support for both in the next version of WCF. His main point is that "It seems like we'll continue to live in a world where multiple integration paradigms and protocol choices exist for applications". Of course, since he's with the WCF product group, regardless of your integration paradigm or protocol choice, WCF is the way to build it (at least on Windows). We'll have to wait a while to see if WCF is really "future-proof", but the ability to add significant changes in a .5 release is a fairly good sign.

(*) It's a very polite dustup, characterized mostly by agreement rather than disagreement. Which makes it, as dustups go, very boring.

Follow My Breadcrumbs

I hope to post my thoughts on protocol state next week. In the meantime, you can check out Isolation Support for Service-based Applications as well as Consistency for Web Services Applications. The papers (from the same authors) deal with the C and I of ACID, but in service scenarios where ACID transactions aren't feasible. Both papers have dramatically impacted my thinking in this space.

This Isn't The Droid I'm Looking For

Since David Ing responded to my REST/CRUD/CRAP entry on his blog, I guess that means I respond to his response here, right?

Actually, this is going to be very short(*) because I mostly agree with what David wrote. For example:

If we say stuff like 'REST shall/must replace all foolish WS-* SOAP systems (insert Nelson-style HaHa here)' then we are just repeating the same lessons as before - One size won't fit all and there is no One Ring to Bind Them illusion in REST as well. If you have something complex that fits the WS-* style then maybe REST isn't the droid you are looking for.

Of course, this begs the question "what is complex?" David described complex as "long running transaction updating multiple distributed stores". Personally, I'd replace "stores" with "systems" and I hate term "long running transaction" so I would have written "long running operation" or "business process", but grammatical nitpicking aside that certainly seems like a fair description. Certainly, most of the scenarios I'm looking at in my day job could be described as being long running and updating multiple systems. David says later that he thinks "a lot of apps don’t need to be modeled that way", but I'm not aware of the alternatives so I'll let him expand on this on his own blog. I'll try to get to writing about my thoughts around protocol state next week.

I did disagree with this one point:

We may say REST is really about a protocol state and not CRUD, but unless the rest of the world gravitates to that view then I'm afraid it just won't be. If, say, through some ongoing groundswell of common usage, people start modelling entities as dereferenceable URI's and using POST to do Create and Updates, then REST will be about CRUD by default. This is all very unacademic and unjust, but thems the breaks peachy. The lesson from SOAP is not to fight it by trying to re-educate the masses after they get a perception in their head.

I think it doesn't really matter how the rest of the world gravitates, it only matters how you as the the service provider choose to expose your service. If you're doing something simple like ATOM publishing, you can probably get away with REST as CRUD. (Would that be hi or lo REST?) If you're doing something more complex, either in terms of being long running or needing multiple updates in one atomic operation like Tim's airline example, you'll probably need to gravitate towards REST as protocol state. But can't the two models can co-exist nicely, even in the same app? No re-education required.

UPDATE - From Fielding's Dissertation: "REST relies instead on the author choosing a resource identifier that best fits the nature of the concept being identified". My point exactly.

(*) Wow, this post wasn't that short at all. Can you imagine how long this would have been if I had mostly disagreed with David?

Morning Coffee 83

• Dottie Shaw - PM on Rome - has a post on Durable Messaging and why she cares. Needless to say, I care about durable messaging too, for all the same reasons...
• PDC07 has been "rescheduled". Feel free to theorize amongst yourselves the reasons why.
• Friend and ex-boss John deVadoss speaks on Software+Services @ the Enterprise Architect Summit. He finds time to present, but his blog remains quiet...
• JD Meier announces a p&p twofer: beta 1 of both their TFS Guide and their Perf Testing Guide. The TFS guide looks like a packaged doc version of their VSTS Guidance Project wiki.
• Facebook announces their Facebook Platform and Microsoft is supporting it with PopFly and Silverlight. TechMeme has much more.
• Pandora on the Go delivers their service to Sprint cell phones for $3 a month. Raging deal, but where's the support for Windows Mobile phones? (via Knowing.NET)

REST is neither CRUD nor CRAP

In the wake of my praise for CRUD is CRAP, David Ing asked “how do you reconcile something like REST (astoria etc) with being CRUD-adverse? Is there a happy place where the two can go for coffee?” Sure there is: I hear Tim Ewald’s XML Nation serves great coffee and the scones are pretty good. (*)

Seriously, the key observation that Tim recently made is that REST != CRUD. Sure, it can be used that way and for simple scenarios it works fine. (I’ll define “simple scenarios” in a second.) But I don’t believe CRUD style REST works in the large. Tim said you can’t build with just CRUD because it’s “to simplistic to be useful”. I’ll go even more fundamental, using REST for CRUD means having giving up transactions entirely. I've already accepted that building loosely coupled services means giving up distributed transactions. But the idea of giving up transactions entirely is just crazy talk.

So when I said "simple scenarios" above, I meant "scenarios that don't need transactions". (I take it as a given that RESTifarians aren't hot for WS-AtomicTransaction.) ATOM Publishing is a simple scenario because the web resource authoring scenario doesn’t need transactions to protect updates to multiple resources at a time. If it did, I don't believe the REST as CRUD approach they use would work.

As you might guess then, I’m not a fan of Astoria. I believe the sweet spot for so called “data services” will be read only (because they don't need transactions, natch). I'm sure there are some read/write scenarios Astoria will be useful for, but I think they will be limited - at least within the enterprise.

If REST != CRUD, then what is it? Let's go back to Tim's post:

Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I'm not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressable by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you're moving to, making that your new state. A state's representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app's protocol can't work that way too. (The ATOM Publishing protocol is the canonical example, though its easy to think that its about entities, not a state machine.)

While I disagree with Tim's disagreement of ATOM (i.e. I believe APP is about entities, but it works because it doesn't need transactions), I agree 100% that REST is about protocol state. Tim lays this very clear in his airline reservation sample. Thus, I can spurn CRUD and still embrace REST if I want to.

Further, Tim's points on the opaque nature of RPC style interactions (which web services appear to have fallen into despite the best of intentions) are spot on. If you're doing simple request/response services, the protocol state is trivial, so that works fine. However, in the scenarios I face, long running services are the norm and managing the protocol state is critical. I've got some ideas on how to do that, but that's a future blog post.

(*) Actually, I have no idea if Tim even likes coffee or scones. FYI, DevHawk Nation would not feature great coffee or pretty good scones. We would, however, have Arrogant Bastard Ale on tap.

Early Afternoon Coffee 81

These morning meetings are really cramping the "morning" style of these posts. But better late than never, I guess.

• Politics 2.0 Watch: Democrats pwn Republicans online. (via Balloon Juice)
• Roger Wolter writes about integrating SSB with WF, something I've experimented with myself. I didn't find the integration quite as natural as Roger did - transactions are a real PITA, and Roger apparently he hasn't looked into that yet - but I agree 100% with the idea that "most SSB programs end up looking a lot like a workflow." Looking forward to seeing your code, Roger.
• Pat explains his Newton vs. Einstein view of distributed systems and then rants about Consistency, Availability and Partition-tolerance. In particular, he discusses the evolution of what consistency (and to less extent availability) means in the face of loose coupling. Do yourself a favor and give up on distributed transactions now. Also, Pat points to another paper from CIDR dealing with isolation in services. Haven't read it yet, but I've added it to "the pile".  
• David Chappell writes about Service Component Architecture vs. Service Oriented Architecture. Since I don't "do" evangelism anymore, I don't spend much time watching what our competitors are doing. According to the SCA website, SCA is supposedly a "a model for building applications and systems using a Service-Oriented Architecture." But according to David, SCA 1.0 focuses on "portability, not interoperability, and so they don’t fully define the interactions between components necessary to create composites that cross vendor boundaries." I realize that we don't industry agreement on all the details of what SOA means, but I think we all agree that it's cross platform and cross vendor. Or maybe we can't even agree on that much.

Morning Coffee 79

• Soma announces PopFly, the "fun, easy way to build and share mashups, gadgets, Web pages, and applications" from the Non-Professional tools team. The PopFly team blog has some videos. Sounds vaguely like Yahoo! Pipes, but cooler. While most of the focus is on their browser-based mashup creator, they also have VS support for the non-non-professionals among us.
• Eric Nelson suggests that the new Dynamics CRM systems is actually a LOB application platform in it's own right. More details in Ben Riga's MIX session. (via Gianpaolo)
• Sam Gentile is worried that C# is becoming to complex, especially when you also consider how fast the platform is moving underneath. When you get your head out of the debugger for a second and look at the Big Picture, it certainly seems overwhelming. Is it just a question of getting used to it? The first time I fired up the VS.net 2002 alpha and looked at all the classes in the BCL, I had the same overwhelmed feeling, but eventually I got over it. Or have things just gotten too big and move to fast now? If so, it's time for some new layers of abstraction...
• Udi Dahan writes about building testable services. Testability has to be a core consideration when building anything, but especially a reusable framework. I've had similar thoughts about language design. How do you unit test a DSL?
• Roberto Medrano of SOA Software thinks "maybe 20 percent of IT folks understand SOA and half of the rest think they do". Personally, I think most IT folks don't agree on what SOA is or should be. Furthermore, we don't even have a common lexicon to discuss it, so we end up talking past each other and arguing about topics we agree on. I think Roberto is really saying is "most people are wrong because they don't agree with what I think SOA is". (via Jack van Hoof)
• Jeffrey Snover talks about the virtuous cycle of .NET language support. His point is that time spent learning .NET pays off as you transition between system programming (C#, VB.NET), shell programming (PowerShell) and script programming (IronPython, DLR). I'm not sure I would break them down that way, but his point spot on.
• Clemens Vasters experiments with the new BizTalk Services with a sample called TweetieBot. I agree 100% with his point about the assumption of centralization will be challenged by the federation of personal services.

Morning Coffee 78

• Ian Griffiths posts a much longer version of "Even though the runtime supports multiple languages, most programmers are only fluent in one." (via Larkware)
• I wrote yesterday that Pat Helland's first post back was light on the tech talk. Luckily (for us) he takes the bus to work so he has plenty of time to write blog entries. Today's post is his "personal opinion about how computers suck". Money Quote: "We try too hard as an industry.  Frequently, we build big and expensive datacenters and deploy big and expensive computers. In many cases, comparable behavior can be achieved with a lot of crappy machines which cost less than the big expensive one."
• Steve Jones wrote that CRUD is CRAP. I agree 100%, but for additional reasons. Not only is it boooooring to write, it also delegates control outside of the service which I think is a mistake. Check out this post from Maarten Mullender who advised to do CRUD only when you can afford it.
• MIT Media Lab has created Scratch "a new programming language that makes it easy to create your own interactive stories, animations, games, music, and art -- and share your creations on the web" targeted at kids 8 and up. It's a dynamic visual programming language that looks like Lego. Between Scratch, Boku and Phrogram I think my kids will have lots of fun learning to program like daddy does. (via GeekDad)
• Halo 3 is coming September 25th! I foresee lots of people calling in sick that day. And the next. And the rest of the week, etc. etc. etc.

Making My Mark at TechEd

One of the things that's different about being in MSIT is that it's cut my travel dramatically. Basically, the only travel I've done since taking this job was Thomas Erl's SOA Workshop last September. So it came as a bit of a surprise when I got tapped to present at TechEd (at fairly close to the last minute).

The last year I ran the architecture track, all track owners were asked to include either a globalization or MSIT session in our track. Since then, MSIT has expanded it's role at TechEd dramatically, with 14 breakouts, 20 chalk talks and our own Technical Learning Center.

I'm doing a two chalk talks on my MSIT project, Rome. I mentioned the project back when I switched jobs, but we've never talked about the project by name before. We haven't accomplished quite as much as I'd hoped since then, but we've progressed to the point that we can talk publicly about what we're doing. Now that we've begun to open the kimono a bit, you should see more on Rome, Not only from me but also my teammates who blog: Dale, Rick and Dottie.

So if you're going to TechEd, make sure you stop by the MSIT Technical Learning Center and say hi. Unlike my last two trips to TechEd, I have very limited responsibilities this time - basically just show up on time and talk about what I do all day, twice - so that leaves plenty of time to attend sessions, chat up attendees and ride roller coasters. Hope to see you there.

ROME: Service Oriented Infrastructure for the Enterprise
Like most enterprises, Microsoft IT is adopting a service oriented approach for the development of their internal systems. However, in order to avoid projects building similar service infrastructures on a per application basis, MSIT realizes the need for a common service oriented infrastructure to build these systems upon. Inside MSIT, the Rome project is tasked with designing, building and maintaining the infrastructure for these services. Come chat with Harry Pierson, lead architect of the Rome project, and discuss the challenges of building a large-scale service infrastructure inside a large enterprise like Microsoft.

• MST02-TLC Monday (June 4th) 1:15-2:30pm
• MST14-TLC Thursday (June 7th) 1-2:15pm

Morning Coffee 76

• Dare Obasanjo sez Cool URIs Don't Change. He's got other versioning advice, but that's the main takeaway. Good advice that dovetails nicely with "It's the URI, Stupid".
• I usually agree with Jack van Hoof's stuff, but I don't agree with his thoughts on loosely coupled transaction processing. It's much better than suggesting the use of 2PC system like WS-AT, but when he writes that "by design every action has a compensating action to undo the original action" I am reminded of Pat's old post Why I hate the phrase "Long Running Transactions". Personally, I'm a fan of using the Tentative Operation or Reservation pattern, described by John Evdemon. Note the lack of a transaction coordinator in that pattern.
• Speaking of service anti-patterns, I wonder how we rationalize the following two statements, both from Microsoft, in documents published by my old team:
  • "CRUD operations are the wrong level of factoring for a Web service. CRUD operations may be implemented within or across services, but should not be exposed to consumers in such a fashion. This is an example of a service that allowed internal (private) capabilities to bleed into the service's public interface." John Evdemon, Principles of Service Design: Service Patterns and Anti-Patterns, Readings in Service Orientation
  • "It is very common for Entity Services to support a create, read, update and delete (CRUD) interface at the entity level, and add additional domain-speciic operations needed to address the problem-domain and support the application’s features and use cases." Shy Cohen, Ontology and Taxonomy of Services in a Service-Oriented Architecture, Journal 11
• Ian Thomas wonders Does ERP suck? In a word: Yes! :) Seriously, I'm a strong believer in what Ian alternatively calls "unbundling" and "disaggregation" of monolithic enterprise systems - ERP is the most glaring example of such systems.
• Jamie Cansdale is figuring out how to host Silverlight's CLR outside of the browser. He's already got a console runner up and running. He's working of adding "Test With Silverlight" option to TestDriven.NET. You go Jamie.

Morning Coffee 71

• It's been almost four months since I started these morning coffee posts. I like the regularity - there's been 84 weekdays so far this year, so 84 - (71 + 6 days missed from vacation) = only seven missed morning coffees. On the other hand, I think my daily blogging fix is keeping me from digging deeper into some issues. So I'm going to start cutting back to only three morning coffee posts per week, with the hope of three deeper technical posts and one wildcard post per week.
• Speaking of cutting back, my parents are in town this weekend so I doubt I'll get a post out tomorrow or Monday. Have a good weekend anyway.
• Windows Server "Longhorn" Beta 3 is out. Now is time to start getting serious with it.
• Joe McKendrick is reporting that Gartner has given the green light to spending more on SOA. Maybe it's because I work for a technology savvy company, but I've never understood outsourcing critical business decisions about technology adoption to a consulting company.
• It's a Joe McKendrick twofer: He also reports that IBM is calling for a new SOA directory / discovery / registry standard to replace UDDI. I totally get the need a "new UDDI", though I'd wager that my issues with UDDI are very different than Big Blue's.
• Yesterday, I made a crack about how un-scalable the Internet would be if every cFonnection went thru a central hub. Two days ago, Clemens has a long post about the implications of an Internet Service Bus. First, I can't wait to see how that thing works. Second, it's fairly obvious that not all traffic will go thru this bus (since the bus ain't out yet and yet you're still reading this via the Internets), so maybe that answers my question about ESB's and centralization? That is to day, use the bus where you it's useful, otherwise don't bother?

Enterprise Service Bus? Give Me an Extra Special Bitter Instead

I went to a talk on BizTalk and ESB at lunch today that was sponsored by the local connected systems user group. Like many terms in this space (SOA and governance to name two others), ESB doesn't seem to have a consistent definition. The industry seems to be inventing terms at a fair clip as vendors attempt to differentiate themselves on what to me seem like fairly minor solution aspects.

Today's speaker talked at length about a "large health care company in California" that he had personally worked with, building an ESB for them with BizTalk. He spoke in glowing terms of the size of the BizTalk environment and the number of messages passing through the bus every day. Then someone asked how many systems this unnamed company had hooked up to the bus. He paused, then admitted: "Six".

Six? Not six whole systems! That's gotta be a record!

Of course, I realize that there are deployed ESB's out there that are integrating more than six systems. My group - the Integration Center of Excellence (ICoE for short) - runs a comparably sized BizTalk environment and we're connecting around 50 internal systems and hundreds of external partners. But 50 is still a fairly small number. I can't help but wonder how well will this ESB approach is going scale as the number of systems goes up a couple orders of magnitude. Frankly, I think the answer is "not well".

The problem I have with ESB is that it's a centralized approach. Given that one of the overriding trends of society in general and IT in particular is decentralization, the ESB approach feels like it's swimming against the current instead of with it.

As an analogy, consider how well would the Internet work if every connection went thru a central hub? See what I mean? Centralized systems don't scale like decentralized ones do.

I admit that there are scenarios where ESB-esque technology solves a practical problem. Transport adaptation and content based routing leap to mind. Services that need those capabilities should leverage ESB-esque technology. But whenever I listen to ESB proponents, I feel that the need for these capabilities is exaggerated to the point that every message exchanged between every service inside your enterprise travels on a central bus, which doesn't seem realistic to me.

Am I wrong about this characterization? Do ESB proponents think that all messages must travel on the bus? How about you? What do you think? Inquiring minds (aka me) want to know...

Morning Coffee 65

• My brother is a VaTech alumni, so the shock of the deadly shootings there yesterday hits very close to home. My heavy heart is with the grieving Hokie nation today.
• Jeff Atwood has a couple of great posts on Language vs. Platform. Earlier in my MSFT career, I spent a significant amount of time explaining .NET, often to companies that had made a significant investment in Java. Picking the Java platform is fine (it's almost the best platform around!), but it seemed many people I spoke to didn't understand the fact that "[w]hen you choose a language, like it or not, you've chosen a platform".
• Ian Thomas riffs on my When is a Service Not a Service post. I like Ian's thinking about SaaS as an analogy for SOA adoption - if for no other reason that SaaS is easier to "get". But trying to realize SOA via SaaS inside the enterprise is a mistake in my opinion (and I think Ian would agree with that). SaaS is a business model, and I don't think you want to turn your enterprise into an internal service marketplace. Instead, this ties back with Nick Malik's points about central planning. Regardless if I'm right or wrong, I subscribed to Ian's blog (and not just because he linked to me - check out his Elements of the Future Business Ecosystem)
• TLA Watch: Oracle coins Application Integration Architecture (aka AIA). Joe McKendrick calls it "Big SOA". Isn't this the market segment that BizTalk has been in for seven years?

Morning Coffee 63

• My friend Christoph Schittko (who used to blog here, but hasn't written anything in almost two years) recently wrote on an internal email thread that he "wonder[s] how many more attempts for “enterprise wide” thingies we need for people to figure out that there’s too much complexity involved to coordinate anything enterprise wide." I couldn't agree more, though I think it's more than just complexity at work here. There are significant forces driving decentralization in society in general and IT in particular, and anything enterprise wide is by definition centralized.
• I'm way behind on this, but Ray Ozzie did an fascinating interview with Knowledge@Wharton. I was especially interested in his separation of "big-I" and "small-i" innovation. Sounds like disruptive and sustaining innovation from The Innovator's Dilemma to me.
• According to Mary Jo Foley, my ex-teammate Mike Walker is "da man" on Office Business Applications, or OBA's.
• I think I have an old PocketPC hanging around in a drawer somewhere. Apparently, I can use it as a caller ID server instead of gathering dust. That's sorta freaky. (via Backstage @ MED)
• Quote of the Day: "If you have a live show on a TV network, Its not good to have a brain fart during a slow news week." - Mark Cuban. Personally, I don't care one way or the other about Don Imus, but Mark's points about the conservatism of media corporations are spot on.

Afternoon Coffee 59

Friday's Morning Coffee didn't happen because I fraked up the DNS settings when I moved devhawk.net to a new registrar. Today's morning coffee was drastically delayed on account of car troubles. Tuesday, I have an 8am meeting so tomorrow's not looking good either.

• The big news for Xbox 360 is details on the Spring Update. Big news is WLMessenger integration + a QWERTY thumb pad that snaps right into the controller. (via Gamerscoreblog and Major Nelson)
• Speaking of Xbox, I completed the Old Spice Experience Challenge today on my lunch break (couldn't go to the office due to the car troubles). My reward is an upgrade to level 2, a gamerpic I'll never use and a free copy of Contra. (Estimated total value: $5)
• Scott Guthrie continues his series on new language features in C#3/VB9. This time it's lambda expressions. This is the "killer" feature in the new language version IMHO, since you can use lambda expressions either as code or data. Furthermore, it's up to the class/method handling the lambda expression to decide if it should be treated as code or data. That decision is made and design time, but the upside is that as a developer, I write my queries exactly the same way regardless if they are to be executed directly (aka code) or analyzed (aka data). Scott also metions a few new LINQ to * projects: LINQ to Amazon, LINQ to NHibernate and LINQ to LDAP.
• Speaking of LINQ to *, here's LINQ to 3D Objects in a C# ray tracer. I think it's safe to say that LINQ to *whatever* is the new hotness. (via DotNetKicks)
• The new version of F# is out. Looks like the big new feature is Active Patterns which is described in this draft paper. If I only had more time to investigate this. (via Don Syme)
• For the third time in the past six months, my laptop power supply has died. I've never had a problem like this before, much less three times. It's not even the same laptop as I recently moved over to a Tecra M4 Tablet. I just don't get it.
• P&P has shipped the 3.0 release of Enterprise Library. Tom Hollander has the details. Personally, I am most interested in the new Policy Injection Block.
• Having worked with self-signed certificates and understanding what a PITA they are, it's nice to see that IIS 7 has explicit support for them.
• I saw a reference to "The Halo Effect" on one of the political blogs I read. Needless to say, as an Xbox gamer, my first reaction was that this had something related to Master Chief. It doesn't.
• Joe McKendrick compares SOA governance to national governance. Given our polarized political climate, this analogy may hurt more than it helps. Also, the next enterprise architectural board that has equal "branch" footing with IT and executive management will be the first.

Morning Coffee 52

• I finally found a use for the free SOA book I got from attending that Thomas Erl workshop. I'm using it to prop up one end of my daughter's mattress while she's sick so she can sleep better.
• Jeff Tash states axiomatically that CASE has evolved into Enterprise Architecture. I agree with his points about why software construction isn't like manufacturing, but he seems to be describing BDUF rather than EA. I'm anti-BDUF too, but why blame EA? (via John deVadoss)
• Joe McKendrick comments on my SaaS/SOA post and wonders if SOA should stand for "SOA Oriented Architecture". He also writes that most organizations these days don't have an SOA, they have an AOS, "Agglomeration Of Services". So true, so true.
• JD Meier talks up the new VSTS guidance available on CodePlex. Looks like some good stuff in there. I like how the p&p guys are moving from documents to wikis to deliver their guidance.
• I've held off on getting the HD-DVD drive for my Xbox, but I think I'm going to cave soon, where soon == about two months. That's when The Matrix Trilogy is released on HD-DVD. Right around my birthday too, how convienent.

When is a Service Not a Service?

Conceptually, I like both Service Oriented Architecture (aka SOA) and Software as a Service (aka SaaS). However, I think we've done the industry a disservice by overloading the term "service".

John deVadoss likes the following definition of SaaS from Wikipedia. So do I.

Software as a service (SaaS) is a model of software delivery where the software company provides maintenance, daily technical operation, and support for the software provided to their client. SaaS is a model of software delivery rather than a market segment; it assumes the software is delivered over the Internet. Software can be delivered using this method to any market segment including home consumers, small business, medium and large business.

To paraphrase, SaaS is software that traditionally you might have bought, installed and run yourself but instead now can access over the network where someone else is responsible for installing and running it. For example, instead of buying, setting up and managing my own mail server to handle a single @devhawk.net email address, I use the WL Custom Domains service.

SOA on the other hand isn't a model of software delivery, it's a model of software segmentation. Again, here's the Wikipedia definition, this time for SOA:

There is no widely-agreed upon definition of Service-oriented architecture other than its literal translation that it is an architecture that relies on service-orientation as its fundamental design principle.

Err, that's not very helpful. Let's check out the OASIS definition (cribbed from Wikipedia).

[SOA is] A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

Well, at least it's not a self-referential recursive definition. But it is littered with committee-speak. (Who talks like that in real life?) Frak it, here's my definition:

SOA is a way of implementing IT systems as a web of interconnected yet independent loosely coupled subsystems (typically called services) instead of as big honking systems we have traditionally built that tend to be unwieldy, in-agile, difficult to change and probably obsolete by the time they were deployed.

We could argue about the language, but you get the point. There would be a ton of argument about the size of the subsystems (i.e. the service granularity), but I think most people can agree that SOA encourages building multiple smaller interconnected subsystems instead of one big (honking) system.

Which brings me back to my original point: Service, in the SOA sense, describes the approach to factoring parts of an software solution. Service, in the SaaS sense, describes a software delivery mechanism. Certainly, you can use both together and take an SOA approach to building a SaaS product. But you don't have to. So having the same term "service" used in both is very confusing.

How many SaaS products use SOA today? I would guess "not many" since there hasn't been much demand for it. When you're selling to the long tail of the LOB market, support for service-oriented integration isn't a critical selling feature. As SaaS becomes more attractive to larger companies (i.e. ones with dedicated IT staffs), using a SOA approach will be more important to SaaS product vendors. So they will converge in a way, but not in the way their naming suggests.

Of the two uses, SaaS seems closer to the dictionary definition of service. Maybe the S in SOA should stand for "Subsystem"? Nah, I like the term "connected systems" better than "service oriented" anyway.

Morning Coffee 34

• Old news, but Reflector 5.0 is out. W00t! Not sure when Scott Hansleman became chief Reflector cheerleader, but he's got the rundown on the new features.
• Politics 2.0 Watch: OpenCongress. Sort of like Wikipedia for government. If we can disseminate information on bills and resolutions via the Internet, couldn't we collect votes on them as well?
• I got my hardcopy of Powershell in Action while I was on vacation. Highly recommended.
• Sam Gentle is starting to dig into WF, and he posts about the difficulty getting data in and out of workflows. He's using the ExternalDataService infrastructure which I don't like very much. I recommend getting friendly with the WorkflowQueuingService which is the low-level communication infrastructure that ExternalDataService builds on top of. The WQS docs are severely lacking, but it's fairly straight forward to figure out.
• Speaking of WF, Tomas Restpro reviews Programming WF. Sounds fairly introductory. Personally, Essential WF is one of the best tech books I've read in a long time, so I'll be skipping this book.
• My teammate Dale is continuing his daily posts on his blog.
• Joe McKendrick wonders if EDA is the new SOA. Frankly, both terms are so poorly defined that it's hard to determine exactly what each term means, much less how they're related. If you're an IT industry analyst, you probably can make a ton of cash describing the differences between them. Maybe it's me, but I don't see that much value in SOA without EDA. In fact, I'd go so far as to say service orientation without events isn't much a new architecture paradigm at all. It's just the Same Old Architecture with better support for interop.

Morning Doughnuts 6

• The Wall Street Journal is reporting that Massachusetts lawmakers are considering a bill to punish retailers for leaks in personal data. I wonder how long it will take for a law like that to go nationwide? Looks like there may be some good jobs in retail IT data security opening up shortly.
• There is an interesting debate on the SAAS architecture in Dr. Dobb's Portal. The money quote for me was as follows:

"Ajax and Web 2.0 are great technologies for casual use, but for mission critical you need the capabilities of a desktop app," RightNow CEO Greg Gianforte says.

I have to admit I don't agree with that quote at all. It seems pretty shortsighted in minimizing the capabilities of web based applications.

• As a follow-up to yesterdays entry about the 12 steps to overcome email addiction here is a 12 step program to help you overcome being a SOAholic. There are also some symptoms you can look for to see if you are a SOAholic.
• Ram Ravishankar posts on if SOA requires web services. He makes pretty good arguments for an against a SOA requiring web services and ultimately doesn't answer the question. I would say that a SOA doesn't require web services, but it is very likely in the range of 90% plus that a SOA within a company is going to have at least some web services in it.
• Harry returns from his secret mission and will be back blogging on Monday. I have really enjoyed stepping in being a replacement blogger this week. While my take on technology is a bit different that Harry's I hope that my entries were interesting and offered a bit of a different perspective on IT.

Morning Doughnuts 4

• According to Reuters surgeons who play video games are more skilled. Remind me to ask the doctor if s/he owns an XBOX 360 the next time I am getting operated on.
• I have reached the National Championship game in dynasty mode of NCAA Football 2007. The opponent of my BYU Cougars...why that would be Harry's alma mater, the USC Trojans. Funny how that worked out.
• Nicholas Allen writes in his blog about when you should use Indigo to write a channel, and more importantly when you should not. As most of you know Harry and I are doing quite a bit of work with WCF so we are interested in this type of advice.
• Our team has been thinking about how to manage a large number of services in an automated fashion. This would include deploying new services, monitoring the services, automatically handling scaling, service discovery, and automated provisioning to name a few possible capabilities. I almost think of it like the next version of UDDI, especially when it comes to provisioning. I think that as systems become more distributed that the ability to automatically manage these systems is going to be key to their success. I know that some thought has already gone on in this area by people far smarter than I, but as I consider how to operate an infrastructure with thousands of services in it it is apparent that the opportunity is there for us to design and implement a system management framework that automates the majority of the tasks. I need to spend some time to consider how the framework would work, and document the capabilities.

Morning Doughnuts 3

• What does it take to be an architect? Skyscrapr.net attempts to answer this question by asking a bunch of architects.
• I have started teaching my children about astronomy. I found an open source product called Stellarium that is excellent for learning about the celestial objects visible in your area.
• A Methodology for SOA adoption? I read an interesting blog on this subject from a couple of weeks ago. It's not a long article, but the author makes some interesting points including an outline for SOA adoption.
• I finally picked up Gears of War on Friday. It really isn't a game I can see playing much, although I can see why it's popular. I guess the best and the worst part of the game is having to utilize cover so you don't die right away.
• Windows Live Writer is a great tool! I use it to author the blogs for my website, and this week I have been using it on these Morning Doughnuts posts. My favorite feature is that you can preview your post and see exactly how it will appear on your website. This has been particularly useful since Devhawk and my site look quite different.

Morning Doughnuts 2

• Joel Dehlin, the CIO of the LDS church has an interesting blog entry on buy versus build this morning. His main point is that buying is often cheaper, but only if you can move your business processes to match the processes in the off-the-shelf software.
• The search for Jim Gray by his friends and colleagues has been called off. Even with a massive high-tech effort no new clues have been turned up. For the sake of his family I do hope that the mystery is solved. I would imagine it is very hard to not know what happened to him.
• I am currently running a Build and Deployment Task Force. We are trying to ensure that our team follows best practices when building new applications. The project that Harry and I are working on seems to be a good test bed for the process.
• For those of you who read my blog you know I am passionate about how we implement Service-Oriented Architecture in the real world. I have been reading a book titled Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology. I find the description of real business objections, and how to solve them quite refreshing.
• It appears that the San Diego Chargers are going to hire Norv Turner to replace Marty Schottenheimer as their head coach. I don't see how fans of the Chargers can possibly see this as an improvement.

Morning Doughnuts 1

Introduction

While Harry is out of the office on a well deserved vacation with his family I will be acting as a guest blogger for his website.

My name is Dale Churchward and I have been working at in Microsoft IT as an application architect since June 2006. Prior to coming to MS I worked in operations in both the telecommunications and healthcare industries. As Harry pointed out I tend to be very operations focused. While I love working on new technologies I am keenly interested in how a design will work in production, and try to ensure we have considered support considerations as part of our designs. Each member of our team comes in with a different background. This helps us as we are each strong in different areas. I also write to my own blog which can be found here. In my own blog I tend to write on a wide array of subjects, depending on my mood at the time. While Harry is away though, I will be posting on technology areas I am interested in, and staying true to his morning coffee vision, albeit with a slightly different take.

The Doughnuts

• The Build Master by Vincent Maraia is an excellent book if you are interested in the build process and how to make it as efficient as possible.
• We had a great meeting with the Patterns & Practices team the other day. Since I am still new to Microsoft it is still a bit overwhelming to meet the authors of documents you have read and used over the years.
• I recently have been spending some cycles working with System Center Operations Manager 2007. I believe that it provides some excellent tools to monitor and repair a system plus it's designed to be service focused.
• Francis Stokes has produced 6 episodes showing what would happen if heaven was being run like a company named God Inc. There are currently 6 episodes. No matter what your belief or lack thereof in a supreme being the videos are hilarious.
  • Episode 1
  • Episode 2
  • Episode 3
  • Episode 4
  • Episode 5
  • Episode 6 
• I have been spending a lot of time thinking about how heartbeat transactions between multiple services should operate. In the drawing below you can see 3 web services and a monitoring one. In the original design the monitoring service was sending heartbeats out to each of the web services to see if they were available. This seem inefficient to me as we really don't care if the monitoring service can reach the web service. What we need to know is if any dependent web services are able to connect. In the drawing we have a web service residing in the extranet (Web Service 1) that sends data to a web service in the corporate network (Web Service 2). We really don't care if the monitoring service can talk with web service 2, but we definitely want to know that web service 1 can get there. Once web service 1 realizes that is can't connect to 2 we then notify the monitoring system so that the owner of web service 2 can take action. Web service 1 still continues sending heartbeats though so that it is aware of when the second web service becomes available again.

Morning Coffee 26

• I wonder what MSBuild would look like if the team had cloned drawn inspiration from Rake instead of Ant. Seems that PowerShell would have made a great foundation for build scripting.
• Looks like the digital music business is about to undergo a dramatic shift. Nick Carr and Mark Cuban have more on the possible ramifications. A friend of mine is about to move over to the Zune team. Sounds like a good time to making that switch.
• Anne Manes of the Burton Group says the time is right for UDDI, calling it the "foundation for governance". Frankly, I think that gives UDDI a lot more credit than it's due. We're looking at UDDI as part of our SO infrastructure project, and I think it's more appropriately called "one piece of the puzzle". In my experience, the major roadblock getting projects to share technical details is desire, not discoverability. Getting information into the registry is much easier than getting teams to use that data rather than succumbing to Not Invented Here syndrome. (via Joe McKendrick)
• Jeff Snover of the PowerShell team left me a comment the I "get" PowerShell. "Getting" it may be a better description, but it's nice to see how well engaged the PS team is in the community.
• After 13 long weeks, Lost is back!

Morning Coffee 25

• I'm surprised we haven't seen a laptop with a flash memory hard drive yet. Given the significant power, heat and performance advantage of flash memory over hard drives, I would have expected the laptop companies to have high-end laptops with flash memory hard drives by now. I'm probably getting a new laptop in the next six months, but I'd hold out until the end of the year if it meant being able to get one with a flash memory hard drive.
• I wrote last month that "The next new language I learn will be F#". I was wrong. It's PowerShell.
• I've been listening to Scott gush about CodeRush for years now. His post yesterday about the new free version of Refactor! for ASP.NET finally kicked me into action. I installed the CodeRush trial and will be commence bugging my boss to buy it.
• Looks like big news brewing in the online identity space.
• Dale is talking about service heartbeats. I'm pretty stoked that Dale is now spending 100% of his work time (when he's not blogging about sports, politics or video games anyway) with me building service oriented infrastructure for MSIT.

Morning Coffee 19

• I find Jim Kobielus' "SOA as 50 square miles of fungus" analogy funny and strangely compelling. The "keep in the dark and feed it shit" jokes practically write themselves. (via Joe McKendrick)
• Politics 2.0 Rising: The number of Americans who got "most of their information" about the 2006 midterm elections was double the number from the 2002 elections.
• Do you use external/flash drives? Do you have issues when you try to "Safely Remove Hardware" with said drive? I do, all the time. Apparently, unlocker is the answer. (via Paul Andrew)
• How come there's no information about LogToTraceListeners in the WF documentation? As far as I can tell, it's not in the Windows SDK docs at all and the only reference to it on MSDN is this year-old article and this year-old blog post. I only discovered because someone on the internal WF discussion alias asked about it. I've added In my SSB/WF work, I subclassed the built-in SQL persistence service in order to add tracing support. If you're developing a WF host, you need to turn this on. I find it mind-boggling this isn't included anywhere in the official WF docs.
• Nice to see Soma bragging about Software Factories. As he writes, our current solution - consisting of the Guidance Automation Toolkit and the DSL Tools - are really just a first step. I'm just starting to experiment with the Web Service Software Factory (WSSF). Aaron Skonnard introduces both the ASMX and WCF version in his two most recent Service Station articles.
• Michael.NET makes Programming Promises and Ted Neward swears the Oath of the Conscientious Programmer. Why stop there? How about the Architect's Affidavit to actually implement the shit we draw on the white board? The Technologist's Testimony of understanding and belief in all things geeky and gadget? Come on, isn't this just called "doing your job"? Do we really need to make promises and swear oaths to take it seriously and do it to the best of our abilities?

Morning Coffee 7

News was expecting inches, but we only got a dusting of snow last night.

• We had dinner last night with my old friend Matt, who moved to Amsterdam a year and a half ago and is getting to travel the world. Kids didn't have a nap yesterday, so they weren't quite on their best behavior, but it was great to see Matt. Hopefully it won't be another 18 months before we see him again.
• For the second time in four months, the power cable for my laptop failed. I wonder if there is something wrong with the power supply that's causing the cable to fail? At least this time I wasn't in Canada.
• There's a high resolution video of the Xbox 360 IPTV up on Xbox.com's CES page. They make it very clear this is "something you need to get from your service provider". Telling quote: "It's kinda like what I have today, but better". Doesn't seem that much better, so far anyway.
• I'm knee deep in WCF security code again. Mucking about with X.509 certificates sucks. I tried to follow these directions to create a dev root CA certificate as well as dev certs signed by said dev root CA, but I get security negotiation errors because the system can't check to see if the cert has been revoked. I guess I'll just install Certificate Services instead
• My nominee for best new acronym: JBOWS (Just a Bunch of Web Services), apparently coined by Joe McKendrick. Web services, to date at least, seem like they're being used primarily for building distributed applications, rather than a loosely coupled services. There's nothing inherently wrong with that, unless you're fooling yourself (or those holding the purse strings) that you'll get integration "automagically" or "for free" just because you're using web services. Joe McKendrick is definitely not an SOA-hole.
• The next new language I learn will be F#. Just not sure when or how.

Morning Coffee 6

"The paper sure loves to talk about
Selling out
Some of us never get the chance"
Stick Around by Mr. Jones and the Previous

• Didn't see that coming. I guess the Buckeyes didn't either. Congrats to the Gators. That makes at least three championships in a row won by the underdog. For all the complaining about the BCS, it's hard to argue they got the champion wrong this year. However, with the exception of the Fiesta Bowl, the BCS games weren't very good this year.
• There's a video of the new Xbox 360 IPTV service up on 10. I realize it's a demo and we're nearly a year away from release, but I'm not impressed. Xbox 360 Fanboy pointed to a blogger who got a deeper look at the service at Microsoft's CE booth. Frankly, it doesn't look or sound like it's much different than standard cable service (though I like the sound of 35Mbps bandwidth at my house). I realize familiarity is good, but do we really have to lock ourselves into the existing TV paradigm?
• I got roped into a webcast today on Optimizing Application Platform Infrastructure. It's at 11am Pacific time. Stop by and say hi.
• My colleague Dale has a rant about Service Oriented Assholes. His definition: "Any person or team that pontificates on Service-Oriented Architecture (SOA) without considering the realities of implementing SOA in a real business environment with real suppliers, customers, and products. These people are great at designing something on a white board or on paper, but couldn't produce a real workable production ready system if their life depended on it." Sort of a more specific (and vulgar) version of Joel's "Architecture Astronauts". How many SOA-holes do you know?

The Two Types of Service Architects

Tomas Restrepo comments on my recent SSB and WCF posts:

Harry Pierson asks how well WCF supports long running tasks. He suggests that WCF does not support them very well, and says that's one reason he likes SQL Server Service Broker so much. I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database. Sure, this is an "expanded universe" with the CLR support in SQL Server 2005, but even so it makes me nervous at times .

You could also consider using a custom service with MSMQ or something like BizTalk Server for this if you had long running processes that were not completely tied to the DB (or a single DB for that matter).

Sam Gentile follows up:

In that same post, but I needed to call it out separate, Tomas rightfully says, "I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database," in response to Harry liking Service Broker so much. Talk about a narrow edge case. That's way I never really got excited or cared about Service Broker. Its a narrow solution to a special edge case when everything is database driven and can be executed totally inside the database. That's the old Microsoft Data-Driven Architecture for sure. Me, I'd rather have a rich Domain-Driven architecture most of the time. Then if you have Oracle databases in your architecture too, where does it leave you? Nowhere.

As you might expect, I have a few comments,  clarifications and corrections.

First, Tomas' statement that Service Broker only supports service logic "executed completely within the database" in flat out wrong. Service Broker can be used from any environment that can connect to SQL Server and execute DML statements. If you can call SELECT/INSERT/UPDATE/DELETE, then you can also call BEGIN DIALOG/SEND/RECEIVE/END CONVERSATION. This includes Windows apps and services, web apps and services, console apps and even Java apps. Of course, you can also access Service Broker from stored procedures if you wish, but you're not limited to them as Tomas suggested.

Tomas' misconception may come from a feature of Service Broker called Activation. Activation is a feature of Service Broker that dynamically scales message processing to match demand. For example, Service Broker can be configured to launch a new instance of a specified stored procedure if messaging processing isn't keeping up with incoming message traffic on a given queue. This is called internal activation and because it uses stored procedures it does execute within the database as Thomas said. Service Broker also supports external activation where it notifies an external application when activation is needed. You do have to build an application to host your service logic and handle these notifications, but that application doesn't execute within the database. So while you could argue that it's easier to execute your service logic within the database (no need to build a separate host app), it's not required.

Given that you don't have host your service logic in the database, then you're also not limited to "a single DB" as Tomas suggests. You don't, in fact, have to put your Service Broker queues in the same database with your business data. So if you have Oracle in your environment, like the scenario Sam mentioned, you would host your service logic in an external application that processed messages from a queue in a SQL 2005 database while accessing and modifying business data from tables in the Oracle database. Using multiple databases does require using distributed instead of local transactions, but if you're using MSMQ as Tomas recommended, you're already stuck with the DTC anyway.

Finally, I didn't get Tomas' "purely database driven" or Sam's "everything is database driven" comments at all. While there are exceptions, the vast majority of systems I've ever seen/built/designed have essentially been one or more stateless tiers sitting in front of a stateful database. If it's a traditional three tier web app, there's a stateless presentation tier, a stateless business logic tier and a stateless data access logic tier. For a web service, there's no presentation tier, but there's is the stateless SOAP processing tier typically provided by the web service stack. Does this mean the vast majority of web apps and services are  "purely database driven" too? If so, then I guess it's a good thing, right?

In the end, maybe there are two types of service architects - those that believe the majority of services will be atomic and those that believe the majority of services will be long running. For atomic services, Service Broker is overkill. But if it turns out that most services are long running, WCF's lack of support is going to be a pretty big roadblock.

I'm obviously in the long running camp. I'm not sure, but I get the feeling this is the less popular camp, at least for now. We'll have to wait to see, but I do know is that whenever someone brings me what they think is an atomic business scenario, it doesn't take much digging to reveal that the atomic scenario is actually a single step of a long running business scenario that also needs to be automated.

Here's a question for Tomas, Sam and the rest of you: Which group do you self select into? Are most services going to be atomic or long running in the (pardon the pun) long run?

The Other Foundation Technology

I mentioned last week that WF "is one of two foundation technologies that my project absolutely depends on". Sam Gentile assumes the other foundation technology is WCF. It's not.

As a quick reminder, my day job these days is to architect and deliver shared service-oriented infrastructure for Microsoft's IT division. These services will be automating long running business operations. And when I say long running, I mean days, weeks or longer. While there will surely be some atomic or stateless services, I expect most of the services we build will be long running. Thus, the infrastructure I'm responsible for has to enable and support long running services.

The other foundation technology my project depends on is Service Broker. Service Broker was expressly designed for building these types of long running services. It supports several capabilities that I consider absolutely critical for long running services:

• Service Initiated Interaction. Polling for changes is inefficient. Long running operations need support for the Solicit-Response and/or Notification message exchange patterns.
• Durable Messaging. The first fallacy of distributed computing is that the network is reliable. If you need to be 100% sure the message gets delivered, you have to write it to disk on both sides.
• Service Instance Dehydration. It's both dangerous and inefficient to keep an instance of a long running service in memory when it's idle. In order to maximize integrity (i.e. service instances survive a system crash) as well as resource utilization (i.e. we're not wasting memory/CPU/etc on idle service instances), service instances must be dehydrated to disk.

In addition to these capabilities, Service Broker supports something called Conversation Group Locking, which turns out to be important when building highly scalable long running services. Furthermore, my understanding is that Conversation Group Locking is a feature unique to Service Broker, not only across Microsoft's products but across the industry. Basically, it means that inbound messages for a specific long running service instance are locked so they can't be processed on more than one thread at a time.

Here's an example: let's say I'm processing a Cancel Order message for a specific order when the Ready to Ship message arrives for that order arrives. With Conversation Group Locking, the Ready to Ship message stays locked in the queue until the Cancel Order message transaction is complete, regardless of the number of service threads there are. Without Conversation Group Locking, the Ready to Ship message might get processed by another service thread at the same time the Cancel Order message is being processed. The customer might get notified that the cancellation succeeded while the shipping service gets notified to ship the product. Oops.

There's also an almost-natural fit between Service Broker and Windows Workflow. For example, a Service Broker Conversation Group and a WorkflowInstance are roughly analogous. They even both use a Guid for identification, making mapping between Conversation Group and WF Instance simple and direct. I was able to get prototype Service Broker / WF integration up and running in about a day. I'll post more on that integration later this week.

Last but not least, Service Broker is wicked fast. Unfortunately, I don't have any public benchmarks to point to, but the Service Broker team told me about a private customer benchmark that handled almost 9,000 messages per second! One of the reasons Service Broker is so fast is because it's integrated into SQL Server 2005, which is is pretty fast in it's own right. Since Service Broker is baked right in, you can do all your messaging work and your data manipulation within the scope of a local transaction.

Service Broker has a few rough areas and it lacks a supported managed api (though there is a sample managed api available). Probably the biggest issue is that Service Broker has almost no interop story. If you need to interop with a Service Broker service, you can use SQL Server's native Web Service support. or the BizTalk adapter for Service Broker from AdapterWORX. However, I'm not sure how many of Service Broker's native capabilities are exposed if you use these interop mechanisms. You would probably have to write a bunch of application code to make these capabilities work in an interop scenario.

Still, I feel Service Broker's unique set of capabilities, its natural fit with WF and its high performance make it the best choice for building my project's long running services. Is it the best choice for your project? I have no idea. One of the benefits of working for MSIT is that I get to focus on solving a specific problem and not on solving general problems. I would say that if you're doing exclusively atomic or stateless services, Service Broker is probably overkill. If you're doing any long running services at all, I would at least give Service Broker a serious look.

Thoughts on the SOA Workshop

Last week, I attended an SOA workshop presented by SOA Systems and delivered by "top-selling SOA author" Thomas Erl. It was two SOA-jammed days + the drive to Vancouver and back primarily discussing SOA with Dale. In other words, it was a lot of SOA. I went up expecting to take Erl to task for his "Services are Stateless" principle. However, that turned out to be a misunderstanding on my part about how Erl uses the term stateless. However, while Erl and I agreed on optimizing memory utilization (which is what he means by stateless), that wasn't much else when it came to common ground. As I wrote last week, Erl's vision of service-orientation is predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse.

Erl spends a lot of time talking about service reuse. I think it's safe to say, in Erl's mind, reuse is the primary value of service orientation. However, he didn't offer any reason to believe we can reuse services any more successfully than we were able to reuse objects. Furthermore, his predictions about the amount of reuse you can achieve are completely made up. At one point, he was giving actual reuse numbers (i.e. 35% new code, 65% existing code). When I asked him where those numbers came from, Erl admitted that they were "estimates" because "there hasn't been enough activity in serious SOA projects to provide accurate metrics" and that there is "no short term way of proving" the amount of service reuse. In other words, Erl made those numbers up out of thin air.

This whole "serious" or "real" SOA is a major theme with Erl. One the one hand, I agree that SOA is a horribly overused term. Many projects labeled SOA have little or nothing to do with SO. On the other hand, it seems pretty convenient to chalk up failed projects as not being "real" SOA so you can continue to spout attractive yet completely fictional reuse numbers. I asked about the Gartner's 20% service reuse prediction and Erl responded that low reuse number was because the WS-* specs are still in process. While I agree that the WS-* specs are critical to the advancement of SO, I fail to see how lack of security, reliable messaging and transactions are holding back reuse. If anything, I would expect those specs to impede reuse, as it adds further contextual requirements to the service.

While I think Erl is mistaken when it comes to the potential for service reuse, he's absolutely dreaming when it comes to the organizational structure and behavior that has to be in place for this potential service reuse to happen in the first place. I'm not sure what Erl was doing before he became a "top-selling SOA author," but I find it hard to believe it included any time in any significantly sized IT shop.

Erl expects services - "real" services, anyway - to take around 30% more time and money than he traditional siloed approach. The upside for spending this extra time and money is the potential service reuse. The obvious problem with this is that we don't know how much reuse we're going to see for this extra time and money. If you spend 30% more but can only reuse 20% of your services (as Gartner predicts), is it worth it? If you end up spending 50% more but are only able to reuse 10% of your services, is it worth it? Where's the line where it's no longer worth it to do SOA? Given that there's no real way to know how much reuse you're going to see, Erl's vision of SOA requires a huge leap of faith on the part of the implementer. "Huge leap of faith" doesn't go so well with "corporate IT department".

Furthermore, the next IT project I encounter that is willing to invest any additional time and money - much less 30% - in order to achieve some theoretical organizational benefit down the road will be the first. Most projects I've encountered (including inside MSIT) sacrifice long term time and money in return for short term gain. When asked how to make this 30% investment happen, Erl suggested that the CIO has to have a "dictatorial" relationship with the projects in the IT shop. I'm thinking that CIO's that adopt a dictatorial stance won't get much cooperation from the IT department and will soon be ex-CIO's.

In the end, I got a lot less out of this workshop than I was hoping to. As long as SO takes 30% more time and money and the primary benefit is the same retread promises of reuse that OO failed to deliver on, I have a hard time imagining SO making much headway.

PS - I have a barely used copy of "Service-Oriented Architecture: Concepts, Technology, and Design" if anyone wants to trade for it. It's not a red paperclip, but it's like new - only flipped through once. :)

Stateless != Stateless

A while back, I blogged that Services Aren't Stateless, in response to some stuff in Thomas Erl's latest book. At the time, I mentioned that I was looking forward to discussing my opinions with Erl when I attended his workshop. I've spent the last two days at said workshop. I'll have a full write-up on the workshop later this week, but I wanted to blog the resolution to this stateless issue right away.

At the time, I wrote "I assume Erl means that service should be stateless in the same way HTTP is stateless." Turns out, my assumption was way way wrong. When he addressed this topic in his workshop, he started by talking about dealing with concurrency and scalability, which got me confused at first. Apparently, when Erl says stateless, he's referring to minimizing memory usage. That is, don't keep service state in memory longer than you need to. So all the stuff about activity data, that's all fine as per Erl's principles, as long as you write it out to database instead of keeping it in memory. In his book, he talks about the service being "temporarily stateful" while processing a message. When I read that, I didn't get it - because I was thinking of the HTTP definition of stateless & stateful. But if we're just talking about raw memory usage, it suddenly makes sense.

On the one hand, I'm happy to agree 100% with Erl on another of his principles. Frankly, much of what he talked about in his workshop seems predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse. So to be in complete agreement with him on something was a nice change of pace. Thomas is much more interested in long-running and async services than I originally expected when I first flipped thru his book.

On the other hand, calling this out as a "principle of service orientation" hardly seems warranted. I mean, large scale web sites have been doing this for a long time and SQL Session State support has been a part of ASP.NET since v1. Furthermore, using the term "stateless" in this way is fundamentally different from the way HTTP and the industry at large uses it, which was the source of my confusion. So while I agree with the concept, I really wish Erl hadn't chosen an overloaded term to refer to it.

Feasible Service Reuse

Yesterday, I posted about services and reuse. More to the point, I posted why I don't believe that business services will be reusable, any more than business objects were reusable. However, "can't reuse business services" isn't the whole story, because I believe in different kinds of reuse.

The kind of reuse I was writing about yesterday is typically referred to as "black box reuse". The idea being that I can reuse the item (object or service) with little or no understanding of how it works. Thomas Beck wrote about colored boxes on his blog yesterday. Context impacts reuse - the environments in which you plan to reuse an item must be compatible with what the item expects. However, those contextual requirements aren't written down anywhere - at least, they're not encoded anywhere in the item's interface. Those contextual requirements are buried in the code - the code you're not supposed to look at because we're striving for black box reuse. Opaque Requirements == No Possibility of Reuse.

As I wrote yesterday, David Chappell tears this type of reuse apart fairly adeptly. Money quote: "Creating services that can be reused requires predicting the future". But black box reuse this isn't the only kind of reuse. It's attractive, since it's simple. At least it would be, if it actually worked. So what kind of reuse doesn't require predicting the future?

Refactoring.

I realize most people probably don't consider refactoring to be reuse. But let's take a look at the official definition from refactoring.com:

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring

Two things about this definition imply reuse. First, refactoring is "restructuring an existing body of code". It's not rewriting that existing body of code. You may be making changes to the code - this certainly isn't black box reuse - but you're not scrapping the code completely and starting over. Second, refactoring is making changes to the code "without changing its external behavior". You care about the code's external behavior because somewhere, some other code is calling the code you're refactoring. Some other existing piece of code that you don't want to change - i.e. that you want to reuse.

When you refactor, you still reuse a significant amount of the code, but you’re not having to predict the future to do it. Refactoring is the kind of reuse I believe in.

In his article, David talks about types of reuse such as business agility, adaptability and easily changeable orchestration. These look a lot more like refactoring than black box reuse to me. Unfortunately, David waves these away, saying  "Still, isn’t this just another form of reuse?". Reconfiguration hardly qualifies as "predict the future" style reuse that he spends the rest of the article arguing against. It's just one paragraph in an otherwise splendid article, so I'll give him a pass this time. (I'm sure he's relieved.)

A Question of Context

A couple of weeks ago, David Chappell posted a great article on SOA and the Reality of Reuse. When someone mentions the idea of using SOA for reuse, I cringe. David does a great job blowing the "SOA for Reuse" argument out of the water. In the future, I will just send that link rather than spending the time arguing out the point.

But something nagged at the back of my brain about that post. David starts by talking about object reuse before making the parallel to services. The problem with that comparison is that object reuse hasn't been a failure. When was the last time you wrote a String class? A Linked List? A Button? There's been support for Buttons in Windows since at least the 3.x days (probably longer, but that's before my time). Whatever your OO language of choice, there's a big collection of reusable objects to go with it.

Given his position as "famous technology author", I'm assuming David is well aware of successes of object reuse. Furthermore, I doubt it was an accident that in his article he writes that "reuse of business objects failed" (emphasis added). While there has been success around object reuse, essentially none of those successes have been in a business scenario. In fact, there have been some high profile projects such as Microsoft's Business Framework and IBM's San Francisco Project that have crashed and burned been significantly less than successful.

So here's the question: given that general object reuse has seen some success, what's so different about business objects that causes reuse to fail utterly? Since we're really interested in service reuse, knowing why some object reuse succeeds and other reuse fails will help us understand which services are likely to be reusable and which wont. I would say that success of object reuse hinges on context.

Wikipedia gives this definition of context: "The context of an event, word, paradigm, change or other reality includes the circumstances and conditions which surround it." (emphasis in original) For example, the word "order" is ambiguous. If you're using a procurement system for the military, you could conceivably be given an order to place an order. (OK, that's silly. But you get the idea.) The word "order" has two different meanings. However, the words that surround the ambiguous term make the meaning clear. An order that you place is different that an order that you give. That's context.

A string or a linked list or a button has very little in the way of contextual needs. That is to say you can use it the exact same way in a wide variety of environments. A business object on the other hand has significant contextual requirements, which makes reuse difficult or impossible. For example, a Purchase Order object from the above-mentioned military procurement system sounds like it might be reusable. At least until you take into account the differences between branches of the military, between ordering tanks and ordering uniforms, between active units and reserve units, etc. Once the generic Purchase Order has been specialized to the point of practical usability for a given scenario, it's no longer reusable in the general case.

Taking this back to the service realm, likewise I figure the reusable services will be the ones with little or no contextual needs. A good example is the identity and directory services such as Active Directory and its competitors. Sure, you use LDAP not SOAP to access it, but AD is certainly both reusable and a service plus it's in wide usage. Other candidates for reusable services my team is looking at are service directory, management and operations, business activity monitoring and provisioning.

I actually think there will be less reuse in services than there was with objects. The value of reuse of services has to exceed not only the contextual issues but also the overhead of distributed access. Calling across the network is an expensive operation - whatever's on the other side better be worth the drive. I'm guessing for services, more often than not, reuse won't be worth the trip (if it's possible at all).

Update - David pointed out to me that the last paragraph of his article begins:

Object reuse has been successful in some areas. The large class libraries in J2EE and the .NET Framework are perhaps the most important examples of this.

Doh! I guess my "assumption" that David is aware of successful object reuse was correct.

SOA Sample Scenario

So now that I'm back, we're beginning work in earnest on my project. For those not following along at home, my project is to deliver shared service oriented infrastructure for Microsoft's internal IT department. We've spent the time since I moved over working on our business justification, and now we're moving into specifications and prototyping. That is, I get to starting getting my hands dirty.

As part of the prototyping efforts, we're looking to build some sample business services on top of our prototype infrastructure. The idea being to both illustrate what we're building as well as have a playground where we can experiment with new ideas. During the prototyping, I'll be pretty involved with the development. But once we start writing production code, the dev team will take that over but I will continue to own the service playground. I've been kicking a playground idea like this around for several years, so I'm pretty excited about it.

The question is, what kind of business scenario should we build? I want the sample business services to be something interesting and real-world-esque. But of course, it can't be too complex since I wasn't hired to build a playground as my primary job function.

So far, we have two primary ideas:

• Enterprise Management System: AdventureWorks is the primary sample database that ships with SQL Server. They have business scenarios around Sales & Marketing, Product Management, Purchasing and Manufacturing. This sounds suspiciously like an ERP/CRM/SFA type enterprise system. On the plus side, MS is an enterprise so things like ERP/CRM/SFA are the types of solutions we need/use/buy/build internally. On the negative side, it's complex to do real-world and teams that actually do ERP/CRM/SFA inside Microsoft might dismiss the infrastructure if the playground isn't real-world enough.
• Prediction Market: If you've ever seen Hollywood Stock Exchange (HSX) or Tradesports, those are prediction markets. The basic idea is that you trade on predictions, rather than companies like you would in a stock market. Using HSX as an example, you get 2 million "Hollywood Dollars" (i.e. play money) to invest in upcoming movies and / or movie stars. Those movies and stars pay out based on the money they make at the box office. They also have derivatives for opening weekend, blockbusters and the Oscars. HSX even sells forecasting and prediction services based on the HSX market. Of course, I can't build something quite so extensive, but we could get pretty far with this idea. The upside is that it's relatively simple (compared to enterprise management systems) and there's little conflict with existing systems inside Microsoft. The downside also is that it's relatively simple and not like anything we're building inside Microsoft.

I'm leaning towards the prediction market, as it sounds more fun to build and experiment with. What do you think?

Cut Out The Middle Man

Nick Malik is an architect in MSIT's Enterprise Architecture group. He's been blogging a while, though I only discovered his blog a couple of weeks ago. Yesterday he posted about OMG's SOA SIG's Draft RFI on EDA and it's relationship to SOA and BPM. That's a veritable alphabet soup of acronyms! To translate, the Object Management Group's Special Interest Group on Service Oriented Architecture has posted a draft Request for Information on Event Driven Architecture and it's relationship to Service Oriented Architecture and Business Process Management. Here's the summary from the actual document:

The EDA Sub-group of the OMG SOA SIG seeks information from members of the EDA, BPM and SOA community as well as anyone interested in promoting standards in this area. Requested information will be evaluated by the EDA Sub-group, resulting in the development of Requests for Proposal(s) (RFP) for standardization of Event definition, relationship between EDA, BPM and SOA that will ultimately allow development of standards for Complete Life Cycle of Events -Ontology of Events, Sense and Respond Services, Events Metrics and processing of complex events. Please note that it is our intent to develop modeling standards for the EDA/SOA and EDA-Business Process interaction and provide standards for the implementation of that interaction as well.

First off, I'm a bit wary about this part: "it is our intent to develop modeling standards". Of course, OMG is responsible for UML and long time readers should be well aware of my opinion of UML. I don't want to set the bozo bit on an entire organization, but I am skeptical that any new modeling "standards" from OMG will be any more effective than the Unwanted Modeling Language.

Secondly, EDA seems to be vaguely defined, if at all. Wikipedia has this to say about EDA:

An event-driven architecture (EDA) defines a methodology for designing and implementing applications and systems in which events transmit between loosely coupled software components and services. An event-driven system is typically comprised of event consumers and event producers. Event consumers subscribe to an intermediary event manager, and event producers publish to this manager. When the event manager receives an event from a producer, the manager forwards the event to the consumer. If the consumer is unavailable, the manager can store the event and try to forward it later. This method of event transmission is referred to in message-based systems as store and forward.
[emphasis in original]

Assuming events are encoded as messages, then you can rewrite "event consumers / event producers" as "message receivers / message senders" and you really blur the line with SOA? 

The big difference in EDA seems to be the use of an "intermediary event manager". The problem I have with this approach is that the "intermediary event manager" works fine if you have a small number of endpoints, but how will it scale to handle hundreds of systems? Thousands? Tens of thousands? I don't see how the centralized event manager approach can possibly scale to handle tens of millions of events delivered between tens of thousands of systems. The management of such a system would be a nightmare? If a business process went south, you would obviously look in the central event manager as the source of the problem, but I would think that would be like finding a needle in a haystack. You could federate the event managers, instead of attempting to scale out the center. But a federated event manager approach would seem to defeat much of the purpose of an EDA in the first place. If you're going to federate your event managers, why not cut out the middle man and make each event producer it's own event manager as well? What is the benefit of separating these capabilities?

I guess fleshing out EDA isn't a bad idea, but it seems more like a solution looking for a problem to me.

Business Processes Are Services Too

I've been having a conversation with Piyush Pant over on his blog that started as a comment he left on my Services Aren't Stateless post. He thinks that I'm "missing the crucial point here by implicitly conflating business process and service state". While Piyush hasn't really defined what he means by these terms, I think I understand what he's getting at. Yes, process and service state are different in many ways, but they are also similar in that they are both service private data.

Pat Helland (side note - I wish Pat would start blogging again) wrote an article some time ago titled Data on the Outside vs. Data on the Inside where he talked about the differences between service private data and data in the space between the services. For example, data on the outside is immutable, requires an open schema for interop, doesn't need encapsulation and is representable in XML. Service private data is not immutable, doesn't need an open schema for interop, requires encapsulation and is typically stored in a SQL RDBMS. So on this front, process and service state are both service private data so conflating them makes some sense.

However, what's not in the article is the idea of Resource and Activity data. Not sure why Pat didn't include this in the article, but he was talking about it as far back as PDC 2003. Stu Charlton described the difference between resource and activity data in his Autonomous Services article:

Activity Data - This is "work in progress" data for any long-running business operation, and is usually encapsulated by business logic. A classic example is a shopping cart in any e-commerce system. This data is mutable, but typically has low concurrency conflicts, as it is not widely shared. Typically activity data retires after a long running operation completes, and may be archived in a decision support system for later analysis.

Resource Data - This is "state of the business" data, which represents the resources of an organization, and is usually encapsulated by business logic. Examples are: room availability in a hotel, inventory levels in a warehouse, account statuses, employee and customer information. Some resources have a small life span, others may last a very long time (years). Resource data is usually volatile with potential for high concurrency conflicts.

So I'm fairly sure that when Piyush says "process state" I should hear "activity data". Similarly "service state" is "resource data". The differences between activity and resource data lead to some interesting implementation artifacts, which I assume he getting at when he says I'm conflating the two. For example, since activity data like shopping cart has low or no concurrency issues, using an optimistic concurrency scheme is entirely appropriate, which you would never use for highly volatile resource data like warehouse inventory levels. In fact, since activity data doesn't have concurrency issues, you could even store it inside an instance of workflow or orchestration, which gets serialized to a persistent store when it's in an idle state.

However, the fact that activity and resource data is handled differently doesn't mean that most services won't have activity data. When Thomas Erl says that that stateless services is a "common principle of service orientation", essentially what I think he's saying that services should only have resource data. And as I said before, this seems wrong to me. Sure, some services will be stateless. But all services? Services implement business capabilities. Most business capabilities are long running processes. Doesn't that imply that most services in the enterprise will need to be long running workflows or orchestrations?

So for the most part, Piyush and I just seem to have different names for the same concepts. The one issue I have with Piyush's descricription of process and service state is that he seems to implicitly assume that processes aren't services. Why not? Again, not all services will be processes, but if you're not exposing processes as services, how exactly are you exposing them?

deVadoss Down on SOA

My old boss's boss seems like he was in a downer mood yesterday. First, he blogged about the "Myth of Reuse in SOA", then the "Achilles Heel of SOA". Actually, truth be told, I agree with him on both counts.

I slam the door on the reuse argument every time it comes up in my new job. Actually, I slam the door on what I call "Naive Reuse". When John talks factoring for agility, he's talking about a form of reuse - similar to how use "reuse" code when you refactor. What does it mean to refactor service? How about refactoring your enterprise?

As for the Achilles Heel "data problem", I think that's an artifact of the prevailing stateless request/response mindset most people have about services that I touched on yesterday. I think Pat Helland described a very good approach for dealing with data in an SOA, but I haven't seen it implemented broadly. Rest assured, many of the concepts Pat described are at the forefront of my thinking as my new project takes shape.

WCF Karma

Last fall, I was presenting to a group of architects about SOA. The previous speaker - Rich Turner - was running way late. As I walked in, he was doing a WCF demo and wanted to show how easy it was to change transport by changing the config file. He wanted to change it to run over named pipes, but he couldn't remember the name of the binding. He asked me, and I confessed that I didn't know either. So he gave up on demoing named pipes, finished his presentation and went on his way.

After he left, I confessed to the assembled architects that I knew *nothing* about WCF beyond the high-level concepts. I hadn't spent any time working with it at all. In fact, the only reason I had it installed was because it got installed automatically when you installed WPF which I was working with at the time. My reasoning, as I explained to them, was that WCF is a low-level abstraction. That is to say, WCF is nearer the bottom of the .NET Abstraction Pile than the top. I figured I'd let the people building the next generation of service-oriented infrastructure to worry about WCF.

Fast forward eight months, and my new job is about building service-oriented infrastructure. You know, the type that builds on WCF. Maybe it's karma, but I'm having to learn a lot about WCF right quick.

So as I get back into the blogging saddle, expect to see a bunch of stuff about WCF.

BTW, Major thanks to Sam Gentile, who's taken the time on email and the phone (on his vacation no less) to help talk some things thru. He suggested the WCF Hands On book, which is pretty good.

Services Aren't Stateless

My teammate Dale and I are going to an SOA Workshop in Vancouver in mid September. The workshop is put on by SOA Systems, which was founded by "top-selling SOA author" Thomas Erl. I have a copy of his first book, but I've never really opened it. Dale let me borrow Erl's second book. I figured since I was going to see him speak, I should at least flip through his books.

I was looking thru the chapter 9 "Principles of Service-Orientation". Most of them are spot on, if not exactly news. Services are loosely coupled, autonomous and share a formal contract. Yep, with you so far. But then I got to this one:

Services are Stateless
Services should not be required to manage state information, as that can impede their ability to remain loosely coupled. Services should be designed to maximize statelessness even if that means deferring state management elsewhere.

This seems way wrong to me on several levels. Now I'm really looking forward to going to Erl's workshop, so I can discuss this with him face-to-face.

First off, his terminology is confusing. I have a hard time believing he really think services in general should have no state at all. I'm sure there are some examples of completely state-free services, but I believe they are both very rare and fundamentally uninteresting. A simple calculator service has no state, but why would you actually build or use one (except as a demo)? I assume Erl means that service should be stateless in the same way HTTP is stateless. IMO, stateless is poor description of HTTP. Connectionless or sessionless would be more accurate.

Regardless of my opinions on poor terminology, the problem with stateless services is that many - perhaps most - business operations aren't stateless. And while HTTP is stateless, as soon as you use cookies, it becomes a stateful protocol. If you don't believe business operations are stateful, try buying something on your favorite ecommerce site with your cookies disabled. Most sites will give you a "Your computer must have cookies enabled" error message. Sites that still work are embedding a session ID in the URL instead of in a cookie (ASP.NET has built in support for this type of Cookieless Session State). Either way, state is required for even the simple task of ordering something from a web site.

If most business operations aren't stateless, why should services that implement business operations be stateless? This seems like a violation of the "but no simpler" part of Einstein's famous paraphrased quote.

Dad getting some link love

After a six month silence, Dad posted about what he called Business Oriented Architecture (BOA?):

If I had to guess what a service oriented architecture will eventually look like, I would guess that it would reflect the business architecture – Business Oriented Architecture (BOA). Business organizations have evolved over many centuries into a number of common “departments” – sales, accounting, personnel, etc. Perhaps that is a good starting place for services.

John forwarded me a post from Richard Veryard where he comments on Dad's post:

The issues addressed in my book are now becoming mainstream as the technological agenda of service-oriented architecture (SOA) starts to converge with the strategic agenda of the service-based business (SBB). This implies an approach to business strategy that involves dynamically managing the geometry of the business. (To achieve a fully adaptive enterprise we typically need to implement a variable geometry.) We can find elements of this thinking in some of the methodologies coming out of IBM and Microsoft, although from what I've seen so far I don't think any of these methodologies go far enough.

Hal calls this Business Oriented Architecture. If anything, I'd prefer to call it Architecture-Oriented Business. As Hal indicates, this calls for architectural thinking at the business level, which need to be aligned with architectural thinking at the information/software level.

This comes back around to the whole SOA top-down vs. bottom-up argument. Something I'll comment further on when I'm not up to my armits in moving boxes.

Crupi Blog

I met John Crupi - CTO of Sun's Enterprise Web Service Practice - at OOPSLA last year. Very cool guy. I hadn't realized he started blogging last month until Steve Vinoski linked to him. There seems to be some disagreement regarding using a top-down approach to SOA. Steve writes that SOA, like other "real-world development" is usually a mix of top-down and bottom-up. John writes that SOA is about a business driven architecture instead of an IT driven architecture. I liked this quote:

"[W]e cannot do SOA without a mutual effort between IT and the BU. Gone are the days of throwing the requirements over the fence and hoping it hits. Not only do these two groups have to work together, they have distinct roles and responsibilities. Basically, the BU runs the show and owns the business drivers, use-cases and processes. IT implements the BU requirements and owns the service definitions.  It's unfortunate that we really do have to refer to this as a "shift", because we should be doing this anyway. But, the reality is that IT and BU typical function as disparate groups and rarely work together to have the business use-cases drive the process and service definition."

John Evdemon made two points  relevant to this issue several months ago:

• SOA does not enable or ensure the alignment of IT and business. The IT industry has been promising this for decades – there is no silver bullet for aligning IT and business. Alignment of IT and business is an organizational issue that will not be resolved by an architectural design philosophy alone.

• Service Orientation will happen in your organization in one of two possible ways: chaotically (typical approach) or in a disciplined manner. The path your organization takes (and the cost of later fixing that path) is up to you.

I'm with John on this one. Traditional approaches to new problems are rarely successful.

Speaking of Steve

BTW, if you haven't seen Steve's entry on Isomorphism, go read it right now. He nails this whole services as contracts vs. services as code debate right on the head. The spot on nature of this post reminds me when he nailed the modeling problem on the head - another must read post if you haven't already.

New SSB Library and Community Site

A couple of weeks ago, Paul Murphy pointed me at a SSB Wrapper Class that's included in one of the SQL 2005 Beta 2 sample apps. That prompted an email from the SSB dev lead with a much better SSB library that they had written. You can get that library, along with two new samples that use it, up on the SQL Server Service Broker Developers Spot. The SSB Dev Spot is a community site dedicated to apps written on top of SSB. It's run by Dan Sullivan of Developmentor, who is also hosting his new blog on the site. I hear Dan and fellow DM-ers Bob Beauchemin and Niels Berglund are all very excited about SSB. Nice to know I'm not the only one.

Spelunking Service Broker - Dialogs

The simplest way to describe SQL Service Broker is "message queues in the database." If the queues are tables in the database then you can do your entire message processing using a local transaction. If you use a separate queuing technology like MSMQ, you can still pull items off the queue, update data in the database and send out messages in the scope of a transaction – it just has to be a distributed transaction. Being able to use a local tx gives us a big performance gain. But, while important, this perf gain isn’t the most compelling reason to use SSB. It turns out that SSB provides important semantic messaging benefits in addition to perf benefits.

If you study at the semantic messaging model defined by message queuing systems such as MSMQ, MQ Series and WS-RM you’ll notice that it is inherently one way. Section 2 of the WS-RM spec defines the reliable messaging model to be between a source sending the messages and a destination receiving them. The problem with this model is that as message patterns between services gets richer, we’re going to want bi-directional reliable messaging. SSB calls this a dialog.

Before you can send messages between services in SSB (assuming everything's been configured), you first have to BEGIN DIALOG and provide the initiating and target service, plus the message contract (more on that in a later post). Note that it's called initiator and target, not sender and receiver. Either side of the dialog can send messages to the other as part of the contract by using SEND ON CONVERSATION - the only requirement is that the initiator sends the first message (hence the name "initiator"). All messages are guaranteed to be delivered exactly once and in order, or both sides of the conversation are notified of the failure and the dialog is torn down. I'm not sure if it guarantees in-order delivery of messages in opposite directions. There are cases where this might be important - I send an order cancellation as I send you a shipping notification - but typically there's a business reason for who "wins" such a conflict. In this shipping/cancellation example, even if you sent me the cancellation before I sent shipment notification, it's not like I can recall the shipment.

My only issue with this bi-directional reliable messaging is that I'm so used to thinking in terms of one-way RM that it's taking me a while to wrap my head around it. Many of the sample interactions I come up with are simple patterns where the target simply acknowledges the incoming message. Order processing is a good example where I may get meaningful messages (i.e. not simple acks) coming from both ends. What are some others?

SSB Wrapper Class

Paul Murphy pointed out a SSB wrapper class that ships as part of the integrated storefront sample that ships with SQL 2005. It's a bit tricky to install, so I refer you to Paul's blog for instructions. (you have to install the sample installer) The wrapper class is in a file called ServiceBroker.cs and supports the following operations:

• Get Conversation Group
• Begin a Dialog
• Begin a Dialog (with related conversation)
• Send a Message
• Receive a Message
• Receive all messages from a conversation group
• End a Dialog

If you're not sure what dialogs and conversation groups are, I'll be blogging about them in the coming weeks.

Starting in on SQL Service Broker

Since Norman joined the team a few months ago, I’m no longer in firefighter mode. For about six months my team was short a marketing manager and I ended up picking up a bunch of extra duties. For example, I took over as TechEd architect track owner when out previous marketing manager left. I don’t know much about marketing, but I guess I did OK – I received a Marketing Impact Award for my work on TechEd. However, I’m very happy to have handed those marketing responsibilities back to their rightful owner. Even still, Norman is trying to educate me about marketing. He made me read the first two chapters of Building Strong Brands by David Aaker. I actually got thru the first three chapters before borrowing Birth of the Chaordic Age by Dee Hock from my father at Thanksgiving.

You would think that I would now have more time for blogging, but alas that has not been the case. When I was firefighting, I had no time for planning. Now, of course, I do. So between planning and thanksgiving vacation I’ve just been too busy to blog much.

One thing I’m getting into recently is SQL Service Broker. I’m working on an interesting community project that is building on top of SSB. Luckily, one of the primary architects of SSB sits down the hall from me. Of course, not everyone is so lucky, so watch this space as I dig deeper on SSB. A good place to start is the SSB First Look article. In order to start getting a handle on SSB, I ported the Hello World example from that article from T-SQL to C#. It’s a bit tricky, as there is no SSB-specific framework – you just use SqlCommand to execute SSB commands like BEGIN DIALOG and RECEIVE – but otherwise it’s pretty straightforward. My sample also demonstrates using the new SQL Management Objects to create databases and SSB related objects (message types, contracts, queues and services). Here’s the code – enjoy.

Roger Sessions on WS-*

Roger Sessions, noted architectural guru, author and Microsoft Architect MVP, has posted his latest newsletter on the WS-* family of specs. In typical Roger fashion, he decides to give them his own name - WS-SCRAM with SCRAM standing for "Secure, Coordinated, Reliable Async Messages". (I forget where, but I once heard Roger refer to "three-phase transactions" instead of the relatively standard "two-phase commit transactions".) Roger claims MS et. al. are pushing WS-STAR where "STAR" is an acronym standing for "Secure, Transacted, Async, Reliable". Personally, I've never seen it written out like a an acronym and I always thought the * in WS-* was used as a wildcard, but it is true that we are focused more on transactions than coordination. On the Longhorn DevCenter, Indigo is described as providing "secure, reliable, and transacted messaging along with interoperability."

This is actually the second time Roger's taken on transactions in a web services architecture. His last newsletter on the topic had a much more detailed but harder to follow example. He points out that there are two related specs in this space - WS-AtomicTransaction and WS-BusinessActivity but that he thinks only WS-BA is going to work in the long run. WS-AT requires holding locks open in the DB, which is highly unlikely between services in different trust boundaries communicating with async messaging. Am I really going to lock the records in my database while I wait for a service that I don't trust to figure out if it is able to commit or abort? I don't think so. Pat's wrote a great scenario showing how unrealistic the concept of long-running transactions really are.

However, at the end of the newsletter, Roger takes Indigo to task for implementing WS-AT and not WS-BA and I don't agree with him. While I think WS-AT shouldn't be used in web services architectures, Indigo is also responsible for moving existing technologies like COM+ forward. Leaving out WS-AT isn't really an option in those scenarios. As for not implementing WS-BA, I've got to wonder just how valuable is WS-BA? While WS-BA avoids the DB locking issue of WS-AT, it still doesn't deal with the delegation of control. WS-BA still leaves me beholden to the decision of some outside coordinator. This seems to violate two of Don's tenets of service orientation: "Boundaries are Explicit" and "Services are Autonomous".

Stage 1 on New WS Specs

I'll be honest, with a quick glance at WS-Enumeration and WS-Transfer I'm at what Don refers to as Stage 1. Especially WS-Transfer which appears at first glance to be CRUD for web services. Maarten talks about using CRUD only when you can afford it. My biggest issue with CRUD is that it assumes a trust relationship - that some other service is responsible for deciding when and how to CUD entities that I manage. I can't imagine exposing an interface like that on any service I build.

But it is nice to see we've started publishing specs in easy to download PDF format.

New SOA Section on Arch Center

By now, everyone has seen the big news - VS2005 Beta 1, Express versions and the new VS2005 Dev Center. I'm provisioning a new VPC to install it on. I'd love to move to VS2005 for all my dev work, but I'm writing a few apps that need to be working in a production environment long before we get to Beta 2, so I'm going to leave those on VS.NET 2003...at least for now. However, there are a few longer-term things that I'm working on that I've been wait to start coding on for VS2005 that can now get started.

On the architecture side, we launched a new SOA resource page. As part of the launch, we are featuring an updated version of Pat's Metropolis presentation and a new SOA whitepaper. The version of Metropolis that has been a part of the Architecture Strategy Series was one of the first times Pat had presented it. Like many presentations, Pat refined his delivery with each new time he delivered the talk. The new version provides a new section that describes SOA as well as a section drilling down on the guidance that the Metropolis model provides. Even if you've seen the previous version, you should check out the new version.

We also published a new whitepaper on Service Orientation and Its Role in Your Connected Systems Strategy. It takes a Crawl/Walk/Run approach to describing the approach and technology for service orientation today, next year with SQL 2005 & VS 2005, and further out with Longhorn and Indigo. It also covers such areas as Service-Oriented Analysis, Designing for Change and Extensibility, Service Management and Pattern of Service-Oriented Solutions.

You can discuss the new whitepaper, the new version of the Metropolis presentation or any other aspect of our SOA content on TheServerSide.NET.

TechEd Sessions Online

Streaming producer presenations from TechEd US have been posted. However, the streaming presenation site has the same lack-of-url-addressability issue that TechEd Breakout session list had. Is it so hard to just post a simple HTML file with a list of all the sessions in a track? <sigh>

Anyway, here are links to the top 5 sessions from this year's architecture track. For those who could not make it to TechEd (or were there, but missed these sessions) enjoy!

1. Realizing SOA by John deVadoss & Ron Jacobs
2. Data in SOA by Harry Pierson (i.e. yours truly)
3. Patterns in the Enterprise by Gregor Hohpe
4. Improving Application Perf & Scale by Chris Kinsman
5. Building Apps with P&P App Blocks by Wojtek Kozaczynski

UPDATE - The streaming site has been taken down, so none of the links work anymore. Sorry about that. We will be updating the Architecture Strategy Series at some point in the future with at least the Realizing SOA and Data in SOA talks.

Endangered Middle-Tier, Revisited

Since this blog is now being syndicated on Architecture Center, I thought I should repost links to a recent pair of entries I wrote entitled "Is the Middle-Tier Endangered?" and "The Endangered Middle-Tier, Part 2". The basic premise of the posts is that as computer hardware gets faster and service-orientation aims to carve our course-grained applications into finer-grained services, the value of running the business logic on a separate tier diminishes greatly. Add an improved programming model to the database (such as the CLR's addition to SQL 2005) and I feel that, eventually, it will make more sense to run the services in-process with the database instead of on a separate tier. We're not there yet - in addition to continued hardware improvements, we need a major improvement to the overall management infrastructure - but I think it will happen. The question is, do you think it will happen?

The Endangered Middle-Tier, Part 2

Among the responses to my endangered middle-tier post was this one by Ed Draper, architect evangelist for MSFT. While Ed makes some interesting points, on the whole his argument doesn't hold much water.

First off, his point about Moore's law is accurate, yet irrelevant. While Moore's law does relate to CPU speed, I was using it as an example of the rate that overall computing power improves. Storage capacity and network speed improve along a similar trajectory. 64-bit machines are at the early stages of becoming commonplace. And while it's true that scalability doesn't equal performance, having better performing hardware can significantly improve scalability.

(Side note - if we're going to be truly picky, Moore's law actually refers to rate of improvement of the number of transistors per IC, which only roughly equates to performance. Intel's Centrino technology is all about using those increasing numbers of transistors for other things like wireless networking and battery management.)

Ed spends quite a bit of time covering very low level computing concepts such as threads, locks, instruction cache, registers and the stack. I'm not sure why he does this. It almost feels like he only read the first part of my post, stopping right before he got to the part where I wrote: "Of course, it will be a long long time before Moore's law can provide a single machine to run a BIG enterprise app". In other words, I don't expect to run my ERP, SCM, CRM or other BFD enterprise-scale app on one machine!

Ed has missed a basic point of my post that I didn't spell out as I thought it was obvious: the independent services that make up a BIG enterprise class app can all run on different machines. I'm guessing he missed that point by the way he ended his post: 

"Yes, distributed computing is a good thing."

As Ted points out, it's not just a good thing, "distributed computing is a necessary thing." And a system of 100's or 1000's of independent services - which is what I'm describing - is significantly more distributable than the multi-tier monolithic applications we build today.

Of course, it is guaranteed that a small percentage of services will continue to need to use scale out techniques to reach their scalability requirements. For example, it will be a very very long time (if ever) before a single piece of hardware could handle the order processing load Amazon.com generates a week before Christmas or the tax return filing load at the IRS on April 15th. This problem isn't unique to application design. To draw a parallel to the database design world, there are a small percentage of tables that benefit from using filegroups to isolate them on separate drives from the rest of the database. It happens - but it doesn't always happen. It doesn't even usually happen. I like Josh's comment that "the vast majority of people who think they have workloads which require partition for scale are actually indulging in delusions of grandeur."

The point I was making is that computers are going to continue getting faster and service-oriented systems are likely to consist of boatloads of independent services with only modest scalability requirements. The combination of these two forces drastically reduces the number of scenarios where you need to use multi-tier scale out techniques to achieve the scalability requirements. If you don't need a multi-tier deployment, then there is a huge performance and scalability benefit to running the service logic in the database process. If you don't need to scale-out to achieve your requirement, why would you take the performance and scalability hit to run your code outside the database?

It's pointless to argue that computer aren't getting faster or that running the code in process with the database doesn't perform better. Ed (and Ted and Josh and everyone else reading this), I'd love to hear you opinions on the following:

• Will a service-oriented approach likely result in of boatloads of independent services where today we have one BIG app?
• If yes, will the vast majority of these boatloads of independent services have modest enough scalability requirements to run on a single piece of hardware in the near future?

Is the Middle-Tier Endangered?

Via Udi and Tiago, I found an article on DevX called "Kiss the Middle-tier Goodbye with SQL Server Yukon". While that article should have been titled "XML Technologies in SQL Server Yukon", both Udi and Tiago have interesting posts about SQL Server as a platform and the role of the database going forward. Personally, I see two primary forces at work that I believe will drive the middle tier into extinction: Moore's Law and Service-Orientation.

Why do we distribute applications across multiple systems today? Is it because we like managing multiple systems? No! It's for scalability. We exploit the fact that tiers of a multi-tier app have different processing loads and scalability methods. Typically, we scale the web/app tier by throwing more servers at the farm while we scale the data tier with bigger servers. However, as Moore's law increases the performance of these machines, the need to scale becomes reduced, From my experience, many smaller apps could easily run a single machine today (esp. when you consider the increased efficiency of eliminating the network and process hops). Moore's law will continue increase the headroom these machines provide and expand the definition of "smaller apps". If you can run the app on a single hardware node, there'd be little reason not to run as much of it as you can inside the database, other than "we might want to scale this out someday".

Of course, it will be a long long time before Moore's law can provide a single machine to run a BIG enterprise app (think something like SAP). This is where service-orientation comes in. While some people see services as a way to interoperate BIG apps, I think the future of services is to free us from building BIG apps, or at least from build BIG apps as monoliths. If you figure a BIG app like SAP has hundreds or thousands of business process or resource management operations, you could build that system where each operation is implemented as an independent service. The benefit of this approach is that it is orders of magnitude more flexible in the face of process change than the BIG app approach. If you haven't seen it, check out the Technology Roadmap session from the Architecture Strategy Series. One of the points that the presenter Norm Judah makes is that "it is the business processes in an organization that are unstable, but the individual things that people do...are the things that are stable." (He says that during Slide 9 - "Why Do Architecture Projects Re-Occur?") If you accept that, then having your business processes hard coded in a BIG inflexible app starts to look like a bad idea. These "individual things that people do" (such as sending out invoices) get mapped to services and are aggregated into business processes by some tool we haven't seen yet (though I think BTS 2004 is a good start).

If service-orientation shreds your BIG app into many pieces, it is highly likely that those pieces will be small enough to run on a single hardware node. In addition, as services communicate with asynchronous messages, they tend to have lower scalability needs than synchronous systems. And since these services all need a database (Pat refers to the database as the soul of the service) it makes sense to run the service inside the database itself. It even makes sense to build a both an application and a messaging infrastructure directly into the database engine. SQL Yukon provides the application infrastructure by hosting the .NET framework and provides the messaging infrastructure with the SQL Service Broker.

However, we need more than faster computers and service-oriented systems to eliminate the middle tier. We also need better management. And not incrementally better, orders of magnitudes better. If you're going to replace a single BIG app with hundreds of independent services, incremental manageability improvements are not going to cut it. Because we realize this too, Microsoft is investing heavily in the Dynamic System Initiative. If you're a developer, you've probably seen the Whitehorse designers coming with Whidbey. That's just part of DSI. We've got a great architectural overview of DSI as part of our Architecture Strategy Series or you can read more on the DSI homepage.

Do you think the middle-tier will become extinct? If it does, is that a good thing? Obviously, it's an unknown and a big change, which makes it hard to gauge. But what does your gut tell you?

Government Online International Network

While digging thru my referral logs, I found a link to a SOA news page from the Government Online International Network (or GOL-IN for short). I have no idea what this site is about, but it's an interesting news feed on SOA. For example, I found this interview with Bob Sutor, IBM's director of WebSphere infrastructure software titled "SOA Is So Necessary". GOL-IN gets their news via News Is Free, too bad then they don't in turn provide their own  RSS feed.

Is BAIT Out of Date?

I was re-reading the Microsoft Architecture Overview by Michael Platt that's up on Architecture Center. It's a little old, but still very valuable. In the overview, Michael discusses four perspectives on the enterprise architecture: business, application, information, and technology which are commonly collectively referred to as BAIT. (Note, I think it was Meta Group coined the term BAIT, but I'm not sure. Whoever did come up with the term, I'm pretty sure that it wasn't MSFT). However, as we march forward building services, I wonder if we need to regularly consider a couple of more perspectives.

I don't think service-orientation dramatically affect business or technology architecture. One of the big advantages of using web services to implement your services is that you can reuse a lot of the web-based technology infrastructure that companies spent the later part of the nineties building out. Likewise, while services will enable organizations to be agile and better achieve their business goals, I don't think it changes the actual goals significantly. However, application and technology architecture change radically when moving from an application-centric to a service-oriented approach.

When considering services, the application perspective is broadened to include both applications and services. According to our conceptual architecture, an application implements a user interface while a service is a discrete unit of logic exposing a message-based interface. Even though they are both typically written in code, I'm not sure lumping them together is the right architectural approach. Building a single service has many architectural similarities to building an application, but an SO design also has to tackle the architecture as a system-of-systems which the application-centric approach never had to worry about.

Likewise, the information perspective currently covers data both inside services (or apps - here the distinction is less important) and inside messages. Obviously, the approach to data private to a service will differ greatly from the approach to data inside messages. Things like transactional integrity, immutability, extensibility and encapsulation apply very differently to data and message architecture.

Each of the perspectives covers many different subtopics, so maybe there's no need to break services out from applications or messages out from information. However, I'm on record stating that I think using services "represents a fundamental change to the architecture model that the vast majority of systems running today were built on". Thus, I think that fundamental change should be surfaced in language we use to describe the architecture, since language influences thought. Granted, BSMAIT isn't as cool an acronym as BAIT, but it's far more representative of the way the next generation of systems are going to be architected.

Services Are Neither Applications Nor Components

When you create a system with web services, are your web services components or are they standalone applications? While this might seem like a semantic question, it is actually a very big deal, with a lot of implications in terms of your overall application design, as well as the design of the web services themselves. [Rockford Lhotka]

What's funny is that it IS a semantic question, and that's what makes it such a big deal. :)

Seriously, Rocky argues that web services should be thought of as applications, not components. I completely agree that a web service is "not a tier in an application" and that both services and applications have clearly defined boundaries. I also think he's on the right track when he writes that many application design principles apply to services. For example, both applications and services have data and business logic layers. But I would call the top layer of a web service the messaging layer, which has only superficial similarities to the presentation layer of an application (i.e. that's where input and output with the outside world is handled).

However, services also share similarities with components. Services, like components, don't stand alone. They may not trust each other, but they need to work together to some extent in order to accomplish work. This leads to design questions for services that are similar to component design questions. How do I determine which pieces of required functionality go in which services? How much process code is included in the data management services? What's the best way to design a service to optimize reuse?

In the end, services are fundamentally different animals than applications or components and I don't think we as an industry have enough experience building systems with them yet.

Pat on SOA

I think it made the rounds in the blogsphere, but I'll post it just the same: Pat Helland's TechTalk on TheServerSide.NET was published late last week. You know it's not your run-of-the mill interview when one of the questions is: "Why is Hooking Shit Together a more appropriate term?" and answers like " Sometimes I want to talk about cleavage a lot".

Seriously, one of the things I think was most important was Pat's answer to the question on integration challenges:

When you've had a couple apps that have been independently designed, they have different assumptions about the world. Different assumptions about inventory, different assumptions about customers; they probably don't even have the same snail mail address format. It's true that we can connect the bytes back and forth with IP and TCP today, but that doesn't do me anymore good than if I can pick up the telephone and talk to China, and if I'm speaking only English, and they're speaking only Mandarin, we still can't communicate very effectively. I can hear some cool sounds, but it doesn't help get work done.

The same thing's occurring when you bring applications together. The semantics of the interaction is a challenge. It's very, very difficult across an enterprise to get a common understanding of what a customer is, is a great example. Because, this half has got these fields, and that half has got those fields, and maybe you can get some common fields of understanding, but there's still all these differences that are in there.

A huge advantage that we have is XML. XML and XML Schema have allowed for at least the expression of what the data formats look like, even though that hasn't helped with all the semantics. But it's a good start. We don't have to worry about parsing, we don't have to worry about a lot of those details. But we still have a huge road ahead of us in terms of rationalizing how we understand the data across differently and independently designed applications.

This leads back to the SOA vs. SOP question. Being able to send a message is one challenge. It's an entirely different challege to get a consistent idea of "customer" or "order" across your enterprise systems. Both challenges are important but distinct and are important to different levels of architect.

I also thought his best practice comments were facinating:

I would argue that the biggest best practice is pragmatism. Don't do anything unless it makes financial sense.

Service-oriented architectures allow you to make whatever business choice fits. Because it talks about these being independent things, that can then hook together with others, using messaging, and using the sequences of messages that flow. That then takes the decisions of about what you bulldoze and you don't bulldoze and puts it back where it should be: in the pragmatic business guy's hands.

This leans heavy to towards Realist, but that doesn't make it any less true.

SOA vs. SOP

Don pointed out to Goran that the Indigo definition of "a service is simply a program that one interacts with via message exchanges." Goran pointed out that that definition "really doesn't highlight how it'll help a customer". I think part of the reason they are both right is that they are talking about different things. I would say Don is talking about Service Oriented Programming where Goran is talking about Service Oriented Architecture. This gets back to the levels of architecture that I blogged about. Platform tools like Indigo are components used in systems. I'm guessing the customer's Goran mentioned are at the system-of-system level for whom the messaging plumbing is below the abstraction level they care about. 
 
 Of course, SO* buzzwords are thrown about with such frequency these days it's hard to keep track of the difference.

SOA vs. OO in Business Process

Ram blogs on abstraction and Simon blogs on intimacy of SOA vs. OO. Here are my two cents on control and process of SOA vs. OO.

I came across this blog entry by Michael Santos who wants to stop the hype about web services. I forwarded his post to my entire team. I feel that it represents the typical old-school, 20th-century, industrial-revolution, application-centric mindset that we encounter regularly when discussing XML Web Services. He talks a lot about using binary protocols instead of XML because of performance. What's interesting is that his over-focus on performance leads down a path to tight coupling (or intimacy as Simon called it).

So, maybe you intend to keep your systems loosely coupled. I understand that. But let me ask you...Should they be loosely coupled in first place? Sometimes two systems are so tightly coupled that they should be just one system, to begin with. This usually happens in big companies, where political reasons force two groups to buy two solutions from two different vendors to solve two parts of the same indivisible problem that cannot be addressed separately. [Michael Santos : Stop the hype about webservices!]

The thing is, there is no such thing as the "indivisible problem" in the enterprise. Enterprises don't solve problems per se, they execute business processes. Developers tend to think in terms of nouns, which map nicely to objects, while business people tend to think in terms of verbs. For example, taking the canonical order processing scenario, the developer sees a single object - the order. Business people see the processes that surround that order - placing it, fulfilling it, paying for it. Typically, the developer sees these processes as methods: Order.Place(), Order.Fulfill(), Order.ProcessPayment(). However, these business processes don't represent things the business object is doing, rather things being done to the business object. It's a subtle difference, but it's very important.

In Ivar Jacobson's Object-Oriented Software Engineering, he talks about how over time objects tend to evolve to have methods that are only used in a single use case. He separated the concepts of the "entity" object - which represents a business object that has persistent state - and the "control" object  - which represents a process that modifies the state of one or more entities. (Note, control objects in this context are different from the controller object in the MVC pattern). In my experience, mapping use cases to control objects is a good first order approximation of your final system design.

However, implementing controls and entities with objects implies an intimate relationship as part of a single autonomous system. In practice, this is very difficult to maintain over time. First off, it's a bad model of reality. Going back to the order processing scenario, different departments and people are responsible for executing the "fulfill order" and "process order payment" business processes. The departments don't have an intimate relationship for good reasons, like trust and security. Those reasons should be reflected in the code. Secondly, business process changes much more often than business entities. You never know when you'll want to change a step, modify the order of steps or completely rethink the process in the face of market and / or technology changes. In other words, you want the connections between the processes and the entities to be loosely coupled. If you tightly couple everything together, then you'll need to change everything every time something changes. This leads to a stagnation that a customer of mine once compared this to carrying a big pile of cow manure - once you put it down, you don't want to pick it back up!

If you step back from the OO mindset, you can model controls and entities in terms services pretty effectively. At PDC, we discussed the idea of resource vs. activity oriented data and the idea of service-masters vs. service-agents. These ideas are very similar conceptually to the control / entity separation. Control objects become business process services (also known as service agents or sometimes emissaries). However, when using services instead of objects, you gain power and flexibility that you just don't get from the OO model.   

Intergrate vs. Interoperate

I found this interview with the BEA deputy CTO Benjamin Renaud via a news post on TSS.NET. BEA's position is summed up in the following quote:

"Microsoft will standardise at the protocol level, but they won't standardise at the API level," he said. "Customers are not that gullible. The real level where integration happens is at the programming level."

I like Ted's response to this:

We're sorry, Mr. Renaud, but integration isn't necessarily what web services are after--interoperability, in the form of loosely coupled components that know how to exchange messages of data, are the key to truly powerful web services. If you want programmatic integration, you have to standabrdize on programming platform and language, and that's not what web services were supposed to be for.

As to his complaint that Microsoft is engaged in a huge bait-and-switch, we believe he's either not putting enough faith in the development community to see this conspiracy, or else he's trying to cry foul over the fact that BEA has to compete with others in the Java space for the web services dollar, where Microsoft stands relatively alone.

However, I'm not sure what Ted's driving at with his interop vs. integration argument. To me, they seem to go hand in hand - two great tastes that taste great together. Web services are important for both. If I'm not integrating disparate systems, I probably don't care about interop that much.

Personally, I care much more about integration than interop. Even if I was going to build a system-of-systems all using only .NET technology, I would still use a service-oriented approach and implement those services using web service technology. The service-oriented approach allows me to be more flexible in the way I stitch my services together. Using web service technology allows me to leverage platform technology (ASMX, WSE, Indigo, etc) so I don't have to roll the whole stack myself. The fact that I get interop "for free" by using this approach is an extra bonus that I don't really care much about - at first. That built-in interop, even in this single-platform-which-never-happens-in-the-real-world scenario, helps make my systems future-resistant (nothing is future-proof). New customers, new partners, mergers - come what may, I have a better-than-average chance of being able to integrate it into my system. That gives me an real advantage ($$$) in the marketplace.

Of course, back in the real-world where you're creating a system-of-systems from a series of stand-alone systems that are all built with different platforms, interop is much more important. But that doesn't mean integration is any less important.

Reliable Syndication

After reading Sam's slides on Atom, Scoble posted three times about how syndication could evolve. Of course, Scoble has his Longhorn-colored glasses on. Dare pointed out that "The major problems with syndication today have little to do with the syndication format and more to do with it's associated technologies." I agree with Dare. IMO, the only thing that the ATOM syndication format has over RSS is a namespace declaration. I care about that because one of the "associated technologies" I care about is SOAP and the lack of an RSS namespace makes it hard to embed an RSS item inside a SOAP message.

I think Scoble should be asking how syndication will evolve in the face of Service Oriented Architecture in general, not Longhorn specifically. Granted, Indigo is going to make Longhorn a great platform for SOA. (If you check out the Longhorn Interoperability and Migration Guide, Chapter 2 is mostly dedicated to describing SOA.) But I think the real change to syndication is going to come from WS-ReliableMessaging. In order to truly evolve syndication, I think we need to break free of the synchronous polling model we have today. Polling only works in scenarios with a central syndication source (like a weblog). However, as the sources of syndicated content get to be more distributed (phones, P2P networks, etc) that polling model breaks down. I need to be able to send messages when things change without regard to network availability. With WS-RM, I can send messages and the infrastructure (i.e. Indigo) can take care of the ugly details of making sure the messages get delivered to their final destination.

Delivering Messages with SQL Service Broker

SQL Service Broker is probably the least known new feature of SQL Server "Yukon", but I can't wait for it. It makes messages a first class object in the database. If you've ever had multiple processes banging on your database or you've ever used a flag on a row to indicate if it's been processed or not, you want SQL Service Broker too.

While there is huge disagreement as to exactly what "Service-Oriented Architecture" is, I think there is some general consensus around the fact that it is an asynchronous message driven model rather than a synchronous RPC model. This means that the thread you receive a message on will never be the same thread that you process the message on. In fact, typically you will write the message to a persistent data store (hello, Yukon native XML support) in order to be handled by a thread in a different process and probably on a different machine. Today, kicking off the thread to handle the message is a pain in the ass. You probably want lots of threads across lots of machines to handle the incoming messages (assuming you're getting lots of incoming messages). In order to synchronize message processing across machines, you need a mechanism to make sure each incoming message is handled once and only once. Today, the closest solution is message queue technology like MSMQ (or MQSeries). However, since that's a different data store from where the data lives (i.e. the database), now you need two phase distributed transactions to get that done. However, since messaging is going to be such a huge piece of architecture going forward, it makes sense to have the concept of messages baked right into the database.

With Service Broker, when the message is received, it is placed into a service broker queue. (It'll probably get stored for archival and retry avoidance reasons, but that's a different blog entry.) Now I can have processes that, within the scope of a local transaction, receive the incoming message, make whatever data changes that message implies and send off any new messages. This is both more productive (manually handling local transactions for async processing is this kind of a scenario much easier than using serviced components) as well as more performant.