Nick's Flawed Vision of a Shared Integration Model

Of all the things you might say about Nick Malik, "thinks small" is not one of them. He takes on a significant percentage of the .NET developer community over the definition of Mort. He wants to get IT out of the applications business. He invents his own architecture TLA: SDA (aka Solution Domain Architecture). He's a man on a mission, no doubt. And for the most part, I'm with him 110% on his ideas.

However, when he starts going on about a shared global integration model, I start to wonder if he has both hands on the steering wheel, as it were.

Nick's been talking about this for over a year. As he points out, SaaS integration layer is the new vendor lock-in. One of the attractions of SaaS is that you could - theoretically, anyway - switch SaaS application providers easily which would drive said SaaS companies to constantly innovate. However, if the integration layers aren't compatible, the cost to switch goes up dramatically, leaving the customer locked-in to whatever SaaS company they initially bet on - even if that bet turns out to be bad.

OK, I'm with him so far. Not exactly breaking news here - we've seen the same integration issues inside the enterprise for decades. SaaS adds new wrinkles - if your ERP vendor goes belly-up, they can't take your data with them or worse sell it to your competition - but otherwise it sounds like the same old story to me.

However, where Nick loses me is when he recommends this solution:

"To overcome this conflict, it is imperative that we begin, now, to embark on a new approach. We need a single canonical mechanism for all enterprise app modules to integrate with each other. If done correctly, each enterprise will be able to pick and choose modules from different vendors and the integration will be smooth and relatively painless."

Yeah, and if a frog had wings, it wouldn't bump its ass when it hopped.[1] There are so many things wrong with this approach, I'm not sure I can get them all into a single web post.

First off, it won't, in fact, be done correctly - at least, not the first time. I realize everyone knocks MSFT for never getting an application right before version 3.0, but I believe it's actually systemic to the industry. Whatever you think you know about the problem to be solved, it's at best woefully incomplete and at worst wrong on all counts. So getting it right the first time is simply not possible. Getting it right the second time is very unlikely. It isn't until the third time that you really start to get a handle on the problem you're really trying to solve. Getting an effort like this off the ground in the first place would be a Herculean task. Keeping it together thru a couple of bad spec revisions would be impossible.

Meanwhile, the vendors aren't going to be waiting around twiddling their thumbs waiting for the specs to be done. We've seen efforts to unify multiple completing vendors around a single interoperable specification. By and large, those efforts (UNIX, CORBA, Java) have been failures. The technologies themselves haven't been failures, but the idea that there was going to be "relatively painless" portability or interoperability among different vendors never really materialized. If it didn't work for UNIX, CORBA or Java, what makes Nick think it will work for the significantly more complex concept of a shared global integration model? Not only more complex in terms of spec density, but the mind-boggling number of vendors in this space.

Nick is worried that either "we do this as a community or one vendor will do it and force it on the rest of us." But if you look at how specifications evolve, retroactive realization of defacto standards is the way the best standards get created. For example, I could argue that TCP was forced on us by the US Military, but I don't hear anyone complaining. I realize there's a big difference between having a vendor force a spec down our throat vs. a single big customer, but either way it's not designed by committee. Besides, if we do see get an enterprise integration standard forced on us, I don't believe it will be the vendors doing the forcing. If I were a betting man, I'd bet on Wal-Mart. Business leverage trumps IT leverage and Wal-Mart has more business leverage than anyone in this space these days.

BTW, would design-by-committee be an extreme example of BDUF? Do we really want to develop a this critical integration model using the same process that produced the XSD spec?

Finally, Nick thinks that this model will improve innovation, where I think it will have the exact opposite effect. Once you lay a standard in place, the way you innovate is to build proprietary extensions on top of that standard. However, by definition, these extensions aren't going to be interoperable. If someone has a good idea, others will copy it and eventually it will become a defacto standard.

A recent example of the process of defacto standardization is XMLHttpRequest. Microsoft created it in 1999 for IE 5, Mozilla copied it for their browser a couple of years later, followed by the other major browser vendors. Google innovated with it, Jesse James Garrett coined the term AJAX, everyone else started doing it and then finally - nearly a decade later and still counting - a standards body is getting around to putting their stamp of approval on it.

However, if you're worried about painless integration and not having something forced on you by some vendor, then you're not going to embrace these innovations - which means, you won't embrace any innovation. Well, there may be some innovation in the systems themselves that doesn't affect the interface, but once that interface is cast in stone, the amount of innovation will go way down. How do vendors differentiate themselves? There's only two ways: price and innovation. Take away innovation with standardization, and you're left with a race to the rock bottom price with no incentive to actually improve the products.

I get where Nick is going with this. He looks around our enterprise and sees duplication of effort and massive resources being spent on hooking shit together. It sure would be nice to spend those resources on something more useful to the bottom line. But standardizing - or worse legislating - the problem out of existence isn't going to work. What will? IMO, applying Nick's ideas of Free Code to interop code. If we're going to get IT out of the app business, can't we get out of the integration business at the same time?

[1] It's exceedingly rare that you get to quote Wayne's World or Raising Arizona in a discussion about Enterprise Architecture, much less both at the same time. Savor it.

Morning Coffee 134

• Bill de Hora responds to a few of my Durable and RESTful ideas. He points out that relying on a client-generated ID can be troublesome, and recommends using multiple identifiers - one created by the sender, one by the receiver and one representing the message exchange itself. However, the sender ID is vulnerable to client bugs & tampering as Bill points out, and neither the receiver ID nor the exchange ID can be used to determine if a given message is a duplicate. If you don't trust the sender, is it even possible to determine if a given message is a duplicate?
• Pablo Castro confirms that there are "practical limits" to what ADO.NET Data Services can do with respect to idempotence. Nothing in his post was surprising, though I hope it will be more explicitly called out in the final docs. Developers used to the comforting protection of a transaction may be in for a rude awakening.
• Dare Obasanjo has a great post comparing the new features in C# 3.0 to dynamic languages like IronPython. I believe many of the productivity aspects of dynamic languages have little to do with being dynamic.
• Pat Helland noodles on durability and messaging, two topics near and dear to my heart (probably from working with him for a couple of years). I'm not sure where he's going with this - his conclusion that "Basically, big, complex, and distributed system are big, complex, and distributed" isn't exactly ground-breaking. But his point that "durable" isn't a binary concept is worth more consideration. Also, his description of IMS only looking at the effects of a committed transaction is very similar to how web sites work, though obviously HTTP isn't durable so you can't make event horizon optimizations like IMS did.
• Tangentially related, Werner Vogels discusses the idea of eventually consistent distributed databases. Today, that's a problem mostly only Internet-scale sites like Amazon deal with. In the near future of continued data explosion + manycore, we'll all have to deal with it.
• Nick Malik argues that categorizing enterprise applications by lifecycle is much less useful than categorization based on organizational impact. He might also need a new chair.
• Jesus Rodriguez digs into one of SSB's new features in SQL 2008: conversation priorities.
• Arnon Rotem-Gal-Oz and Sam Gentile are mixing it up over the definition of SOA. Sam thinks SOA has to include business drivers and Arnon doesn't. I'm with Sam on this, defining "SOA" independently from "Applying SOA" seems pointless. Then again, rigorously defining SOA - much less arguing about said definition - seems like a waste of time in the first place IMHO.
• Wow, this guy Zed is mad at the Ruby community.
• Andrew Baron has 8 Reasons Why The TV Studios Will Die. Personally, I think reason #2 - Expendable Middle-Person - is the most important. If content producers can reach consumers directly, what value-add will the networks provide? (via United Hollywood)

Afternoon Coffee 123

• Morning Coffee is late this morning because we went for our Christmas portrait this morning and it took forever. The pictures turned out great though.
• Nick Malik finishes up his series on business operation models by covering the diversification model. Also, Nick's points about the synergy between a diversified model and the coordinated model are spot on. I happen to be a big fan of those models (aka the models with low standardization) which probably drives some of the  more my "unique" perspectives on SOA.
• Scott Guthrie starts out a new series and future technology, this time it's ASP.NET MVC Framework that gets the series treatment. The first entry in the series is a general overview. I wonder why there's no cool code name for the MVC framework? Whatever it's named, I like the auto routing and action rules - it seems very Rails-inspired.
• Over the weekend, Don Box points out that the REST authentication story "blows chunks". I've recently given up on the reliable part of the original "Secure, Reliable, Transacted Web Services" vision - and I never believed the transacted part. Security, on the other hand, is the one part of that original vision that has worked out IMO. My experience with the WS-* security stack has been pretty good, though Dare Obasanjo thinks that OpenID and OAuth are the final nail in the WS-* coffin.
• Speaking of Dare, he goes on to say WS-* is to REST as Theory is to Practice. He makes the point that "The only times I encounter someone with good things to say about WS-* is if it is their job to pimp these technologies or they have already “invested” in WS-* and want to defend that investment." I gave up pimping evangelizing technology a while back and I don't want to be in the position of defending a bad investment, so I'm spending lots of time looking at REST.
• Jesus Rodriguez takes a look at the Managed Services Engine and comes away excited. Jesus is a self-described "strong believer" in SOA governance. I'm a self-described strong disbeliever in SOA governance, so MSE sounds like more of the Worst of Both Worlds to me.
• A little light reading: I pulled Applied Cryptography and A New Kind of Science out of my garage last weekend. Plus my copies of RESTful Web Services and Programming Erlang just arrived yesterday.

The Importance of Idempotence

Every organization has some operations or processes that have to happen Exactly Once. Your employer needs to make sure they issue your paycheck exactly once. Your bank needs to make sure that paycheck is deposited in your account exactly once. Exactly Once isn't something that just "traditional" enterprises like banks care about. Google needs to make sure your AdSense check is issued exactly once. Amazon needs to make sure your credit card is charged exactly once. Especially when there's money involved, the company wants to make sure it gets handled correctly - Exactly Once.

In application (aka siloed) development, transactions are often used to ensure stuff happens Exactly Once, to good effect. But how do we guarantee Exactly Once now that we're connecting systems together? Given how well transactions work inside applications, it's not surprising that early attempts to guarantee Exactly Once between systems relied on distributed transactions, this time to not-so-good effect. Pat Helland summarized the problems with distributed transactions this way:

"The two-phase commit protocol will ensure perfect consistency given infinite time.  I say that because it will wait and wait and wait until the transaction is resolved and then provide perfect consistency.   Of course, while partitioned and waiting, arbitrary swaths of the application's database may be locked up rendering the application unusable.  For this reason, I've frequently referred to the two phase commit protocol as the "Anti-Availability Protocol". "
Pat Helland, SOA and Newton's Universe

So now we're faced with a dilemma. Transactions are, for all practical purposes, unusable to ensure Exactly Once processing between connected systems. And yet, the business requirement to ensure Exactly Once hasn't gone away. We need another way.

The first fallacy of distributed computing is that the network is reliable. It's usually works, but usually isn't a guarantee. If I send a message to a remote system but don't get an acknowledgement, which got lost: the original message or the ack? There's no way to know, so I have to send the message again. But if I send it again and it's the ack that got lost, then the target system will receive the message multiple times.

Since the network is not reliable, there's no way to guarantee that a message will be delivered exactly once. The best we can go is ensure a message will be delivered at least once. However, that implies the target system will receive some messages multiple times. If we need to ensure Exactly Once, we need to make sure the target system won't duplicate the work if it receives duplicate messages. In other words, we need the target system to be idempotent. 

"In computer science, the term idempotent is used to describe method or subroutine calls which can safely be called multiple times, as invoking the procedure a single time or multiple times results in the system maintaining the same state i.e. after the method call all variables have the same value as they did before.

Example: Looking up some customer's name and address are typically idempotent, since the system will not change state based on this. However, placing an order for a car for the customer is not, since running the method/call several times will lead to several orders being placed, and therefore the state of the system being changed to reflect this."
Wikipedia, Idempotence (Computer Science)

Or more succinctly:

"Idempotent Means It’s OK to Arrive Multiple Times"
Pat Helland (again)

I can't overstate the importance of designing your cross-system communication to be idempotent. If you care about ensuring Exactly Once, each step of your process has to be either transactional or idempotent, or you'll be screwed. It's interesting to note that you have to be transactional *OR* idempotent, but not both. You can chain together multiple steps in long business process, across multiple disparate systems, but as long as each step is either transactional or idempotent, you can guarantee Exactly Once across the entire process. In other words:

Transactional/Exactly Once == Idempotent/At Least Once

This implies that you can substitute an idempotent operation for a transactional operation, and still ensure Exactly Once.

Let's look at an example. Typically you ensure Exactly Once processing with MSMQ by receiving messages within the scope of a transaction along with whatever other work you're doing. But what if you can't use a transactional receive, say because it's a remote queue? What would an idempotent equivalent for transactional receive look like?

How about:

1. Peek a message from the remote queue
2. Insert the message into the target system database, using the unique MSMQ Message ID as the primary key
3. Remove the message from the queue by ID

Each of those steps is idempotent. Peek is a read, which is naturally idempotent. Inserting the message into the database is idempotent, since we use the message ID as the primary key. As long as that ID is unique, we can never insert it into the database more than once. Finally, removing a message based on it's unique ID is also naturally idempotent. Once the message is in the target system database, we can use traditional transactions to ensure it gets processed Exactly Once.

So we took a single transactional operation and turned it into a series of idempotent steps. Both ensure each message is processed Exactly Once. Given the choice, I'd rather write the transactional operation - it's much less code since we're we can use existing infrastructure - aka the distributed transaction coordinator. But if the transactional infrastructure isn't available, I'd rather write multiple idempotent steps and ensure Exactly Once rather than risk losing or duplicating messages.

I've got more on this topic, but in the meantime think about this: How do you think durable messaging infrastructure like MSMQ ensures exactly once delivery? You can use that pattern, even if you're not using durable messaging infrastructure.

Morning Coffee 122

• Sorry for the posting lag. Had a few technical difficulties around here. In the process of moving hosts, so expect more glitches.
• My talk at the p&p Summit on Monday went really well. At least, it felt good and the applause at the end felt genuine. I recorded the audio on my laptop, so I'll be posting a Silverlight version as soon as I figure out how to adjust the levels so their somewhat consistent. Paraesthesia and #2872 have reactions.
• Speaking of the p&p Summit, Scott Hanselman posted his ASP.NET MVC demo from his talk. Said ASP.NET MVC bits aren't available yet, so you can't, you know, run the demo for yourself. But at least you can review what the ASP.NET MVC code will look like.
• I stopped by the SOA/BPM conference last week and saw Jon, Sam and Jesus among others. Spent quite a bit of time talking to Sam and his Neudesic colleagues about this "physically distributed/logically centralized" approach that I think is hogwash. It sounds to me like Neudesic approach is really federated not centralized, though I'm not sure David Pallmann would agree. Federated makes much more sense to me than centralized.
• Nick Malik continues his series on SOA Business Operations Model. I especially like his point that this isn't a series of choices, you need to "look at your company and try to understand which model the business has selected. "
• The first CTP of PowerShell 2.0 is out! Check out what's new on the PowerShell team blog and Jeffrey Snover's TechEd Presentation. (via Sam Gentile)
• Soma announced updates to VC++ coming next year, including TR1 support and a "major" MFC upgrade to support creating native apps that look like Office, IE or VS. I get supporting TR1, but the idea that people are clamoring for MFC updates is kinda surprising. Many years ago when I first came to MSFT, a friend asked "But don't you hate Microsoft?" to which I responded "No, I just hate MFC". Obviously, not everyone agrees with that sentiment.
• Steve Vinoski thinks there's no hope for IT. Funny, I keep agreeing with Steve's overall point but disagreeing with his reasoning. I still don't buy the serendipity argument. I like compiled languages. And I think he's overstating the amount of "real, useful guidance" for REST floating around. Basically, there's "the book".
• In widely reported news, Windows Live launched their next generation services. Don't bother with the press release, just go to the new WL home page.
• Speaking of WL, Dare Obasanjo points to the Live Data Interactive SDK page where you can experiment with the WL Contacts REST API. It gives you a good sense of how the Web3S protocol works. Pretty well, IMO. However, how come WL Contacts Schema doesn't include some type of update timestamp for sync purposes? If you wanted to build say a Outlook <--> WL Contacts sync engine, you'd have to download the entire address book and grovel thru it for changes every sync.
• Speaking of Web3S, I'd love to see some info on how one might implement a service using Web3S. Yaron Goland positions Web3S as an alternative to APP that WL developed because they "couldn't make APP work in any sane way for our scenarios". I'm sure other folks have similar scenarios.

Morning Coffee 121

• My daughter had her tonsils & adenoids out on yesterday. It was a routine procedure and it went by-the-numbers, but any parent will tell you it's hard to see your kid in a hospital bed.
• Given the previous bullet, I'm not at the SOA/BPM conference for the big announcement. Don't worry, there's lots of other folks covering the news.
• It was a crappy sports weekend in the Pierson house. Va Tech snatched defeat from the jaws of victory, Southern Cal never led at Oregon, the Capitals lost twice, and the Redskins got blown out by the Pats. At least the Caps won big yesterday in Toronto.
• Speaking of the Capitals, Peter Bondra retired Monday. I still think it's a travesty that he didn't spend his whole career in DC, but I've made my peace with it.
• Nick Malik has a great series on business operations models and how they apply to SOA. Regular readers should be unsurprised that I favor low standardization, though I can see the value of high integration. That makes the Coordinated Operating Model my fav, though I can see the benefit of the Diversified Model as well. I can't wait to read what Nick has to say on changing models.
• Speaking of Nick, I'm doing a roundtable with him on "Making SOA Work in the Enterprise" @ the Strategic Architect Forum. Should be fun. Sorry for the lack of linkage on this, but it's an invite-only event.
• Jezz Santos has a new series of white papers on building software factories. First up "Packaging with Visual Studio 2005"
• Aaron Skonnard has a new whitepaper on using the WCF LOB Adapter SDK with BTS 2006 R2. I've been building one of these things recently, so I'm looking forward to checking that out. (via Sam Gentile)
• Tim Ewald looks at Resource Oriented Architecture (when did ROA become a TLA?) and wonders "what if your problem domain is more focused on processes than data?" I wonder that all the time. (via Jesus Rodriguez)
• It's not just durable messaging - Libor Soucek also disagrees with my opinions on centralized control. I agree 100% with Libor that centralized management would make operation's lives "much, MUCH easier" as he puts it. However, that doesn't make it feasible at any significant scale. Furthermore, I wouldn't describe an approach that requires that "all services adopt [the] same common management interface" as "pragmatic". Frankly, just the opposite.

The Worst of Both Worlds

David Pallmann of Neudesic responded to my comment that "Physically distributed but logically centralized" didn't make any sense to me at all:

What exactly does this mean? To some this may sound like a contradiction.

This simply means that a bus is physically more like the point-to-point architecture (spread out, no hub) but functionally more like the hub-and-spoke architecture (pub-sub messaging, centralized configuration and activity tracking, easy change management).

Unfortunately, I wasn't confused about the seeming contradictory nature of these concepts. In other words, I understand the "what" and "how" of David's physically distributed/logically centralized approach.

I don't understand the "why". As in, "why would you want to do this?" or "why do you think this would work at any significant scale?".

If we check out Neudesic's page on their ESB product (which David pointed me to) we find the following blurb:

Centralized Management
The distributed nature of service oriented programming can create a management nightmare. Neuron·ESB supports this distributed architecture while simultaneously centralizing monitoring and configuration.

SOA's "distributed nature" is it's primary strength. SOA's not primarily about standards or ease-of-connectivity - though those obviously play a role. It's about enabling decentralized decision making. Since you can't be both centralized and decentralized, enforcing centralized management basically negates SOA's primary strength. This seems like the worst of both worlds to me. All the hassle of distributed decision making combined with all the hassle of centralized management.

Yes, decentralized decision making can create a management nightmare. Personally, a management nightmare is much more attractive anything centralized approaches have ever delivered in the IT industry.

Dare Obasanjo recently wrote "If You Fight the Web, You Will Lose". He was talking about the Web as a Platform, but it's good general advice. Can you imagine applying the marketing blurb above to the Internet at large?

Centralized Management
The distributed nature of service oriented programming the Internet can create a management nightmare. Neuron·ESB supports this distributed architecture while simultaneously centralizing monitoring and configuration.

If the Internet can somehow get by without centralized management, why can't you?

Throwing Gasoline on the Fire

Steve Vinoski has raised a bit of a flame war by admitting he has lost the ESB religion. Given that I've never been a fan of ESB's anyway, there's a lot there that I agree with. In particular I liked the description of "magical framework" middleware, blaming enterprise architects for driving ESB's as the "single integration architecture" even though a single *anything* in the enterprise is untenable and his point that flexibility means you don't do any one thing particularly well.

However, Steve goes on to bash compiled languages and WS-* while suggesting the One True Integration Strategy™ is REST + <insert your favorite dynamic language here>, then acts surprised that the conversation denigrates into "us vs. them". When you start by saying that compiled language proponents "natter on pointlessly", I think you lose your right to later lament the depreciating level of conversation .

All programming languages provide their own unique model of the execution environment.  Dynamic languages have a very different model than compiled languages. Arguing that this or that model is better for everyone, everywhere, in all circumstances seems unbelievably naive and arrogant at the same time.

On the other hand, I do agree with Steve's point that most developers only know a single programming language, to their detriment. One language developers often miss a better solution because their language of choice doesn't provide the right semantics to solve the problem at hand. Developers could do a lot worse than learn a new language. And I don't mean a C# developer should learn VB.

The most pressing example of picking the right language for the right problem today is multi-threading. Most languages - including dynamic languages - have shitty concurrency semantics. If you're building an app to take advantage of many-core processing, "mainstream" apps like C#, Java and Ruby won't help you much. But we're starting to see languages with native concurrency semantics like Erlang. Erlang is dynamically typed, but that's not what makes it interesting. It's interesting because of it's native primitives for spawning tasks. I don't see why you couldn't add similar primitives for task spawning to a compiled functional language like F#.

As for REST vs. SOAP/WS-*, I thought it was interesting that Steve provided no rationale whatsoever for why you should avoid them. The more I listen to this pissing match debate, the more I think the various proponents are arguing over unimportant syntactical details when the semantics are basically the same. SOAP is just a way to add metadata to an XML message, much as HTTP headers are. WS-* provides a set of optional message-level capabilities for handling cross-cutting concerns like security. Past that, are the models really that different? Nope.

For system integration scenarios like Steve is talking about, I'm not sure how important any of the WS-* capabilities are. Security? I can get that at the transport layer (aka HTTPS). Reliable Messaging? If I do request/response (which REST excels at), I don't need RM. Transactions? Are you kidding me? Frankly, the only capability you really need in this scenario is idempotence, and neither REST or SOAP provides any standard mechanism to achieve that. (more on that in a later post)

I understand that some vendors are taking the WS-* specs and building out huge centralized infrastructure products and calling them ESBs. I think Steve is primarily raging against that, and on that point I agree 100%. But Steve sounds like he's traded one religion for another - "Born Again REST". For me, picking the right tool for the job implies much less fanaticism than Steve displays in his recent posts.

The One Business Case for Integration

Nick Malik lays out what he thinks are the four business cases for integration:

Assume we succeeded, and our systems are all optimally integrated.  What has changed? 

1. We have better business intelligence.  We have better understanding of our customers, our partners, our products, and our business.  And from that understanding, we make better decisions.  Those decisions are made in a federated manner using self-apparent information.
2. We have end-to-end business processes that cross multiple systems, multiple roles, multiple geographies, and multiple data stores, all aware of and supporting the needs of the customer.
3. We have end-to-end awareness of the metrics that drive both dissatisfaction and cost, and we can take that knowledge and apply it to making our business better.
4. We have a more efficient enterprise, more able to grow to a larger size, at an accelerated rate, and still respond with agility to changing business opportunities.

I put to you that, in fact, we only have one business case for integration: better business intelligence. The other reasons Nick lists are either redundant or not as important to the business - at least in the general case - as you might think.

First off, #3 from Nick's list sounds suspiciously like #1. If there's a difference between "better understanding driving better decisions" and "applying awareness of metrics to making our business better", I don't know what it is. We'll send one of them off to the Dept. of Redundancy Dept. and be done with it.

Second, I don't think the business cares that IT has multiple systems or multiple data stores. If the business could run on one big centralized system that could meet the needs of the customer (aka the ERP fantasy), they'd be fine with that. The fact that realities of modern enterprise IT require splitting up capabilities across many systems is an implementation detail that frankly isn't a concern of the business.

Besides, what's the business benefit here? News flash: the enterprise already has end-to-end business processes that cross multiple systems, multiple roles, blah blah blah. They're just not automated end-to-end. Does the business care that their not automated? Not a bit. Sure, they care about processes are slow, costly and error-prone, which manual processes tend to be. But it's those negative characteristics that the business cares about, not integration. Besides, making processes quick, cheap and error-free sounds a lot like making them efficient. In other words, more work for the Dept. of Redundancy Dept.

Finally, I don't think efficiency and agility is as important to the enterprise as Nick makes it out to be. I mean, the enterprise will say it cares about efficiency - especially in front of the stock holders. But when it comes to putting it's money where it's mouth is, the enterprise doesn't, more often than not. Think about how success is measured in the typical IT project. Is efficiency one of the criteria for judging success? Not really. Will your project stakeholders let you run over budget and ship a few months late, just to improve efficiency? Probably not, unless that efficiency gain is both demonstrable and dramatic.

Of course, there are certainly specific examples where a automation or efficiency business case for integration can be made. For example, if replacing a specific manual process with an automated one has a large and measurable ROI, the business will likely be interested in making that investment. If you have a certain process that you do over and over that's core to the business, the business will probably be interested in optimizing the frak out of it. For example, I would guess a delivery company like UPS or FedEx has spent a lot of time and money on optimizing their delivery processes.

But what it sounds like Nick's talking about here is making a general case for making all our systems "optimally integrated". Given that our current systems aren't, this would take significant time and money. Yet the tangible benefit to the business is at best nebulous. Nick thinks improved integration will allow the business to "respond with agility to changing business opportunities." He's probably right. But how do you quantify this agility? How much will we save in the future for what we're spending today? For the general case, the answer is "it depends". It's really hard to fund a project when it's projected ROI is "it depends" .

However, business intelligence is a no brainer for the enterprise to invest in. Giving decision makers better and more up-to-date information, that's a tangible benefit that the organization can quantify now. If you can quantify the value of a project, you've got the start of a budget. Of course, all that juicy data is smeared across a variety of systems, which means integration. Again, the enterprise doesn't really care about said multiple systems or integration, but they care about the outcome.

Nick recommends to SOA folks that "if you aren't already working with your BI team, pick up the phone. Their mature processes and practices are able to address many of your issues, and the natural synergy between BI and SOA can make them a strong ally in the fight for a better, faster, cheaper, and more intelligent enterprise." Good advice. Otherwise, selling integration to the business isn't much different than selling them SOA. In other words, don't sell it - just do it.

Morning Coffee 113

• I'm in Chicago today and tomorrow for a reunion of sorts. In my last job, I managed a group of external architects called the Microsoft Architecture Advisory Board (aka the MAAB). We discontinued the program a while back, but the core of the group found the program valuable enough they have continued to meet anyway. I found the MAAB meetings incredibly valuable and insightful, so I'm really excited to be invited to continue my involvement with the group.
• I picked up Bioshock Tuesday (Circuit City had it on sale) on my way to my bi-weekly campus excursion. My meetings were over around 2pm so I headed home early, expecting to surprise the kids. But Jules had decided to skip naps and go shopping with them. Her cell phone was dead, so I ended up at home with a couple of hours to myself and a brand new copy of Bioshock. Wow, is that a good game. Certainly deserving of the amazingly good reviews it's garnered.
• Speaking of reviews, this transparently biased review of Bioshock over at Sony Defense Farce Force is frakking hilarious. Somehow, I doubt their dubious review will stem the tidal wave of Bioshock's well-deserved hype. Can't wait to read their Halo 3 review.
• Pat Helland writes at length on master-master replication. I reformated it into PDF so I could read it - the large images were messing up the text flow on my system. As usual for Pat, there's gold in that thar post. His thoughts on DAGs of versions and vector clocks as identifiers are very exciting. However, I think he glosses over the importance of declarative merging. I would think programmatic merge would likely be non-deterministic across nodes. If so, wouldn't you end up with two documents with the same vector-clock identifier by different data?
• Joe McKendrick points to a few people who predict the term "service-oriented" will eventually be subsumed under the general heading of "architecture". Not to brag, but I made that exact same prediction almost three years ago.
• Erik Johnson thinks that SOA 2.0 centers on transformational patterns. The idea (I think) is that if systems "understand each other more deeply", then we can build a "smarter stack" and design apps via new constructs to promote agility and simplicity. Personally, I'm skeptical that we can define unambiguously system semantics except in the simplest scenarios, but Erik talks about using "graph transformation mathematics" to encode semantics. I don't know anything about graph transformation mathematics, but at least Erik has progressed beyond hand waving to describing the "what". Here's looking forward to the "how".
• New dad Clemens Vasters somehow finds time to implement an XML-RPC binding for WCF 3.5. I was encouraged that it didn't require any custom attributes or extensions at the programmer level. Of course, XML-RPC fits semantically into WCF's interface based service model, so it shouldn't be a huge surprise that it didn't require any custom extensions. But did it need WCF 3.5? Would this work if recompiled against the 3.0 assemblies?
• Phil Haack writes a long post on Duck Typing. VB9 originally supported duck typing - the feature was called Dynamic Interfaces - when it was first announced, but it was subsequently cut. I was really looking forward to that feature. Between it and XML Literals, VB9 was really stepping out of C#'s shadow. I guess it still is, even without dynamic interfaces.
• Since I've been doing some LINQ to XML work lately, I decided to go back and re-write my code in VB9 using XML literals. While XML literals are nice, I don't think they're a must have. First, LINQ to XML has a nice fluent interface, so the literals don't give you that much cleaner code (though you do avoid writing XElement and XAttribute over and over.) Second, I find VB9's template syntax (like ASP <%= expression %>) clunky to work with, especially in nested templates. Finally, I like the namespace support of XNames better. As far as I can tell, VB9 defines namespaces with xmlns attributes just like XML does. So I'm not dying for literal XML support in a future version of C#. How about you?

Morning Coffee 110

• Monday @ Gamefest, the XNA team announced XNA Game Studio 2.0. The two big new things are support for the entire VS product line (1.0 only works on VC# Express) and the addition of networking APIs. Let's Kill Dave has a good wrapup of the announcements from Gamefest Day One.
• Speaking of Xbox 360, I played thru the demos of Stranglehold and Bioshock. Two thumbs up on both. It's gonna be an expensive year for Xbox gamers.
• Mark Cuban noodles on taking your house public. "Why not create a market or exchange where homeowners can sell equity in their homes?" I've thought about this myself from time to time. However, Mark thinks making it happen would "probably take the country's biggest banks working together". I wonder if there's a more Web 2.0 social lending approach that would work better.
• Jeff Atwood calls virtualization as "the next great frontier for computer security". I agree 100%. But I don't think the action is going to be in "full-machine" virtualization like Virtual PC. Rather, it's going to be sandbox virtualization. Jeff mentions GreenBorder (now part of Google) but it's not the only solution. Some time ago, Microsoft acquired SoftGrid which uses sandbox virtualization for application deployment, but using SystemGuard for security sandboxing seems like a logical step.
• The WCF LOB Adapter SDK has released. Sonu Arora has the details. As part of the Integration team @ MSIT, I have a feeling we're going to become fairly familiar with this technology. (via Jesus Rodriguez).
• Speaking of Jesus, he thinks the six new SCA4SOA committees are "going to help". Why? Because inventing technology in committee has turned out so well in the past?
• John deVadoss cements BPM's fad du jour status by contrasting "big" BPM and "little" BPM. It's fairly obvious to me that big *anything* just doesn't work in the enterprise. But I worry that little *anything* doesn't work that well either. So how long until someone (probably Nick) starts arguing for "middle out" BPM?
• David Bressler wonders "What is it about registries that everyone thinks is a panacea for all things SOA?" Amen, Brother! Joe McKendrick claims it's required for governance, but then gets to what I think is the *real* reason for focus on registries: the "registry is a tangible offering" that vendors can sell. Just because it's productizable doesn't mean you need it.
• Hartmut Wilms responds to my retire the tenets post, but he seems to contradict himself. On the one hand, he suggests that "the four tenets just expressed, what “almost” everybody outside the MS world knew already". But then he goes on to dispute that the SO paradigm shift has even occurred! Hartmut, I'll grant you that WCF (among other similar stacks) are way too focused on "you write the classes, we'll handle the contracts and messages". On the other hand, if you don't provide a productive interface that most everyone can pick up and run with, the technology won't get adopted in the first place.

Retire the Tenets

John Heintz and I continue to be in mostly violent agreement. It's kinda like me saying "You da architect! Look at my massive scale EAI Mashup!" and having him respond "No, you da architect! The SOA tenets drive me bonkers!" Makes you wonder what would happen after a few beers. What's the architect version of Tastes Great, Less Filling? [1]

Speaking of the tenets, John gives them a good shredding:

Tenet 1: Boundaries are Explicit
(Sure, but isn't everything? Ok, so SQL based integration strategies don't fall into this category. How do I build a good boundary? What will version better? What has a lower barrier to mashup/integration?)

Tenet 2: Services are Autonomous
(Right. This is a great goal, but provides no guidance or boundaries to achieve it.)

Tenet 3: Services share schema and contract, not class
(So do all of my OO programs with interface and classes. What is different from OO design that makes SOA something else?)

Tenet 4: Service compatibility is based upon policy
(This is a good start: the types and scope of policy can shape an architecture. The policies are the constraints in a system. There not really defined though, just a statement that they should be there.)

Ah, I feel better getting that out.

As John points out, the four tenets aren't particularly useful as guidance. They're too high level (like Mt. Rainier high) to be really actionable. They're like knowing a pattern's name but not understanding how and when to use the actual pattern. However, I don't think the tenets were ever intended to be guidance. Instead, they were used to shift the conversation on how to build distributed applications just as Microsoft was introducing the new distributed application stack @ PDC03.

John's response to the first tenet makes it sound like having explicit boundaries is obvious. And today, maybe it is. But back in 2003, mainstream platforms typically used a distributed object approach to building distributed apps. Distributed objects were widely implemented and fairly well understood. You created an object like normal, but the underlying platform would create the actual object on a remote machine. You'd call functions on your local proxy and the platform would marshal the call across the network to the real object. The network hop would still be there, but the platform abstracted away the mechanics of making it. Examples of distributed object platforms include CORBA via IOR, Java RMI, COM via DCOM and .NET Remoting.

The (now well documented and understood) problem with this approach is that distributed objects can't be designed like other objects. For performance reasons, distributed objects have to have what Martin Fowler called a "coarse-grained interface", a design which sacrifices flexibility and extensibility in return for minimizing the number of cross-network calls. Because the network overhead can't be abstracted away, distributed objects are a very leaky abstraction.

So in 2003, Indigo folks came along and basically said "You know the distributed object paradigm? The one we've been shipping in our platform since 1996? Yeah, turns out we think that's the wrong approach." Go back and check out this interview with Don Box from early 2004. The interviewer asks Don if WCF will "declare the death of distributed objects". Don hems and haws at first, saying "that's probably too strong of a statement" but then later says that the "contract, protocol, messaging oriented style will win out" over distributed objects because of natural selection.

The tenets, IMHO, were really designed to help the Windows developer community wrap their heads around some of the implications of messaging and service orientation. These ideas weren't really new - the four tenets apply to EDI, which has been around for decades. But for a generation of Windows developers who had cut their teeth on DCOM, MTS and VB, it was a significant paradigm shift.

These days, with the tenets going on four years old, the conversation has shifted. Platform vendors are falling over themselves to ship service/messaging stacks like WCF and most developers are looking to these stacks for the next systems they build. Did the tenets do that? In part, I think. Mainstream adoption of RSS was probably the single biggest driver of this paradigm shift, but the tenets certainly helped. Either way, now that service orientation is mainstream, I would say that the tenets' job is done and it's time to retire them. Once you accept the service-oriented paradigm, what further guidance do the tenets provide? Not much, if any.

[1] Not that you would catch me drinking Miller Lite. Ever.

Morning Coffee

• Libor Soucek continues our conversation about durable messaging. We still don't agree, but he says he "fine" with durable messaging. He does go out of his way to differentiate between *enterprise* and *supporting* systems. But when you're building connected systems, does that differentiation still matter?
• After taking a few months off, John deVadoss is back at the blog. Check out his Big SOA/Little SOA post. I especially like his snowball analogy "How do you build a big snowball? You start with a small snowball.") though he's also on this "middle out" bandwagon. Do we really believe "middle out" works, or are we just saying it because we know top down and bottom up don't? And John: You're welcome!
• Anyone coming to the Microsoft SOA & Business Process Conference this fall? Maybe we can have a shindig / blogger dinner / unconference / something?
• Remus Rusanu writes about SSB's dynamic routing. One of the (many) cool things about SSB is that all the addressing is logical, not physical. Routing is what binds logical addresses to physical addresses, and it's extensible.
• Martin Fowler discusses the value of sticking to one language. I agree with his points about large frameworks being as difficult to learn as a new language. I've said for a long time "If you build a framework, build tools to make it easy to use your framework". Language is obviously a core example of a tool. Another interesting point Martin makes is the traditional "intimate relationship" between scripting languages and C, but that the rise of JVM & CLR makes them impossible to ignore. Does the need to play well in a managed environment hinder a C based language like Ruby when compared to a natively managed scripting language like Powershell? Finally, Martin's "jigger of 80 proof ugliness" quote made me laugh.
• Politics 2.0 Watch: EJ Dionne says that DailyKos is doing for Democrats what Rush Limbaugh did for Republicans almost twenty years ago: mobilization. Josh Marshall points out that "what's happening today is vastly more participatory and distributed...than anything happening back then."

Where Have All the SOA Mashups Gone?

John Heintz responded to my serendipitous reuse post. Nice to see I misunderstood his opinions about how easy RESTful systems are to integrate:

I didn't mean to imply that building RESTful system would lead to magical integration without any hard work. I can see how that came across in my post, and I guess I got the reaction I asked for ;)

I get the feeling that John would be a good guy to have a beer with.

John spends most of his post writing about the SOA in the Real World book. I've flipped thru it and I'm familiar with the model (it is my old team after all) but I haven't read it so I don't really want to comment about the book specifically. But there were two things John mentioned that I did want to comment on.

First, at the end of his post John writes:

Can some of the constraints of REST be applied to SOA? Absolutely. I think an asynchronous, message-passing architecture with a uniform interface would be astoundingly interesting! I'm not the only one: see MEST, AMPQ, and Erlang.

This goes back to a REST question I asked two months ago: is it still REST if you don't use HTTP? I'm guessing John would say yes.

I might be going out on a limb here, I'll bet the core of John's problem with SOA is how toolkits like WCF all but force you to build RPC style services that can easily be modeled as method calls. That's certainly one of my problems with SOA. Tim Ewald said it best:

It's depressing to think that SOAP started just about 10 years ago and that now that everything is said and done, we built RPC again. I know SOAP is really an XML messaging protocol, you can do oneway async stuff, etc, etc, but let's face it. The tools make the technology and the tools (and the examples and the advice you get) point at RPC. And we know what the problems with RPC are. If you want to build something that is genuinely loosely-coupled, RPC is a pretty hard path to take.

If SOA == RPC and REST == loosely coupled messages, then I'll start growing dreadlocks right now. Frankly, as Tim says, I think it's a problem with the tools (I'm looking at you WCF) and not the underlying architecture, but how many people can distinguish the architecture from the tools? Not many, I'm afraid.

Second, John asks an interesting question:

Where are the SOA mashups?

That's easy! They're inside the firewall where you can't see them! ;)

Seriously, I'm not sure about "SOA" mashups, but I'm working with what you might call a huge "enterprise" mashup system inside Microsoft. Our Enterprise Data Integration Services push around massive amounts of data to downstream systems. There are over fifty datasets in production, each with scores of tables, millions of rows and hundreds of subscribing systems. One example, our Products dataset, has over 100 tables and nearly 300 subscribing systems.

Is it "service oriented"? No, but then again it was originally developed ten years ago on SQL 6.5. But is it a mashup? Is it an "application that combines content from more than one source into an integrated experience"? Yep. Is it easy to work with? No, but guess why I'm involved? We're looking at ways to "modernize" the system. Am I going to build RPC style services as part of this modernization? Hell, no.

So John, am I right or wrong about that beer?

Is Serendipity the Heart of the WS-*/REST Debate?

Thanks to Technorati, I found this post by John Heintz. He's checking out John Evdemon's e-book on SOA and has a problem with this overview:

SOA is an architectural approach to creating systems built from autonomous services. With SOA, integration becomes forethought rather than afterthought. This book introduces a set of architectural capabilities, and explores them in subsequent chapters.

To which John H. responds:

I, for one, would rather build on an architecture that promotes integration as an afterthought, so I don't have to think about it before hand!!!

Yeah, I'd rather not have to think about integration before hand either. On the other hand, I want integration that actually works. It sounds like John H. is suggesting here that REST somehow eliminates the need to consider integration up front. It doesn't. Consider this: if you're building a Web 2.0 site then you are expected to expose everything in your site via APP, RSS and/or RESTful POX services. In other words, the Web 2.0 community expects you to have the forethought to enable integration. If you don't, Marc Canter will call you out in front of Bill Gates and Tim O'Reilly. 

This integration by afterthought approach seems to be big among RESTifarians. John H. links to a REST discussion post by Nick Gall advocating the principle of generality, "unexpected reuse" and "design for serendipity". Money quote:

The Internet and the Web are paradigms of Serendipity-Oriented Architectures. Why? Largely because of their simple generality. It is my belief that generality is one of the major enablers of serendipity. So here I immodestly offer Gall's General Principle of Serendipity: "Just as generality of knowledge is the key to serendipitous discovery, generality of purpose is the key to serendipitous (re)use."

Serendipity means "the accidental discovery of something pleasant, valuable, or useful". "Serendipitous reuse" sounds an awful lot like accidental reuse. Most enterprises have been there, done that and have nothing to show for their efforts or $$$ except the team t-shirt. Yet Tim Berners-Lee believes "Unexpected reuse is the value of the web" and Roy Fielding tells us to "Engineer for serendipity". What gives?

First off, enterprises aren't interested in unexpected or serendipitous reuse. They want their reuse to be systematic and predictable. The likelihood of serendipitous reuse is directly related to the number of potential reusers. But the number of potential reusers inside the enterprise is dramatically smaller than out on the public Internet. That brings the chance for serendipitous reuse inside the enterprise to nearly zero.

Second, enterprise systems aren't exactly known for their "simple generality". If Nick's right that "generality of purpose is the key to serendipitous (re)use", then enterprises might as well give up on serendipitous reuse right now. As I said last year, it's a question of context. Context is specifics, the opposite of generality. Different business units have different business practices, different geographies have different laws, different markets have different competitors, etc. If an enterprise operates in multiple contexts - and most do - enterprise solutions have to take them into account. Those different contexts prevent you from building usable - much less reusable - general solutions.

Finally, I think the amount of serendipitous reuse in REST is overstated. If you build an app on the Facebook Platform, can you use it on MySpace? Nope. If you build an app that uses the Flickr services, will it work with Picasa Web Albums? Nope. Of course, there are exceptions - pretty much everyone supports the MetaWeblog API - but those exceptions seem few and far between to me. Furthermore, the bits that are getting reused - such as identifier, format and protocol - are infrastructure capabilities more suitable to reuse anyway. Serendipitously reusing infrastructure capabilities is much easier than serendipitously reusing business capabilities, REST or not.

The problems that stand in the way of reuse aren't technology ones. Furthermore, the reuse problems face by enterprises are very different than ones faced by Web 2.0 companies. REST is a great approach, but it isn't a one-size-fits-all technology solution that magically relegates integration and reuse to "afterthought" status. Serendipity is nice, when it happens. However, by definition it's not something you can depend on.

Afternoon Coffee 106

Lots of meetings today, so my coffee post is late...

• The Big News^tm: Visual Studio 2008 and .NET Framework 3.5 Beta 2 is available for download. Soma and Scott have more. Silverlight 1.0 RC and the Silverlight Add-in for VS08 will apparently be available in a couple of days. Finally, there's a go-live license for the framework, so you get a head-start deploying apps before VS08 and NETFX 3.5 RTM. Time to build out a new VPC image.
• Next week, I'm attending the p&p Service Factory v3 Customization Workshop. I'm looking forward to playing with the new Service Factory drop, but I'm really interested in learning more about building factories. I wonder if they're going to discuss their VS08 plans.
• Nick Malik recently wrote about making "middle out SOA" work. I hate that term "middle-out". It feels like we're pinning our hopes on middle-out because we know top-down and bottom-up don't work. My old boss John DeVadoss (who assures me he'll be blogging regularly again "soon") big vs. little SOA, with big SOA being "dead". I like the term "little SOA" better than "middle-out SOA", but just because big SOA is a big failure, doesn't mean little SOA will make any headway.
• There's a new F# drop available. Don Syme has the details. Looks like they've got some interesting new constructs for async and parallel programing.
• ABC announced yesterday that they are streaming HD on their website. So you can check out the season finale of Lost in HD for free. They embed commercial