Morning Coffee 171

• Big news for IronRuby out of OSCON. John and Jim have the details. Congrats to the IronRuby folks on reaching these milestones and paving the way for others (i.e. IPy) to follow some of the same paths.
• One of those OSCON announcements, is a project my teammate Jimmy Schementi has been working on: Silverline, which "let's you write Rails code that can run on the client".
• Shri Borde - the dev manager for IPy, IRuby and F# - tackles a tricky subject of static compilation of dynamic Python code. This came up on the mailing list recently as one of the outstanding requests for IPy to do is support custom attributes, which requires static compilation. Shri lays out some of the big issues with this approach. However, the community has been fairly clear on this, so it's obviously something we need to look at.
• I met someone from MS Research at the MS Product Fair who pointed me to the Institute for Personal Robots in Education, a joint effort between Georgia Tech and Bryn Mawr College and sponsored by Microsoft Research. Their Myro software (myro == my robot) is written in CPython, but there's an effort underway (aka Miro 3.0) to build a .NET version that uses IronPython. Must investigate.
• Seshadri shows how easy it is to extend C# types in IronPython. It's also shows how simple it is to host DLR code in your app - it's like 6 lines of code!
• Early reviews of IronPython in Action are good.  
• If you want to run an IronPython IDE in your browser with Silverlight, check out SilverShell from Dan Eloff.
• The XNA team has announced their business plans for community games. Basically, you set a price point between 200 and 800 points (aka between $2.50 and $10) and receive a "baseline" of 70% of the revenue the game generates. More details are available in the FAQ. This is pretty excited. I'd like to build some co-op kids games.
• Speaking of XNA, Caligari is now offering TrueSpace 7.6 for free . David Weller and Glenn Wilson provide an XNA viewpoint on the announcement, Chris Pendleton shows how to upload your models to VirtualEarth.
• Congrats to the CodePlex team on their latest drop, which features that a cool new feature - Mailing Lists! IronPython has had a Mailman mailing list for years, so I'm not sure we'll use this feature on IPy, but I'll investigate it
• Two PDC notes: First, Rick Rashid - VP of MS Research - will be delivering a PDC keynote. Second, the PDC team has put up a video podcast on Producing a Ginormous Conference in 10 Minutes or Less! It's the "inaugural episode" so watch for more Countdown to PDC video podcast episodes in the future.
• I recently discovered Chris Smith's F# blog. He's got recent posts on Mastering F# Lists and Guidelines for Readable F# code. For the F# novice, check out his F# in 20 Minutes posts (part one, part two)
• Pat Helland is moving to the SQL team. Good luck Pat!
• I like Nick Malik's formal definition of use cases, but I can't help be reminded of Charlie Alfred's Value-Driven Architecture article in Architecture Journal 5 where he said use cases were "easy to teach and explain" but that "if simplicity were the only goal that counted, we'd all still be walking or riding horses to get from one place to another."

Morning Coffee 166

Yes, I realize it’s been a while. I tried in vain to catch up with my blog reading after my Hawaii vacation and finally just gave up and hit “mark all as read”.

Dynamic Languages

• There's a new version of the DLR hosting spec available (doc, pdf). The DLR implementation is still in motion, so there are some inconsistencies between the spec and the code, but the spec should give you the high level overview you need if you want to host DLR languages inside your app.
• Oleg Tkachenko recently joined the dynamic languages team. He's the creator of the Interactive IronRuby Web Shell, an IronRuby version of Try Ruby. Of course, it’s not as cool as using SL2to execute the code directly in the browser. Michael Foord has his Python in the Browser and my teammates John and Jimmy demoed a Silverlight version of Try Ruby @ TechEd.
• Jim Deville, also of the dynamic languages team, recently started blogging.
• I have a new boss, Dave Remy. He doesn't have a blog - yet - but you can follow him on Twitter as daveremy. When Twitter is actually working that is.
• There's a new homepage/wiki for IronRuby though I’m not sure why there's a picture of Matz wearing a Python shirt on the home page.
• My teammate Jimmy Schementi provides some "continued hope" for a better (heck, I'll take current) ASP.NET and ASP.NET MVC story for DLR languages.
• Via Michael Foord, sounds like IronClad is making good progress. V0.4 can run the bz2 module "in its entirity" (maybe run a spellcheck on your site, guys?) and now apparently, it's now able to load numpy.core. Very exciting!

Other Stuff

• Pat Helland, who has blogged even less than me for the past few months, has a post up about controller and doers in the IT department. After 18 months in MSIT, put me in the doer camp, please.
• The F# team has pushed out a spec for v1.9.4 of the language. Don Syme says it's not official, but it's a huge improvement over the old informal spec
• Speaking of F#, my friend Matthew Podwysocki recently published FsTest, a testing DSL for F#. I wrote about F# unit testing as part of my PEG parsing series, and I really like the direction Matthew has taken this project. You can pull it down from CodePlex.
• When I did my PEG talk @ Lang.NET, Gilad Bracha mentioned I should check out oMeta. It looks really cool, though with the job change I haven’t had the time to play with it. Now I discover that Jeff Moser is working on a version for CLR called oMeta# that I’ve got to spend some time with. And in the comments to that post, I discovered pyMeta from Allen Short, though it apparently doesn’t work on IronPython (must investigate why).
• James Kovacs introduces psake, a PowerShell based build automation tool which uses a rake-inspired internal DSL syntax similar to one I blogged last year. I'd love to see this take off, but given MSBuild's tool integration, I wonder if that's feasible.
• I upgraded my home wireless network almost exactly a year ago. I've been happy with the range and coverage, but not so happy with the Buffalo Tech firmware. The built-in DHCP server is pretty flaky. So I upgraded to the open-source Tomato firmware. Upgrade was smooth, though I did need to reset my cable modem. But even that was smooth - Comcast has an automated service for that now,

Kitchen Sink Variability

Nick Malik forwarded the last ZapFlash newsletter to me. I gave up on analyst newsletters like this long ago, but Nick shared it with me because it "hit directly on what [Nick] thinks an ESB is and does, and why an ESB is not a hub." I'm not a fan of the whole ESB concept and frankly this article didn't do much to change my opinion. However, this passage did sorta jump out at me.

[T]he main concept of SOA is that we want to deal with frequent and unpredictable change by constructing an architectural model, discipline, and abstraction that loosely-couples the providers of capability from the consumers of capability in an environment of continuous heterogeneity. This means that we have to somehow develop capabilities without the knowledge of how they might be used...[T]his means that we need to have significant variability at a variety of aspects including the implementation, infrastructure, contract, process, policy, data schema, and semantic aspects. Having this variability allows companies to have stability in their business and IT even though the IT and business continue to change. Agility, predictability, reliability, visibility, and cost-effectiveness all become that much more realistic when we can achieve that sort of abstraction.

My reading of this is that the author, Ronald Schmelzer, is advising organizations to introduce "significant variability at a variety of aspects" in their services in order to deal with what he openly admits is "unpredictable change".

This sounds like a mind-boggling awful idea to me.

At it's heart, any practical design - including a service-oriented one - needs to be an exercise in tradeoff analysis. You can't add "significant variability" without also adding significant complexity, effort, time and cost. So the real question is: Is the significant variability Ronald describes worth the inevitable tradeoff in significant time, effort, cost and complexity?

I seriously doubt it.

Since unpredictable change is - by definition - unpredictable, you have no way of knowing if you will actually need any specific aspect of variability down the road. Ronald's strategy - if you can call it that - seems to be let everything he can think of vary except the kitchen sink. That way, when said unpredictable change happens, you might get lucky and have already enabled the variability you need to handle the change with a minimum of effort.

Getting lucky is not a strategy.

Chances are, a specific aspect of variability won't ever be needed. In other words, most of the the time, effort and money you spent building these aspects of variability will be wasted. And remember, this isn't just a one time cost - the increased complexity from including this significant variability means you'll pay the price in additional time, effort and money every time you have to change the system.

I wonder if Ronald is familiar with the term "You Aren't Gonna Need It". He talks about increasing business agility, but he eschews many of the principles of agile development. I realize they aren't the same thing, but I have a hard time believing that they are so diametrically opposed that a core principle of agile development should be readily violated in order to enable business agility.

Maybe it's cliche, but I try to always come back to "What's the simplest thing that could possibly work?". I would think that building a ton of currently-unnecessary variability into your system on the off chance that someday you'll need it fails the "simplest thing that could possibly work" test spectacularly.

Personally, given the choice of taking advice from Ward Cunningham or pretty much any enterprise analyst on the planet, I'll pick Ward every time.

What is the ROI on EA?

Nick Malik took me to task for my suggestion that Enterprise Architecture provides no value.

You implied that I could not answer the question, "How does EA demonstrate value." That is not true. I can readily answer the question, from my viewpoint, but I chose instead to *ask* the question to see if my answer matches the various answers that I may hear back. I got a lot of valuable input, both on the blog and on the forum on Shared Insights where I asked the same question.

You are the ONLY person to reply and say that EA provides no value.

Perhaps you should read about the role and value of Enterprise Architecture from established sources before you bash the entire profession.

EA is real, my friend. It is as real as city planning. The only major city in the US without city planning is Houston. I have visited a few times, and I can honestly say that without city planning, they are a mess.

Nick also provided a link to an article in Architecture and Governance magazine. I was going to read it, however their web site is down as I write this. I feel comfortable interpreting that as a sign that I'm right... :)

Actually, Nick's got a point. It was wholly unfair of me to say that EA provides no value. However, I do believe the return on investment of enterprise architecture is fairly low, perhaps even negative. In other words, I shouldn't have argued that EA doesn't deliver any value, but instead that I don't think it delivers enough value, given what we spend on it.

Architecture ROI is hard enough to calculate on a project by project basis. I would argue that measuring it at the EA level is probably impossible, but I think that's both a blessing and a curse. It's a curse because EA can't justify their existence in terms the business can understand. It's a blessing because if EA is running as deep in the red as I suspect they might be, the company would cut EA entirely in a heartbeat.

Since Nick started my asking a question about value, let me turn it around and ask some questions of my own:

1. How much do you think your organization spends on EA per year?
2. What do you think your organization's EA ROI is?
3. What can you do to improve your organization's EA ROI?

Morning Coffee 150

• Yesterday was the NHL trading deadline, and the Capitals were very busy. They obtained Huet from Montreal, Federov from Columbus and Cooke from Vancouver. Given they are fighting just to make the playoffs, going for three soon-to-be unrestricted free agents seems like an odd choice. However, the consensus (among my parents anyway) was that it's critical to get this very young Caps team some playoff experience. Even if all three walk at season's end, it'll be worth if the Caps make a playoff run. Besides it's not like we gave up much: an extra second round pick in '09, a 19 year old defensive prospect (who was apparently 14th on the depth chart) and an underachieving winger.
• Speaking of the Caps playoff chances, they are currently one and a half games back of the division leading Hurricanes and two games behind the current eighth seed Flyers. Yes, I rank hockey teams using baseball's standings system. Otherwise, you have to talk about games in hand (i.e. the Caps are five points behind Carolina with two games in hand).
• The writer's guild ratified the new contract, so Hollywood labor strife is now officially behind us. At least until July when the the actors may go on strike.
• It seems like a slow week for Microsoft geek news, which is odd since WS08, VS08 and SQL08 all launch today. I'm guessing it's the calm before the Mix storm next week.
• After going dark for six months, Linq to XSD has been re-released to work with the RTM version of VS08. Scott Hanselman demonstrates Linq to XSD by applying it to OFX, an XML Schema he calls "goofy" but apparently helped develop. OFX uses derivation by restriction, which has no direct corollary in C#, but Linq to XSD's  is able to translate between XML and objects without loosing any of that type fidelity. Nice to know Linq to XSD can tolerate OFX's level of goofiness, though I'm guessing most people use much more straightforward schemas.
• Speaking of Linq, I discovered LINQPad via a comment on Rob Conery's blog (which I found via DNK). It's basically a code snippet IDE for C# 3.0 and VB9, with it also has built in database connection support, so it can fulfil much the same role as SQL Management Studio. I only played with it for a few minutes, but I was really impressed.  This is definitely going in my utilities folder. I wonder if they're interested in supporting F#?
• Not sure how I missed this, but you can get MSDN Magazine via same Syndicated Client Experience as Architecture Journal. Unlike AJ which is divided into issues, the MSDN magazine client is divided into topics which is harder to square with the physical magazine. On the other hand, since MSDN Mag has been around longer, perhaps topics + search is a better discovery mechanism.
• Soma announces the Visual Studio Gallery, a repository of VS Extensions. It's kinda cool, but the whole discovery mechanism is clunky. I might like to experiment with some free or even free trial products, but there's no way to filter on cost so finding them is a hassle. Also, there's no way for community members to vote, rate or comment on the products in any way.
• Nick Malik can't answer the question "how does Enterprise Architecture demonstrate value?" I could be snarky and say "it doesn't", but that's only half the answer. It doesn't, but it should. My opinion, since you asked Nick, is that EA fails to deliver value because it tries to control the uncontrollable. Trying to gain efficiency thru establishing standards and eliminating overlap via reuse are pipe dreams, though literally millions of $$$ have been poured into those sink-holes. There are a few areas where centrally funded infrastructure projects can solve big problems that individual projects can't effectively tackle on their own. EA should focus their time there, they can actually make a difference. Otherwise, they should stay out of project's way.

Morning Coffee 147

• My son Patrick turns five today. The big treat was his cousin Jack coming up for a visit. Here's a picture of the two of them at Patrick's party on Saturday. My wife has all the details on her blog. Update: My wife just posted a whole slew of Early Patrick Pictures.
• If my son is five, it means this blog is also five - I started this blog about a month before Patrick was born. I never remember to mark the occasion until Paddy boy's big day comes around.
• Major props to the House of Representatives for growing a backbone and not caving to President 30% Approval on telecom immunity...yet. Personally, I'd like to see the House bury the measure completely, though I'm not holding my breath. But given that even the right-wing Washington Times reports "Analysts say FISA will suffice", maybe the House Dems will do the right thing.
• After tearing it up since Thanksgiving, the Caps have gone a little cold. 5-4-1 in their last ten and 2-2-1 in their last five. In the month of February, they're 1-3-1 against SE division opponents. Good news is that they're still even with Carolina (two points behind with two games in hand), half a game up on Atlanta, a game and a half up on Florida and two and a half games up on Tampa Bay.
• Bill Gates announced a new program called DreamSpark to provide college students access to all of Microsoft's developer and designer tools, including Visual Studio, Expression, SQL Server, Windows Server and XNA Creators Club membership. This looks like an outgrowth of the MSDN Academic Alliance program. I think it's a great idea. Update: Looks like high-school students will be able to access the DreamSpark program too. However, since they're minors, they have to get the software via their teachers. (via LiveSide)
• The winners of the XNA Silicon Minds contest have been announced. Of the five winners, Specimen looks the coolest to me. I wish I had more time to get into game development. (Via LetsKillDave)
• Speaking of game development, this week is the Game Development Conference, so be on the lookout for lots of game-related news. Xbox Live VP John Schappert is giving a keynote on "Unleashing the Creative Community". XNA GM Chris Satchell said last year they would "announce full details on, and ... vision for, opening XNA creations to the community" sometime this year. I'm guessing this is said announcement.
• Speaking of Xbox, there's a rumor that Microsoft and Netflix will announce this week that Netflix is bringing their Watch Instantly service to Xbox 360. If true, sign me up!
• Grigori Melnik announces the GAX/GAT February 2008 final release. Key feature is VS08 support. Is it just me, or does calling it the "final release" make it sound like they won't be upgrading GAX/GAT further?
• Speaking of p&p, Grigori also announces the Feb 2008 CTP of Unity, p&p's new IoC container. I've seem lots of folks echoing the announcement, but not much in the way of specifics on Unity itself. For example, Chris Brandsma describes IoC and mentions Unity, but he doesn't cover any Unity specifics. :(
• MSIT EA Nilesh Bhide has started blogging. His first post is on Customer perception of Service Quality in S+S/SaaS. I've worked closely with Nilesh in the past two years, so I'm excited to see him take to the blogosphere. (via Nick Malik)
• I don't know how I missed it, but the MSDN Code Gallery launched last month. As Charlie Calvert explained, this is logical successor to GotDotNet's user samples area. Between Code Gallery and CodePlex, GotDotNet has finally been shuttered for good.
• Telligent, makers of the very popular Community Server, have released Graffiti CMS, which looks like a more flexible content platform than Community Server. (via DNK)
• In somewhat unexpected news (at least, unexpected by me) Microsoft has released specs for the Office binary file formats. I'm not sure why this is happening now, rather than say when we released the specs for the Open Office XML file formats. (via DNK)

Nick's Flawed Vision of a Shared Integration Model

Of all the things you might say about Nick Malik, "thinks small" is not one of them. He takes on a significant percentage of the .NET developer community over the definition of Mort. He wants to get IT out of the applications business. He invents his own architecture TLA: SDA (aka Solution Domain Architecture). He's a man on a mission, no doubt. And for the most part, I'm with him 110% on his ideas.

However, when he starts going on about a shared global integration model, I start to wonder if he has both hands on the steering wheel, as it were.

Nick's been talking about this for over a year. As he points out, SaaS integration layer is the new vendor lock-in. One of the attractions of SaaS is that you could - theoretically, anyway - switch SaaS application providers easily which would drive said SaaS companies to constantly innovate. However, if the integration layers aren't compatible, the cost to switch goes up dramatically, leaving the customer locked-in to whatever SaaS company they initially bet on - even if that bet turns out to be bad.

OK, I'm with him so far. Not exactly breaking news here - we've seen the same integration issues inside the enterprise for decades. SaaS adds new wrinkles - if your ERP vendor goes belly-up, they can't take your data with them or worse sell it to your competition - but otherwise it sounds like the same old story to me.

However, where Nick loses me is when he recommends this solution:

"To overcome this conflict, it is imperative that we begin, now, to embark on a new approach. We need a single canonical mechanism for all enterprise app modules to integrate with each other. If done correctly, each enterprise will be able to pick and choose modules from different vendors and the integration will be smooth and relatively painless."

Yeah, and if a frog had wings, it wouldn't bump its ass when it hopped.[1] There are so many things wrong with this approach, I'm not sure I can get them all into a single web post.

First off, it won't, in fact, be done correctly - at least, not the first time. I realize everyone knocks MSFT for never getting an application right before version 3.0, but I believe it's actually systemic to the industry. Whatever you think you know about the problem to be solved, it's at best woefully incomplete and at worst wrong on all counts. So getting it right the first time is simply not possible. Getting it right the second time is very unlikely. It isn't until the third time that you really start to get a handle on the problem you're really trying to solve. Getting an effort like this off the ground in the first place would be a Herculean task. Keeping it together thru a couple of bad spec revisions would be impossible.

Meanwhile, the vendors aren't going to be waiting around twiddling their thumbs waiting for the specs to be done. We've seen efforts to unify multiple completing vendors around a single interoperable specification. By and large, those efforts (UNIX, CORBA, Java) have been failures. The technologies themselves haven't been failures, but the idea that there was going to be "relatively painless" portability or interoperability among different vendors never really materialized. If it didn't work for UNIX, CORBA or Java, what makes Nick think it will work for the significantly more complex concept of a shared global integration model? Not only more complex in terms of spec density, but the mind-boggling number of vendors in this space.

Nick is worried that either "we do this as a community or one vendor will do it and force it on the rest of us." But if you look at how specifications evolve, retroactive realization of defacto standards is the way the best standards get created. For example, I could argue that TCP was forced on us by the US Military, but I don't hear anyone complaining. I realize there's a big difference between having a vendor force a spec down our throat vs. a single big customer, but either way it's not designed by committee. Besides, if we do see get an enterprise integration standard forced on us, I don't believe it will be the vendors doing the forcing. If I were a betting man, I'd bet on Wal-Mart. Business leverage trumps IT leverage and Wal-Mart has more business leverage than anyone in this space these days.

BTW, would design-by-committee be an extreme example of BDUF? Do we really want to develop a this critical integration model using the same process that produced the XSD spec?

Finally, Nick thinks that this model will improve innovation, where I think it will have the exact opposite effect. Once you lay a standard in place, the way you innovate is to build proprietary extensions on top of that standard. However, by definition, these extensions aren't going to be interoperable. If someone has a good idea, others will copy it and eventually it will become a defacto standard.

A recent example of the process of defacto standardization is XMLHttpRequest. Microsoft created it in 1999 for IE 5, Mozilla copied it for their browser a couple of years later, followed by the other major browser vendors. Google innovated with it, Jesse James Garrett coined the term AJAX, everyone else started doing it and then finally - nearly a decade later and still counting - a standards body is getting around to putting their stamp of approval on it.

However, if you're worried about painless integration and not having something forced on you by some vendor, then you're not going to embrace these innovations - which means, you won't embrace any innovation. Well, there may be some innovation in the systems themselves that doesn't affect the interface, but once that interface is cast in stone, the amount of innovation will go way down. How do vendors differentiate themselves? There's only two ways: price and innovation. Take away innovation with standardization, and you're left with a race to the rock bottom price with no incentive to actually improve the products.

I get where Nick is going with this. He looks around our enterprise and sees duplication of effort and massive resources being spent on hooking shit together. It sure would be nice to spend those resources on something more useful to the bottom line. But standardizing - or worse legislating - the problem out of existence isn't going to work. What will? IMO, applying Nick's ideas of Free Code to interop code. If we're going to get IT out of the app business, can't we get out of the integration business at the same time?

---

[1] It's exceedingly rare that you get to quote Wayne's World or Raising Arizona in a discussion about Enterprise Architecture, much less both at the same time. Savor it.

Morning Coffee 134

• Bill de Hora responds to a few of my Durable and RESTful ideas. He points out that relying on a client-generated ID can be troublesome, and recommends using multiple identifiers - one created by the sender, one by the receiver and one representing the message exchange itself. However, the sender ID is vulnerable to client bugs & tampering as Bill points out, and neither the receiver ID nor the exchange ID can be used to determine if a given message is a duplicate. If you don't trust the sender, is it even possible to determine if a given message is a duplicate?
• Pablo Castro confirms that there are "practical limits" to what ADO.NET Data Services can do with respect to idempotence. Nothing in his post was surprising, though I hope it will be more explicitly called out in the final docs. Developers used to the comforting protection of a transaction may be in for a rude awakening.
• Dare Obasanjo has a great post comparing the new features in C# 3.0 to dynamic languages like IronPython. I believe many of the productivity aspects of dynamic languages have little to do with being dynamic.
• Pat Helland noodles on durability and messaging, two topics near and dear to my heart (probably from working with him for a couple of years). I'm not sure where he's going with this - his conclusion that "Basically, big, complex, and distributed system are big, complex, and distributed" isn't exactly ground-breaking. But his point that "durable" isn't a binary concept is worth more consideration. Also, his description of IMS only looking at the effects of a committed transaction is very similar to how web sites work, though obviously HTTP isn't durable so you can't make event horizon optimizations like IMS did.
• Tangentially related, Werner Vogels discusses the idea of eventually consistent distributed databases. Today, that's a problem mostly only Internet-scale sites like Amazon deal with. In the near future of continued data explosion + manycore, we'll all have to deal with it.
• Nick Malik argues that categorizing enterprise applications by lifecycle is much less useful than categorization based on organizational impact. He might also need a new chair.
• Jesus Rodriguez digs into one of SSB's new features in SQL 2008: conversation priorities.
• Arnon Rotem-Gal-Oz and Sam Gentile are mixing it up over the definition of SOA. Sam thinks SOA has to include business drivers and Arnon doesn't. I'm with Sam on this, defining "SOA" independently from "Applying SOA" seems pointless. Then again, rigorously defining SOA - much less arguing about said definition - seems like a waste of time in the first place IMHO.
• Wow, this guy Zed is mad at the Ruby community.
• Andrew Baron has 8 Reasons Why The TV Studios Will Die. Personally, I think reason #2 - Expendable Middle-Person - is the most important. If content producers can reach consumers directly, what value-add will the networks provide? (via United Hollywood)

Morning Coffee 130

• Michael Klucher announces the release of XNA Game Studio 2.0 and Major Nelson points to the press release announcing the release. You can get the bits from XNA Creators Club Online (the XNA dev center has yet to be updated).
• Speaking of XNA, David Weller points out the warm-up challenge for Dream-Build-Play 2008. I assume networking will be a big part of this years' entries, but the warm-up challenge is to "Create a new and innovative use of Artificial Intelligence in a game".
• Still speaking of XNA, Gamasutra has an interview with XNA GM Chris Satchell where he hints at a publishing channel for XNA games on the Xbox 360, with "full details" coming sometime in the new year.
• The Capitals beat the Rangers in overtime last night. Since changing coaches on Thanksgiving, they're 6-3-1. That's great, but they're still five games under .500. The good news is that even though the Caps tied for last in the league, they're only six points out of a playoff spot with about fifty games left in the season.
• My old team puts on an event every year called the Strategic Architects Forum. It's invite-only, but they've posted some of the videos, slides and transcripts from this year's event.
• J.D. Meier discusses the new Guidance Explorer release. They're now up to 3,000 "nuggets" of guidance and they've moved the guidance store to MSDN. (via Sam Gentle)
• Arnon Rotem-Gal-Oz explains further why arbitrary tier-splitting is bad. I'd also suggest reading Chapter 7 of PoEAA which provides another version of the same story: You can't take an object that's designed for fine-grained local access and make it remote without really screwing yourself up.
• Eric Lippert thinks immutable data structures are "the way of the future in C#" so he's written a series on immutability. Posts include kinds of immutability, an immutable stack, an immutable covariant stack and an immutable queue. As I've discussed, immutable data structures are HUGE in functional programming. Eric's immutable stacks and queues are similar to F#'s native list type. (via Jomo Fisher)

Morning Coffee 127

• I'm back at the office today after almost two weeks away. So "catch-up" is the official hyphenated word of the day.
• Big news last week was an update on Silverlight. The next version (the one with the embedded cross-platform CLR in it) was rebranded Silverlight 2.0 and will include higher-level UI framework features and richer networking support. Look for a go-live beta Q1 next year.
• According to Scott Guthrie, there should be a CTP of the ASP.NET 3.5 Extensions this week, including the new ASP.NET MVC framework that many folks await with baited breath. Scott posted the second in new series on this framework, this time covering URL Routing.
• After a long blog silence, Pat Helland posted four presentations that he delivered @ TechEd Barcelona. They're pretty much required reading as far as I am concerned. Here's hoping the TechEd Barcelona folks recorded at least the audio of these sessions.
  • Metropolis: Interchangeability of Operations
  • Life Beyond Distributed Transactions: an Apostate's Opinion
  • Data on the Inside versus Data on the Outside
  • The Irresistible Forces Meet the Movable Objects
• There's a new drop of F#, v1.9.3.7. Don Syme calls it "a release candidate for F# 1.9.3". Full release notes are over on his blog.
• Speaking of F#, Douglas Stockwell explains how to roll-your-own F# Express by combining the F# download above & the VS08 shell.
• Still speaking of F#, Robert Pickering has written an article on Active Patterns as well as a series on concurrency in F#. So far he's covered Async Workflows (twice), Erlang style messages and message queuing.
• Speaking of concurrency, Soma announced the CTP of ParallelFX last week. This includes Parallel LINQ (aka PLINQ) and Task Parallel Library (aka TPL). Also check out their whitepaper, team blog and dev center.
• I'm a big fan of the Architecture Journal. Now you can get all that great content in a great WPF-based app. Simon Guest has the details.

Morning Coffee 118 - ITARC SoCal Edition

I'm not back on blog sabbatical, but between finishing my presentation and attending ITARC SoCal earlier this week - not to mention being sick - I didn't have time to write anything. Normal Morning Coffee resumes tomorrow, here's a summary of my notes from on my two days at ITARC.

• Scott Ambler did the opening keynote on agile enterprise architecture strategy.
• He claims that success is more prevalent in the industry that people think, because the industry has a narrow definition of success. If you change (aka widen) the definition, the success rate goes way up! That's not exactly useful, but he referred to an as-yet-unpublished survey on project success rate that should be up on DDJ "soon". I'd like to see that raw data.
• While I agree with most of his points, Scott's presentation style is very abrasive. For example, he makes the point that there is no one-size-fits-all process, which I couldn't agree with more. But does he say it like that? No, he says "Repeatable processes? What an incredibly stupid idea!" even though the room is full of folks who probably think repeatable process is actually a good idea.
• Scott suggested that unit tests are the best way to specify requirements. I've heard this before from agile practitioners, but something nags at me about it. Certainly, having executable requirements is a huge plus. But how can you be sure they're the right requirements if the stakeholders can't read them?
• This keynote setup what turned out to be a major theme for the conference - traditional vs. non-traditional enterprise architecture. Or as I would characterize it: Industrial vs. Post Industrial architecture.
• Simon Guest presented on user experience in architecture, which is his specialty these days. He lays out a UX model that was very compelling. I'm not sure if there's a whitepaper version of this model (there should be) but you can see the model as he lays it out in powerpoint. I've seen Simon's UX decks, but never actually seen him present it, so that was a treat.
• I skipped Ted Neward's session in order to take in something new. So I went to see Daniel Brookshier of No Magic talk about DoDAF - the Dept. of Defense Architecture Framework. I had met Daniel the night before at dinner and while No Magic primarily sells UML modeling tools, we seemed to agree that UML is most useful (in my opinion "at all useful") when you imbue the vanilla models with custom semantics - aka you turn them into a DSL. So while I liked hanging out with Daniel, his DoDAF session did nothing except ensure I never work for the DoD. There's no amount of money that's worth dealing with the two dozen or so bureaucratic models that are all wholly isolated from anything that actually executes. Daniel kept saying how easy these models are to build. I'm sure they are, but that's not the problem. Since they're not an intrinsic part of a construction process, they won't stay up to date. This was a very industrial approach - Daniel even stated at one point that he was "anti-Ambler".
• David Chappell did the second keynote on grid-enabled SOA.
• When did David join Oracle? I guess I haven't been paying much attention to competitors since I moved to MSIT.
• There's an article version of this presentation available, but I haven't read it yet.
• For me, the best part of this presentation was him acknowledging that there's a need for non-stateless services, something he has blogged about recently. I'm not sure I agree with his framework for stateful interaction, but at least he's admitting that it's needed. Now if I could only convince the Connected Systems Division...
• The rest of his talk was basically a sales pitch for the Coherence product Oracle recently bought. Basically, it's a huge, multi-node, redundant, in-memory database. While I'm sure there are a few high-end problems out there - my immediate thought was travel and David mentioned SABRE is one of their customers - this is not a good general purpose solution, though David was positioning it as such.
• My talk on "Moving Beyond Industrial Software" was after the second keynote. It was good, if sparsely attended. I'm doing it again @ the p&p Summit so I'll post the slides and hopefully a recording after that.
• I skipped the last session of the day to decompress, so the next session I went to was the day two opening keynote by Fred Waskiewicz, OMG's Director of Standards. His talk, unsurprisingly, was on the value of standards - in particular, OMG's standards. This was about as anti-Ambler, anti-agile, pro-industrial a presentation as you could make. I'd heard this spiel before, so I mostly tuned out. I did challenge Fred on his point that the UML models are at a higher level of abstraction than code. They're not - they're a visualization and they're very useful, but they're at the exact same level of abstraction as code. That's why you can automatically generate the visualization in tools like Visual Studio's class designer. Fred didn't have much of a response to my question, though he did point out that some models like Business Process Models are, in fact, higher levels of abstraction.
• Next was what I thought was the best presentation of the entire show, IASA Founder Paul Preiss on what architects need to know. Note, I'm not brown-nosing Paul here - I'm the guy that first decided to commit Microsoft as an IASA sponsor, so he has to like me even if I thought his session was crap. Paul talked about architect as a career, comparing it to doctors. He worries that he's over-using that analogy, but software architect has much more in common career wise than it does with building architects IMO. I wonder where one might do their architecture residency? He also thinks of architects as "living governance", saying that project managers answer to the stakeholders while architects are beholden to the stockholders. I like that approach to governance.
• Finally, I attended Vince Casarez's session on Web 2.0 in the enterprise. Vince is an Oracle VP and this turned into a sales pitch like David Chappell's keynote did. I'm not sure what product it was, but it reminded me of QEDWiki from IBM that I saw at ETech last year, which isn't a complement. If you're going to build an enterprise mashup designer, is it just me or is "lots of code spew" a poor model. Why not go for something like Popfly or Pipes?
• I left early the second day in order to get home before my kids went to sleep (which I failed at due to lack of naptime). Overall, the conference was pretty good, though a bit sparsely attended in part I think because they held it in San Diego. The Orange Country IASA user group is very popular, so I don't understand why they didn't just hold it around there somewhere. Live and learn, I guess. They did have to postpone the DC event until next year sometime. Here's hoping I get invited to that as well as well as ITARC SoCal '08 (note, that *is* brown-nosing a bit)

The DevHawk 2007 World Tour

After spending almost all of fiscal year 07 (July '06 thru June '07) not traveling and not presenting, I'm going to be doing a few public talks to finish out the year. If you, dear reader, are going to one of these please drop me a line. Invariably, it's the side meetings and discussions that are the most valuable at these conferences.

IT Architect Regional Conference 2007
October 15th - 16th, San Diego, CA

I'm a huge fan of IASA, so I'm thrilled to be doing their west regional conference. I've presented to a packed house for the local chapter before, so I think these folks will put on a good conference. They sure have a good selection of topics and speakers.

My session is called "Moving Beyond Industrial Software". Here's the abstract:

Computers have been instrumental in ushering in the post-industrial age. Yet, most enterprises today are run with an industrial mindset and the IT department is organized like a factory. This creates a tension between the forces of industrialization that define the organization and the forces of post-industrialization that define today’s marketplace. For example, our post-industrial world is becoming more decentralized by the day. Yet many organizations believe the key to a successful service oriented architecture – a very decentralized system design – is to have a central service repository.

In this session, Harry Pierson will examine this tension, get you thinking outside the industrial mindset and help you think about software development in a post-industrial way.

I'm very excited about this talk.

MS SOA & Business Process Conference
October 29th - November 2nd, Redmond, WA

I'm not presenting at MSSOABPC (that's a mouthful) but looks like most of my team is going. So if you're going and want to hang out with the guys who are doing this stuff in the trenches @ MSIT, let me know. Also, I put out the call for anyone interested in a geek dinner. From the agenda, looks like they're keeping us busy until 8pm every night Mon-Wed, so we can either a) have geek dinner Thursday or Friday or b) have geek beers after one of the receptions in the early part of the week.

patterns & practices Summit USA West
November 5th - 9th, Redmond, WA

I did the p&p Summit back in 2005, a very successful debut of my Developer 2.0 talk. (I'm doing that talk at a different conference this year, details below.) This year, I'm not 100% sure what I'm going to talk about yet. I'm currently slated to talk about the Rome project that I'm doing in MSIT, but given our current slow progress on that project, I'm probably going to talk about something else. I'm thinking either the "Moving Beyond Industrial Software" talk described above or the "Facing the Fallacies of Distributed Computing" talk described below. Any other suggestions?

DevTeach Vancouver 2007
November 26th - 30th, Vancouver, BC

This is a brand new experience for me. Frankly, I'd never heard of DevTeach before my friend Mario Cardnial suggested I submit a couple of sessions. Since it's only a few hours drive away, I'm bringing the family along. We'll see how that goes. And when I'm not doing my sessions or hanging out with the family, I might take in a session or two in the XNA track.

Here are the sessions I'm doing:

Developer 2.0
Finding Your Way in the Future of Software Development
The one constant in software development is change. Software development in 2007 is dramatically different than it was in 2000, which was in turn dramatically different than in 1993. You can be guaranteed that the platforms, languages, and tools will continue to evolve. Learn how Harry Pierson, Architect in Microsoft IT, believes software development is going to evolve in the next five years and what you must do today to remain competitive.

Facing the Fallacies of Distributed Computing
Sun Fellow Peter Deutsch is credited with authoring "The Eight Fallacies of Distributed Computing". These are near-universal assumptions about distributed systems that “All prove to be false in the long run and all cause big trouble and painful learning experiences.” In this session, we will examine these fallacies in depth and learn how to avoid them on the Windows platform by leveraging Web Services, WCF and SQL Service Broker.

The Integration Business Case, continued

Nick responds to my visceral thoughts on the integration business case. There's no point in excerpting it - go read the whole thing. I'll wait.

It looks like for case #2, he added the ability to "change readily and inexpensively", which is to say he made it overlap even further with #4 than it used to. He also changed #3 to make it clear that he was collecting metrics to give us "awareness of process efficiency". That makes #3 overlap with #4 on efficiency instead of #1 on BI, but either way it's still redundant.

So we're still left with the business cases of Business Intelligence, Efficiency and Agility. Nick conflates Efficiency and Agility both in his original post and his follow-up, but I think it makes sense to separate them. I still stand by my original point that the business is only interested in directly funding Business Intelligence.

Nick is willing to bet a nice lunch that MSFT has invested more in improving operational efficiency that we have on BI in the past four years. He's probably right, but he missed the point I was making. The business will readily invest in improving a specific process they can measure the ROI on improving. MSFT has lots of processes, I'm sure most of them have significant room for improvement.

But Nick's list isn't about specific improvements. He's explicitly wrote that he's describing a scenario where "our systems are all optimally integrated". Selling the business on generally improving efficiency is very different that selling the business on improving the efficiency of a specific process. I'd bet the same nice lunch that the vast majority - if not all - of integration infrastructure running at MSFT was originally deployed as part of a specific business scenario that needed to be solved.

My point here is most businesses are better at funding projects to meet specific business needs than it is at funding pure infrastructure projects.

As for agility. Martin Fowler pointed out once that adding flexibility means adding complexity. But chances are, you'll be wrong about the flexibility you think you'll need. So you actually end up with the additional complexity but none of the flexibility benefit. Martin recommends "since most of the time we get it wrong, just don't put the flexibility in there". Instead, you should strive for simplicity, since simpler systems are easier to understand and thus easier to change.

Does the same philosophy apply to process? I think so, though there is one thing I'd be willing to risk being wrong on. We all expect the steps in a process to change over time, so moving to a declarative model for process definition sounds like a good idea. Luckily, there's existing platform infrastructure that helps you out here. But beyond that, I can't think of a flexibility requirement that I'm so sure of that I'm willing to take on the additional complexity.

Again, I'm not saying efficiency or agility (or integration for that matter) are bad things. I'm saying they're a tough sell to the business in the absence of specific scenarios. Selling the business on automating the ordering processing is feasible. Selling the business on building out integration infrastructure because some future project will leverage it is much tougher. If you can sell them on it, either because the company is particularly forward thinking or because you can sell ice to Eskimos, then more power to you. But for the rest of us, better to focus on specific scenarios that the business will value and keep the integration details under wraps.

Morning Coffee 115

• Scott Guthrie has two new posts in his series on LINQ to SQL. The first covers updating the database using stored procs instead of dynamic SQL. I was somewhat surprised that there wasn't the capability to auto-generate vanilla Insert, Update and Deleted procs, but I guess DBA's probably hate that anyway. The second shows how to use ExecuteQuery to execute arbitrary SQL instead of using the cool LINQ query syntax. I'm doing a bunch of loosely-typed SQL work right now, so I'm going to take a deeper look at this.
• Speaking of LINQ, I just discovered this great series on IQueriable by Bart De Smet. It's four months old, but takes an incredibly detailed look at what happens under the hood with LINQ. Bart also has a reference implementation of LINQ's standard query operators as well as LINQ to Sharepoint.
• Dan Maharry has pulled together what looks like the definitive guide for really slimming down and speeding up your VPC. It's XP specific, but I'd bet most of the guidance would also apply to WS03, which is what I mostly use in my VPCs. (via Larkware)
• Jimmy Nilsson thinks it's the operations department that holds the power in today's IT world. I agree 100% That's why I value Dale's input so much.
• Nick Malik wonders if it's time to translate the Federal Enterprise Architecture for use in the commercial sector. My dad just retired from 5 years in the FAA and he thinks FEA is too high level to be particularly useful.
• The 2007 edition version of Scott Hanselman's ultimate tool list is now available.
• A bunch of XNA Gamefest sessions are now available for on-demand viewing.

Morning Coffee 114 - MoMAAB Edition

• We spent all day yesterday discussing four topics: SaaS, Tools for Scrum, Web 2.0 and Domain Specific Languages. Even though it was just a day, my brain is full. These were deep and challenging discussion. I need to let the discussions stew a bit before posting anything about them here. But I will.
• Next time we do one of these, I'm bringing a video camera. I took notes, but looking over them the next morning they seem woefully incomplete. OneNote's integrated audio/video recording capabilities would nicely augment my notes.
• We ran this meeting using Open Space, and it worked very well. Of course, we only had 8 people, so we didn't need a lot of process to self organize. However, it did whet my appetite for having a larger Open Space style un-conference for architects. Is that something other folks might be interested in?
• Major thanks to the folks at Clarity Consulting who graciously gave us space to meet and fed us yesterday. Their CTO Jon Rauschenberger sat in on most of our meeting, and drove our Web 2.0 discussion. I said I wanted to stew a bit on the discussions, but Jon's slides are available on line if you're interested.
• Scott Colestock showed me Diigo, a social annotation tool. Where del.icio.us lets you tag and annotate individual pages, Diigo lets you annotate and highlight specific parts of the page. They also have blogging tools, where these annotations and highlights become blog posts, but they don't support dasBlog. However, since FeedBurner doesn't support Diigo for link splicing, I'm afraid my use of it will be limited.
• Jim Wilt introduced me to Virtual PC's command line. He recommends using "-pc <vpc name> -launch -singlepc" which launches a single virtual environment without the VPC console. I rarely run more than one VPC at a time and I hate stuff cluttering up my taskbar and notification area, so I like this a lot.
• Loren Goodman demonstrated the SharePoint Explorer Client. SharePoint & MOSS came up several times in all of our topics, so this is going to get a second look. I always thought it was strange that MSFT ships a smart client for editing WSS & MOSS, but not viewing it. SP Explorer looks like it fills that gap nicely.
• Shannon Braun sent us all a link to the 50/70 rule, which seems like a good rule of thumb. Of course, assuming that things won't progress linearly is almost always a good rule of thumb. But the 50/70 rule has reasoning behind the assumption.
• Chicago is nice, but the weather has been a little freaky. It's either been hot & humid, downporing thunderstorms or tornados. Keith Powell showed me FlightAware, which shows you flight departure and arrival history. My flight hasn't left within an hour of scheduled departure in a week. I'm going to try and grab an earlier flight, but I have a feeling it's going to be a long trip home.

Morning Coffee 113

• I'm in Chicago today and tomorrow for a reunion of sorts. In my last job, I managed a group of external architects called the Microsoft Architecture Advisory Board (aka the MAAB). We discontinued the program a while back, but the core of the group found the program valuable enough they have continued to meet anyway. I found the MAAB meetings incredibly valuable and insightful, so I'm really excited to be invited to continue my involvement with the group.
• I picked up Bioshock Tuesday (Circuit City had it on sale) on my way to my bi-weekly campus excursion. My meetings were over around 2pm so I headed home early, expecting to surprise the kids. But Jules had decided to skip naps and go shopping with them. Her cell phone was dead, so I ended up at home with a couple of hours to myself and a brand new copy of Bioshock. Wow, is that a good game. Certainly deserving of the amazingly good reviews it's garnered.
• Speaking of reviews, this transparently biased review of Bioshock over at Sony Defense Farce Force is frakking hilarious. Somehow, I doubt their dubious review will stem the tidal wave of Bioshock's well-deserved hype. Can't wait to read their Halo 3 review.
• Pat Helland writes at length on master-master replication. I reformated it into PDF so I could read it - the large images were messing up the text flow on my system. As usual for Pat, there's gold in that thar post. His thoughts on DAGs of versions and vector clocks as identifiers are very exciting. However, I think he glosses over the importance of declarative merging. I would think programmatic merge would likely be non-deterministic across nodes. If so, wouldn't you end up with two documents with the same vector-clock identifier by different data?
• Joe McKendrick points to a few people who predict the term "service-oriented" will eventually be subsumed under the general heading of "architecture". Not to brag, but I made that exact same prediction almost three years ago.
• Erik Johnson thinks that SOA 2.0 centers on transformational patterns. The idea (I think) is that if systems "understand each other more deeply", then we can build a "smarter stack" and design apps via new constructs to promote agility and simplicity. Personally, I'm skeptical that we can define unambiguously system semantics except in the simplest scenarios, but Erik talks about using "graph transformation mathematics" to encode semantics. I don't know anything about graph transformation mathematics, but at least Erik has progressed beyond hand waving to describing the "what". Here's looking forward to the "how".
• New dad Clemens Vasters somehow finds time to implement an XML-RPC binding for WCF 3.5. I was encouraged that it didn't require any custom attributes or extensions at the programmer level. Of course, XML-RPC fits semantically into WCF's interface based service model, so it shouldn't be a huge surprise that it didn't require any custom extensions. But did it need WCF 3.5? Would this work if recompiled against the 3.0 assemblies?
• Phil Haack writes a long post on Duck Typing. VB9 originally supported duck typing - the feature was called Dynamic Interfaces - when it was first announced, but it was subsequently cut. I was really looking forward to that feature. Between it and XML Literals, VB9 was really stepping out of C#'s shadow. I guess it still is, even without dynamic interfaces.
• Since I've been doing some LINQ to XML work lately, I decided to go back and re-write my code in VB9 using XML literals. While XML literals are nice, I don't think they're a must have. First, LINQ to XML has a nice fluent interface, so the literals don't give you that much cleaner code (though you do avoid writing XElement and XAttribute over and over.) Second, I find VB9's template syntax (like ASP <%= expression %>) clunky to work with, especially in nested templates. Finally, I like the namespace support of XNames better. As far as I can tell, VB9 defines namespaces with xmlns attributes just like XML does. So I'm not dying for literal XML support in a future version of C#. How about you?

Morning Coffee 112

• The Lee Holmes over at the Powershell Team Blog writes about alternatives to the "decades-old" Windows console host. Powershell Plus looks awesome. PoshConsole also looks pretty cool (though far from finished yet) and is free.
• WL ID Web Authentication SDK has been released. Details on the WL ID team blog. It looks like what Passport SDK provided for quite some time, but now it's free. There's also a client auth SDK in development. (via Dare Obasanjo)
• Libor Soucek leaps to the wrong conclusion about not differentiating enterprise & support systems. Of course, different systems will have different availability requirements. But what happens when we connect them together? We can't let the support system effect the availability of the enterprise system, right? To me, that implies either a) the support system now needs to conform to enterprise system availability requirements or b) we need some other mechanism (like async durable messaging) to act as a buffer between them. Personally, I like "b".
• Nick Carr points to an article The Trouble with Enterprise Software by Cynthia Rettig. Cynthia writes that while the massive complexity of enterprise software, especially large-scale ERP systems like SAP, significantly hinder it's value. It's a must read. Choice quotes:
• "It is estimated that for every 25% increase in complexity in the tasks to be automated, the complexity of the software solution itself rises by 100%."
• "The notion of reusable software works on a small scale. Programmers have successfully built and reused subroutines of standard functions. But as software grows more complex, reusability becomes a difficult or impossible task."
• "Hope, unfortunately, has never been a very effective strategy."
• "Is enterprise software just too complex to deliver on its promises? After all, enterprise systems were supposed to streamline and simplify business processes. Instead, they have brought high risks, uncertainty and a deeply worrying level of complexity. Rather than agility they have produced rigidity and unexpected barriers to change, a veritable glut of information containing myriad hidden errors, and a cloud of questions regarding their overall benefits."

Morning Coffee 111

• I'm not sure if I should laugh or cry at Nick Malik's definition of politecture. I mean, it's funny so I'm laughing, but it's so true that it makes me want to cry.
• Don Box comments on retiring the tenets. It's good to see him say "please God tell me we can do better" than CLR interfaces or WSDL.
• Looks like the P2P APIs are finally getting the managed treatment in .NET FX 3.5. A long time ago, John deVadoss asked me what an enterprise system like CRM might look like if it used a peer-to-peer approach instead of client-server. If I had any free time, I'd prototype one out on this API. (via Mike Taulty)
• Scott Guthrie goes back to his LINQ to SQL series to tackle Stored Procs and UDFs. Being able to use UDFs inline with LINQ queries is very cool. However, it seems to me that LINQ discourages the use of stored procs. As a developer, I'd rather write LINQ queries than stored procs, if I can. The probably puts me at odds with DBAs who'd rather all DB access be via stored procs they control.
• Soma writes about new MSBuild enhancements in VS08: multi-targeting and parallel build.
• I just discovered Vista Battery Saver. Basically, it turns off Aero and Sidebar when you're on battery. I'm traveling to Chicago next week, so we'll see if it has much impact on my battery life. (via Plenty of Code and Larkware)

The Durable Messaging Debate Continues

Last week, Nick Malik responded to Libor Soucek's advice to avoid durable messaging. Nick points out that while both durable and non-durable messaging requires some type of compensation logic (nothing is 100% foolproof because fools are so ingenious), the durable messaging compensation logic is significantly simpler.

This led to a very long conversation over on Libor's blog. Libor started by clarifying his original point, and then the two of them went back and forth chatting in the comments. It's been very respectful, Libor calls both Nick and I "clever and influential" though he also thinks we're wrong on this durable messaging thing. In my private emails with Libor, he's been equally respectful and his opinion is very well thought out, though obviously I think he's the one who's wrong. :)

I'm not sure how much is clear from Libor's public posts, but it looks like most of his recent experience comes from building trading exchanges. According to his about page, he's been building electronic trading systems since 2002. While I have very little experience in that domain, I can see very clearly how the highly redundant, reliable multicast approach that he describes would be a very good if not the best solution.

But there is no system inside Microsoft IT that looks even remotely like a trading exchange. Furthermore, I don't think approaches for building a trading exchange generalize well. So that means Nick and I have very different priorities than Libor, something that seems to materialize as a significant amount of talking past each other. As much as I respect Libor, I can't shake the feeling that he doesn't "get" my priorities and I wouldn't be at all surprised if he felt the same way about me.

The biggest problem with his highly redundant approach is the sheer cost when you consider the large number of systems involved. According to Nick, MSIT has "over 2700 applications in 82 solution domains". When you consider the cost for taking a highly redundant approach across that many applications, the cost gets out of control very quickly. Nick estimates that the support staff cost alone for tripling our hardware infrastructure to make it highly redundant would be around half a billion dollars a year. And that doesn't include hardware acquisition costs, electricity costs, real-estate costs (gotta put all those servers somewhere) or any other costs. The impact to Microsoft's bottom line would be enormous, for what Nick calls "negligible or invisible" benefit.

There's no question that high availability costs big money. I just asked Dale about it, and he said that in his opinion going above 99.9% availability increases costs "nearly exponentially". He estimates just going from 99% to 99.9% doubles the cost. 99% availability is almost 15 minutes of downtime per day (on average). 99.9% is about 90 seconds downtime per day (again, on average). 

How much is that 13 extra minutes of uptime per day worth? I would say "depends on the application". How many of the 2700 applications Nick mentions need even 99% availability? Certainly some do, but I would guess that less than 10% of those systems need better than 99% availability. What pretty much all of them actually need is high reliability, which is to say they need to work even in the face of "hostile or unexpected circumstances" (like system failures and downtime).

High availability implies high reliability. However, the reverse is not true. You can build systems to gracefully handle failures without the cost overhead of highly redundant infrastructure intended to avoid failures. Personally, I think the best way to build such highly reliable yet not highly available systems is to use durable messaging, though I'm sure there are other ways.

This is probably the biggest difference between Libor and me. I am actively looking to trade away availability (not reliability) in return for lowering the cost of building and running a system. To someone who builds electronic trading systems like Libor, that probably sounds completely wrongheaded. But an electronic trading system would fall into the minority of systems that need high availability (ultra high five nines of availability in this case). For the systems that actually do need high availability, you have to invest in redundancy to get it. But for the rest of the systems, there's a less costly way to get the reliability you need: Durable Messaging.

Morning Coffee 102

Seems like a slow week.

• Jules and I went to see the latest Harry Potter movie this past weekend. It's easily the weakest of the six HP stories so far. The first two stories were about discovering this magical world, the next two about discovering Harry's past, and the last two about confronting said past. That leaves OotP as the odd-story-out, mostly bridging from the end of the fourth story to the start of the sixth.
• Speaking of movies, the new movie feature of Mobile Search v2 rocks, though I have two quick suggestions. First, it would be nice to have a time-sorted view of when a given movie is playing. So if it's playing at 4pm at one theater and 4:30pm at another, you'd see them in a list ordered that way. Second, how about an option to buy tickets directly from the phone?
• If you're interested in WPF and 3D, Eric Sink has a series for you.
• Old news, but Windows Home Server RTMed on Monday. I'm really looking forward to this product.
• I was looking for some information on how WCF pumps messages in the service host and I found this post from Maheshwar Jayaraman. Between that post and Reflector, I think I've got a good handle on how ChannelDispatcher works.
• Larry O'Brein calls out three MS Research Projects. Microsoft Research Accelerator is a high-level data-parallel library that targets GPUs. Graph Layout Execution Engine (aka GLEE) is a library for graph layout and viewing. VirtualEarth MapCruncher converts existing maps (PDF and bitmaps) to work with Virtual Earth.
• Ted Neward weighs in on the David Chappell's Korean War REST vs. WS-* analogy. Skim the history lesson, but make sure you read his points about security and reliability interop. WS-* has addressed these areas, so if you need those capabilities, why wouldn't you use WS-* to get them rather than re-invent the wheel? As for the history lesson, Ted does say he thinks software development is more analogous to making war than building a house. He expands on that idea and recommends Robert Greene's The 33 Strategies of War. I want to read the book and mull it over a bit, but I certainly see where Ted's coming from.

Morning Coffee 99

• Mladen Prajdic has a great post on handling a database in your unit tests. He mentions NDbUnit but seems mostly to favor SQL 2005's database snapshot feature. He's got sample code for creating and restoring a snapshot. (via DNK)
• Microsoft Robotics Studio 1.5 released yesterday. Tandy Trower - GM of the Robotics group - has the details on what's new.
• Herb Sutter has a new column in Dr. Dobbs on concurrency. First up, "building a consistent mental model for reasoning about concurrency". Sounds like a must read column. (via LtU)
• Scott Hanselman describes "Sez You Architecture". I wonder, do architecture ninjas get to wear a Shinobi shozoku?
• From the Not Everyone Agrees With DevHawk Dept.: Libor Soucek disagrees with me and thinks that durable messaging should be avoided. I had a hard time following Libor's logic but needless to say, I disagree with his disagreement. He writes that one of the reasons to use DM is for "Cooperating on transaction with external system". While multiple systems may be cooperating on a business transaction, in no way do I believe they are going to cooperate on a database transaction. But since he started talking about the DTC, I suspect we're talking past each other. Libor, drop me a line and we can discuss further.

Morning Coffee 92

• Brad Wilson blogs about SvnBridge, a tool that lets you use Subversion clients like TortoiseSVN to talk to Team Foundation Server. While I think that's cool, I wonder is anyone interested in subversion clients other than TortoiseSVN? For example, will people choose AnkhSVN instead of the Team Explorer Client?
• Speaking of TortoiseSVN, I wonder if those guys are interested in building a TortoiseTFS project? I did find two other TFS shell extensions projects: Dubbelbock TFS and Turtle, though neither appears as full featured as Tortoise.
• Scott Guthrie details VS08's multi-targeting support. Of course, the three versions of the .NET Framework VS08 can target all use the same underlying runtime, which probably made it easier to build.
• Michael Platt refactors Don Box's original tenets of service orientation so he can include some information about how these services get built.
• Scott Hanselman tackles the tricky question of assembly granularity.
• PowerShell Analyzer is now available for purchase. Among other things your $59 gets you, besides a 50% savings, is "Feature request priority". That's pretty cool. I wonder how many other micro-ISV's take the approach of "pay me now and you get to help me pick some of the new features."
• Brandon LeBlanc writes about dual monitor support in Vista. I'm loving the dual monitor support, though I have a somewhat strange setup. I keep my primary monitor rotated in portrait mode, which is great for reading and writing. I typically use my second monitor for blogs and mail. I even wrote a custom multi-mon wallpaper utility so I could easily generate new wallpapers for my non-standard monitor layout, including bitmap rotate support. If there's interest, I can post it. (via Sam Gentile)
• Nick Malik continues to write about Mort, with the usual response from the usual folks. I liked his point that "You cannot fight economics with education", but otherwise I'm staying out of this discussion.
• In the same vein, Martin Fowler writes about Technical Debt. I completely agree with his hypothesis that short changing design may save time in the short term but will cost much more in the long term. However, the problem is that the people who are making the tradeoff - i.e. the people paying for the project NOT the people building the project - either don't understand the tradeoff or are more than happy to sacrifice the long term cost for the short term gain. How are most projects measured? Being on time and on budget with the planned set of features. Very few projects - and none that I've ever seen - are goaled on long term maintainability. Until you can change that, this issue will continue to linger.

Morning Doughnuts 11

Harry will be back on Monday so I will returning to blogging on my website, while I will let the expert return to his normal posts here (not that he really took a break). I agree with Harry's post in that I really want to get something built so that we can talk about more than theoretical models. Like last time I appreciate the opportunity to sub for the master this last week. I hope that you found some of my entries interesting.

• Sam Gentile wrote the other day why it's great to be a Microsoft developer. I enjoyed that post because I just celebrated the end of my first year here at Microsoft. At this point I am not sure what I have contributed, but I have learned a great deal and want to apply that knowledge over the next year to help the company to succeed. We really do have great people and great technologies.
• The Seattle/Oklahoma City Sonics hired a GM who is only 30 years old. You know you must be getting old when the people running the sports teams are younger than you. He comes from the Spurs organization though so at least he has a background from a successful franchise.
• Ben Pearce listed out his top 5 questions about PowerShell this week at TechEd. He also recommends the book "PowerShell in Action" by Bruce Payette. I heartedly agree with this endorsement as the book is excellent.
• It looks like there are going to be more family friendly games for the XBOX360. I for one am glad to hear that. The other day as I was trying to find some games my 4 year old with the broken leg could play I realized how many games I have that wouldn't be appropriate for him. This is very good news in my opinion.

Morning Doughnuts 10

• I am a big fan of PowerShell, and I know Harry likes it as well. Of course I have aliased many of the commands so they appear more Unix like. I mention this because David Aiken mentions a new product for PowerShell called NetCmdlets produced by N Software. I downloaded a trial and have been impressed so far. If you use PowerShell it might be worth giving this a look.
• The New Yorker has an interesting article about feature choices in technology. Basically it comes down to customers choose products with more features and customization options if given the choice, but when they actually have to use the products they prefer simplicity. I think that is something lost on those of us who design things. Giving users every possible option can make our products seem more difficult to use over simpler less feature rich choices. (via Coding Horror)
• Well it took longer than some may have guessed, but Microsoft has been sued over the name Vista by a French TV company. Apparently they were going to launch a television station with the same name. I wonder what the courts will decide since apparently the TV company didn't register the name in the software category.

Making My Mark at TechEd

One of the things that's different about being in MSIT is that it's cut my travel dramatically. Basically, the only travel I've done since taking this job was Thomas Erl's SOA Workshop last September. So it came as a bit of a surprise when I got tapped to present at TechEd (at fairly close to the last minute).

The last year I ran the architecture track, all track owners were asked to include either a globalization or MSIT session in our track. Since then, MSIT has expanded it's role at TechEd dramatically, with 14 breakouts, 20 chalk talks and our own Technical Learning Center.

I'm doing a two chalk talks on my MSIT project, Rome. I mentioned the project back when I switched jobs, but we've never talked about the project by name before. We haven't accomplished quite as much as I'd hoped since then, but we've progressed to the point that we can talk publicly about what we're doing. Now that we've begun to open the kimono a bit, you should see more on Rome, Not only from me but also my teammates who blog: Dale, Rick and Dottie.

So if you're going to TechEd, make sure you stop by the MSIT Technical Learning Center and say hi. Unlike my last two trips to TechEd, I have very limited responsibilities this time - basically just show up on time and talk about what I do all day, twice - so that leaves plenty of time to attend sessions, chat up attendees and ride roller coasters. Hope to see you there.

ROME: Service Oriented Infrastructure for the Enterprise
Like most enterprises, Microsoft IT is adopting a service oriented approach for the development of their internal systems. However, in order to avoid projects building similar service infrastructures on a per application basis, MSIT realizes the need for a common service oriented infrastructure to build these systems upon. Inside MSIT, the Rome project is tasked with designing, building and maintaining the infrastructure for these services. Come chat with Harry Pierson, lead architect of the Rome project, and discuss the challenges of building a large-scale service infrastructure inside a large enterprise like Microsoft.

• MST02-TLC Monday (June 4th) 1:15-2:30pm
• MST14-TLC Thursday (June 7th) 1-2:15pm

Morning Coffee 64

• I took my son to the Pacific Science Center this past weekend to see the Grossology exibit. His favorite was the barfing machine, but we also got to play Urine: The Videogame, stand inside a giant nose, work the burping machine, climb the scab wall and slide down the throat into the stomach and come out thru the colon. We also checked out the dinosaur exibit (Patrick's favorite part: petrified dino poop), Kids Works and the Peter and the Wolf laser show. It was awesome.
• This week's Big News^tm is the rebranding of WPF/E as Silverlight. Tim Sneath has the rundown, including the news that more news about Silverlight is coming at MIX. Personally, I think Silverlight is a great name. I was worried it was going to be another W*F name. (I'm waiting for the day that MSFT marketing tries to rebrand Win32 as the Windows Technical Foundation).
• Gianpaolo Carraro writes about what happens when a SaaS company bites the dust - i.e. "what happens to my data?" I expect that this is one of the aspects of SaaS that you have to weigh, though I doubt most companies will explicitly think about what happens if their SaaS provider goes belly up. As the SaaS market expands and more companies will go belly up (I agree w/ GP 100% that this isn't SaaS specific, rather a natural force of any market) how much will that drag on SaaS adoption? I'm thinking it'll be a fairly significant drag, but the SaaS market will eventually rebound.
• Nick Malik picks up the decentralization meme is started last Friday and compares enterprise architecture to city zoning boards. In general, I agree with Nick's "not in a vacuum and not with a heavy hand" comment, but even more with his point that "we haven't figured all it out yet." Most EA efforts I've seen have been heavy handed and fairly divorced from reality (aka in a vacuum). More on this topic in the future.
• Kirk Evans closes the loop on city planning with a reference to Pat Helland's Metropolis work. Pat's work has been a huge influence on me. I often repeat Pat's point about cities being "interconnected yet independent". Services need to be interconnected yet independent also.
• Roger Wolter has a new Master Data Management whitepaper out, this one MDM hubs. I was literally talking about MDM on a conf call this morning, so Roger's timing is impeccable.
• I haven't read RADAR Architecture yet, but the fact that DAR in the acronym stands for "Dumb-Ass Recipient" made me laugh. (via Sam Gentile)
• MIT Media Lab has cointed the latest 2.0-ism: Human 2.0. I love Nick Carr's take on it: "We're definitely overdue for an upgrade - it seems like we've been stuck in Version 1.x for a few hundred thousand years, and that was after a beta that went on for freaking ever. Still, I think I'll probably hold off until 2.01 or 2.02. I don't want to be on the bleeding edge for this one."

Morning Coffee 61

• Nick Malik wonders if architecture is code or if it's data? Frankly, I have nothing to add to this, but thought I should link to something Nick wrote since he's letting me share his office for the next few months while I'm engaged with one of the teams he mentions, though he begged me not to disclose which one. :)
• Ted Neward's Five Minute Management Lessons for Developers made me snicker.
• Xbox.com is running a new contest called "My Mom's a Gamer". Mine is. These days, it's mostly casual games on MSN Games, but back in the day she played both the Atari 2600 and Colecovision. She would play Space Invaders for hours. And curse. A lot. Most kids learned to curse on the playground, I learned from my mother.
• Mark Cuban claims the HDTV is the new PC. TV and PC technologies are certainly evolving as they merge, but will that platform be as open as the desktop PC or the browser? It better be.
• According to Nick Carr, Citigroup is looking to cut $4.6 billion in spending over the next three years and that IT will be one of the "cornerstones" (i.e. hardest hit) of that effort. I had a chat with an Meta analyst in Australia a few years ago who suggested that IT spending was going to go thru an innovator's dilemma phase. Huge companies (like Citi) with huge IT budgets are facing significant competition from small companies that can't afford huge IT budgets. These smaller companies get used to running a tighter ship and tend to be more competitive as they grow and are able to directly face off against the big fish.

Morning Coffee 52

• I finally found a use for the free SOA book I got from attending that Thomas Erl workshop. I'm using it to prop up one end of my daughter's mattress while she's sick so she can sleep better.
• Jeff Tash states axiomatically that CASE has evolved into Enterprise Architecture. I agree with his points about why software construction isn't like manufacturing, but he seems to be describing BDUF rather than EA. I'm anti-BDUF too, but why blame EA? (via John deVadoss)
• Joe McKendrick comments on my SaaS/SOA post and wonders if SOA should stand for "SOA Oriented Architecture". He also writes that most organizations these days don't have an SOA, they have an AOS, "Agglomeration Of Services". So true, so true.
• JD Meier talks up the new VSTS guidance available on CodePlex. Looks like some good stuff in there. I like how the p&p guys are moving from documents to wikis to deliver their guidance.
• I've held off on getting the HD-DVD drive for my Xbox, but I think I'm going to cave soon, where soon == about two months. That's when The Matrix Trilogy is released on HD-DVD. Right around my birthday too, how convienent.

When is a Service Not a Service?

Conceptually, I like both Service Oriented Architecture (aka SOA) and Software as a Service (aka SaaS). However, I think we've done the industry a disservice by overloading the term "service".

John deVadoss likes the following definition of SaaS from Wikipedia. So do I.

Software as a service (SaaS) is a model of software delivery where the software company provides maintenance, daily technical operation, and support for the software provided to their client. SaaS is a model of software delivery rather than a market segment; it assumes the software is delivered over the Internet. Software can be delivered using this method to any market segment including home consumers, small business, medium and large business.

To paraphrase, SaaS is software that traditionally you might have bought, installed and run yourself but instead now can access over the network where someone else is responsible for installing and running it. For example, instead of buying, setting up and managing my own mail server to handle a single @devhawk.net email address, I use the WL Custom Domains service.

SOA on the other hand isn't a model of software delivery, it's a model of software segmentation. Again, here's the Wikipedia definition, this time for SOA:

There is no widely-agreed upon definition of Service-oriented architecture other than its literal translation that it is an architecture that relies on service-orientation as its fundamental design principle.

Err, that's not very helpful. Let's check out the OASIS definition (cribbed from Wikipedia).

[SOA is] A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

Well, at least it's not a self-referential recursive definition. But it is littered with committee-speak. (Who talks like that in real life?) Frak it, here's my definition:

SOA is a way of implementing IT systems as a web of interconnected yet independent loosely coupled subsystems (typically called services) instead of as big honking systems we have traditionally built that tend to be unwieldy, in-agile, difficult to change and probably obsolete by the time they were deployed.

We could argue about the language, but you get the point. There would be a ton of argument about the size of the subsystems (i.e. the service granularity), but I think most people can agree that SOA encourages building multiple smaller interconnected subsystems instead of one big (honking) system.

Which brings me back to my original point: Service, in the SOA sense, describes the approach to factoring parts of an software solution. Service, in the SaaS sense, describes a software delivery mechanism. Certainly, you can use both together and take an SOA approach to building a SaaS product. But you don't have to. So having the same term "service" used in both is very confusing.

How many SaaS products use SOA today? I would guess "not many" since there hasn't been much demand for it. When you're selling to the long tail of the LOB market, support for service-oriented integration isn't a critical selling feature. As SaaS becomes more attractive to larger companies (i.e. ones with dedicated IT staffs), using a SOA approach will be more important to SaaS product vendors. So they will converge in a way, but not in the way their naming suggests.

Of the two uses, SaaS seems closer to the dictionary definition of service. Maybe the S in SOA should stand for "Subsystem"? Nah, I like the term "connected systems" better than "service oriented" anyway.

Morning Coffee 49

• The eBay Architecture SD Forum presentation that spawned the whole Transactionless meme is available here. As I reported yesterday, it doesn't call for going completely transactionless as Martin suggested. It calls for going without distributed transactions, which I agree with 100%.
• More interesting than the transactional aspects, I found the data tier functional segmentation information facinating. Too bad those guys aren't using our platform, SSB was expressly designed for exactly this sort of segmentation. I also liked that step 1 for "massively scaling J2EE" is to "throw out most of J2EE".
• After going mostly dark since last august, the manager of my old team John deVadoss has been blogging up a storm since the beginning of March. So has my old boss Mike Platt. I wonder what happened at the begining of March? Here's hoping this blogging fever spreads on my old team.
• Joe McKendrick: "The bottom line is that ROI on SOA is an enterprise challenge, not an IT challenge." Truer words are rarely spoken.
• The rumor mill on the Black Xbox 360 "Elite" are coming fast and furious. I don't care about the HDMI port (my HDTV is five years old and doesn't have one) but I would like a bigger hard drive...

Morning Coffee 43

• This week is the MVP summit. Hopefully, I'll make it over there and see the Architect MVPs. Otherwise, things seem sort of quiet in the Microsoft wing of the blogoshphere.
• Saw 300 yesterday on a relatively rare day out with just Jules. Really enjoyed it.
• Nick Malik describes what goes into an enterprise platform roadmap.
• Joe McKendrick recaps a vendor SOA suites podcast. With SOA, you can revisit your platform decision on a project by project basis which allows you to avoid the dreaded "vendor lock-in". But supporting lots of platforms is an operational nightmare, so maybe lock-in isn't as bad as it sounds.

Morning Doughnuts 6

• The Wall Street Journal is reporting that Massachusetts lawmakers are considering a bill to punish retailers for leaks in personal data. I wonder how long it will take for a law like that to go nationwide? Looks like there may be some good jobs in retail IT data security opening up shortly.
• There is an interesting debate on the SAAS architecture in Dr. Dobb's Portal. The money quote for me was as follows:

"Ajax and Web 2.0 are great technologies for casual use, but for mission critical you need the capabilities of a desktop app," RightNow CEO Greg Gianforte says.

I have to admit I don't agree with that quote at all. It seems pretty shortsighted in minimizing the capabilities of web based applications.

• As a follow-up to yesterdays entry about the 12 steps to overcome email addiction here is a 12 step program to help you overcome being a SOAholic. There are also some symptoms you can look for to see if you are a SOAholic.
• Ram Ravishankar posts on if SOA requires web services. He makes pretty good arguments for an against a SOA requiring web services and ultimately doesn't answer the question. I would say that a SOA doesn't require web services, but it is very likely in the range of 90% plus that a SOA within a company is going to have at least some web services in it.
• Harry returns from his secret mission and will be back blogging on Monday. I have really enjoyed stepping in being a replacement blogger this week. While my take on technology is a bit different that Harry's I hope that my entries were interesting and offered a bit of a different perspective on IT.

Morning Doughnuts 3

• What does it take to be an architect? Skyscrapr.net attempts to answer this question by asking a bunch of architects.
• I have started teaching my children about astronomy. I found an open source product called Stellarium that is excellent for learning about the celestial objects visible in your area.
• A Methodology for SOA adoption? I read an interesting blog on this subject from a couple of weeks ago. It's not a long article, but the author makes some interesting points including an outline for SOA adoption.
• I finally picked up Gears of War on Friday. It really isn't a game I can see playing much, although I can see why it's popular. I guess the best and the worst part of the game is having to utilize cover so you don't die right away.
• Windows Live Writer is a great tool! I use it to author the blogs for my website, and this week I have been using it on these Morning Doughnuts posts. My favorite feature is that you can preview your post and see exactly how it will appear on your website. This has been particularly useful since Devhawk and my site look quite different.

Morning Doughnuts 1

Introduction

While Harry is out of the office on a well deserved vacation with his family I will be acting as a guest blogger for his website.

My name is Dale Churchward and I have been working at in Microsoft IT as an application architect since June 2006. Prior to coming to MS I worked in operations in both the telecommunications and healthcare industries. As Harry pointed out I tend to be very operations focused. While I love working on new technologies I am keenly interested in how a design will work in production, and try to ensure we have considered support considerations as part of our designs. Each member of our team comes in with a different background. This helps us as we are each strong in different areas. I also write to my own blog which can be found here. In my own blog I tend to write on a wide array of subjects, depending on my mood at the time. While Harry is away though, I will be posting on technology areas I am interested in, and staying true to his morning coffee vision, albeit with a slightly different take.

The Doughnuts

• The Build Master by Vincent Maraia is an excellent book if you are interested in the build process and how to make it as efficient as possible.
• We had a great meeting with the Patterns & Practices team the other day. Since I am still new to Microsoft it is still a bit overwhelming to meet the authors of documents you have read and used over the years.
• I recently have been spending some cycles working with System Center Operations Manager 2007. I believe that it provides some excellent tools to monitor and repair a system plus it's designed to be service focused.
• Francis Stokes has produced 6 episodes showing what would happen if heaven was being run like a company named God Inc. There are currently 6 episodes. No matter what your belief or lack thereof in a supreme being the videos are hilarious.
  • Episode 1
  • Episode 2
  • Episode 3
  • Episode 4
  • Episode 5
  • Episode 6 
• I have been spending a lot of time thinking about how heartbeat transactions between multiple services should operate. In the drawing below you can see 3 web services and a monitoring one. In the original design the monitoring service was sending heartbeats out to each of the web services to see if they were available. This seem inefficient to me as we really don't care if the monitoring service can reach the web service. What we need to know is if any dependent web services are able to connect. In the drawing we have a web service residing in the extranet (Web Service 1) that sends data to a web service in the corporate network (Web Service 2). We really don't care if the monitoring service can talk with web service 2, but we definitely want to know that web service 1 can get there. Once web service 1 realizes that is can't connect to 2 we then notify the monitoring system so that the owner of web service 2 can take action. Web service 1 still continues sending heartbeats though so that it is aware of when the second web service becomes available again.

Morning Coffee 28

• From the "Ask and ye shall receive" department: A couple weeks ago I wondered how good or bad my Gamerscore conversion rate is. MyGamerCard.net just launched a completion leaderboard where they rank you on your Gamerscore times your completion rate.
• Shane Courtrille pointed out that the prize you receive in from the Xbox Rewards program gets better if your Gamerscore is higher. With a meager 1090 points, I'm in level 1. But those with 10,000+ or more can get a copy of Fuzion Frenzy 2 for completing the challenge.
• Yesterday, I complained that code in my RSS feed looks awful. It appears to be a problem with dasBlog. In validating the HTML is actually XHTML, it screws up the white space. Of course, usually that's not a big deal, but inside a <pre> tag, it is. Until I get a chance to submit a patch to dasBlog to fix this, I'm using CodeHTMLer, which has a "convert white space" option that doesn't use the <pre> tag at all. As a bonus, it even support PowerShell! Note, you have to use the website, not their WLWriter plugin, if you want the convert white space option.
• There's a new beta of Ruby.NET available. Now that I've moved on to PowerShell, I'm only slightly interested in Ruby these days. If I can figure out how to create internal DSLs with PS, what would I need Ruby for? (via Larkware)
• My old team just shipped a single-instance multi-tenancy SaaS sample called LitwareHR. Details are on Gianpaolo's blog, code is up on CodePlex.

Morning Coffee 18

• I'm sure glad Heroes is back. And so far, I'm liking this season of 24 better than the last couple. I especially like how they're introducing Jack's family to the storyline - very cool.
• Bill Gates will be on The Daily Show with Jon Stewart next Monday. I rarely miss an episode (though I typically watch it a day or two later via my DVR) so I'm looking forward to this.
• Roger Sessions latest ObjectWatch newsletter is available. Roger does his usual brand of relating architectural concepts to everyday situations (this time, planning his daughter's wedding). This one was funnier than most - especially since I have a 1 year old daughter - though I'm still waiting for one of these that doesn't contain the words "doppio machiato". If you're reading this Roger, don't worry, I am enjoying every moment.
• Apparently, I am the keeper of obscure development knowledge on my team. My teammate Buzz was getting an error in shdocvw.dll when trying to open an XSD with the BizTalk editor. He's on an interim build of BTS06 R2, so bugs are to be expected, but he wasn't sure what shdocvw was, so he asked me. In case you're curious, shdocvw is the WebBrowser control.
• Did I mention that I left my laptop power cord at the office again on Tuesday? That's three times in four the last five trips to my office (not counting weekends, sick days and training). My boss actually got a spare from his boss that I can leave at the house 24/7.

Common Ground with My Conservative Teammates

I came in this morning to discover my boss and next cube neighbor Rick had decided to spruce up his cube with camo netting. He's ex-Army, so it's not like it's out of character for him. Of course, the camo netting has the exact opposite of it's indented effect, making Rick's cube very easy to find in the farm.

Unlike my last team, most of my teammates are conservatives. But apparently we can find common ground in our opinions of Defense Secretary Rumsfeld. Rick called him an abysmal failure. I couldn't agree more. Dale joked that Rumsfeld was joining our team and moving into Rick's newly camo festooned cube. Rick countered that Rumsfeld was actually joining the Enterprise Architecture group. Heh.

Update: Dale points out he made the joke about Rumsfeld joining EA, not Rick. My bad Dale.

The Two Types of Service Architects

Tomas Restrepo comments on my recent SSB and WCF posts:

Harry Pierson asks how well WCF supports long running tasks. He suggests that WCF does not support them very well, and says that's one reason he likes SQL Server Service Broker so much. I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database. Sure, this is an "expanded universe" with the CLR support in SQL Server 2005, but even so it makes me nervous at times .

You could also consider using a custom service with MSMQ or something like BizTalk Server for this if you had long running processes that were not completely tied to the DB (or a single DB for that matter).

Sam Gentile follows up:

In that same post, but I needed to call it out separate, Tomas rightfully says, "I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database," in response to Harry liking Service Broker so much. Talk about a narrow edge case. That's way I never really got excited or cared about Service Broker. Its a narrow solution to a special edge case when everything is database driven and can be executed totally inside the database. That's the old Microsoft Data-Driven Architecture for sure. Me, I'd rather have a rich Domain-Driven architecture most of the time. Then if you have Oracle databases in your architecture too, where does it leave you? Nowhere.

As you might expect, I have a few comments,  clarifications and corrections.

First, Tomas' statement that Service Broker only supports service logic "executed completely within the database" in flat out wrong. Service Broker can be used from any environment that can connect to SQL Server and execute DML statements. If you can call SELECT/INSERT/UPDATE/DELETE, then you can also call BEGIN DIALOG/SEND/RECEIVE/END CONVERSATION. This includes Windows apps and services, web apps and services, console apps and even Java apps. Of course, you can also access Service Broker from stored procedures if you wish, but you're not limited to them as Tomas suggested.

Tomas' misconception may come from a feature of Service Broker called Activation. Activation is a feature of Service Broker that dynamically scales message processing to match demand. For example, Service Broker can be configured to launch a new instance of a specified stored procedure if messaging processing isn't keeping up with incoming message traffic on a given queue. This is called internal activation and because it uses stored procedures it does execute within the database as Thomas said. Service Broker also supports external activation where it notifies an external application when activation is needed. You do have to build an application to host your service logic and handle these notifications, but that application doesn't execute within the database. So while you could argue that it's easier to execute your service logic within the database (no need to build a separate host app), it's not required.

Given that you don't have host your service logic in the database, then you're also not limited to "a single DB" as Tomas suggests. You don't, in fact, have to put your Service Broker queues in the same database with your business data. So if you have Oracle in your environment, like the scenario Sam mentioned, you would host your service logic in an external application that processed messages from a queue in a SQL 2005 database while accessing and modifying business data from tables in the Oracle database. Using multiple databases does require using distributed instead of local transactions, but if you're using MSMQ as Tomas recommended, you're already stuck with the DTC anyway.

Finally, I didn't get Tomas' "purely database driven" or Sam's "everything is database driven" comments at all. While there are exceptions, the vast majority of systems I've ever seen/built/designed have essentially been one or more stateless tiers sitting in front of a stateful database. If it's a traditional three tier web app, there's a stateless presentation tier, a stateless business logic tier and a stateless data access logic tier. For a web service, there's no presentation tier, but there's is the stateless SOAP processing tier typically provided by the web service stack. Does this mean the vast majority of web apps and services are  "purely database driven" too? If so, then I guess it's a good thing, right?

In the end, maybe there are two types of service architects - those that believe the majority of services will be atomic and those that believe the majority of services will be long running. For atomic services, Service Broker is overkill. But if it turns out that most services are long running, WCF's lack of support is going to be a pretty big roadblock.

I'm obviously in the long running camp. I'm not sure, but I get the feeling this is the less popular camp, at least for now. We'll have to wait to see, but I do know is that whenever someone brings me what they think is an atomic business scenario, it doesn't take much digging to reveal that the atomic scenario is actually a single step of a long running business scenario that also needs to be automated.

Here's a question for Tomas, Sam and the rest of you: Which group do you self select into? Are most services going to be atomic or long running in the (pardon the pun) long run?

Essential Windows Workflow Foundation

On Don's recommendation, I picked up Essential WF. In the forward, Don writes "[S]omething big is about to happen." I'm only part way thru chapter one, and this is already a must read. Go get it. Now.

In the preface, they define the term "Reactive Program", which I'm adding to my personal lexicon.

"Windows Workflow Foundation (WF) is a general-purpose programming framework for creating reactive programs that act in response to stimulus from external entities. The basic characteristic of reactive programs is that they pause during their execution, for unknown amounts of time, awaiting input."

That "unknown amounts of time" is the kicker. Here's a paragraph from early in chapter one that expands on that:

"Real-world processes take a long time - days, weeks, or even months. It is wishful thinking to assume that the operating system process (or CLR application domain) in which the program begins execution will survive for the required duration."

Gee, that sounds familiar doesn't it?

Is WCF "Straightforward" for Long Running Tasks?

My father sent me a link to this article on SOA scalability. He thought it was pretty good until he got to this paragraph:

Long-running tasks become more complex. You cannot assume that your client can maintain a consistent connection to your web service throughout the life of a task that takes 15 minutes, much less one hour or two days. In this case, you need to implement a solution that follows a full-duplex pattern (where your client is also a service and gets notified when the task is completed) or a polling scheme (where your client checks back later to get the results). Both of these solutions require stateful services. This full-duplex pattern becomes straightforward to implement using the Windows Communications Foundation (Indigo) included with .NET 3.0.

When I first saw duplex channels in WCF, I figured you can use them for long running tasks also. Turns out that of the nine standard WCF bindings, only four support duplex contracts. Of those four, one is designed for peer-to-peer scenarios and one uses named pipes so it doesn't work across the network, so they're obviously not usable in the article's scenario. NetTcp can only provide duplex contracts within the scope of a consistent connection, which the author has already ruled out as a solution. That leaves wsDualHttp, which is implemented much as the author describes, where both client and the service are listening on the network for messages. There's even a standard binding element - Composite Duplex - which ties two one-way messaging channels into a duplex channel.

Alas, the wsDualHttp solution has a few flaws that render it - in my opinion at least - unusable for exactly these sorts of long running scenarios. On the client side, while you can specify the ClientBaseAddress, you can't specify the entire ListenUri. Instead, wsDualHttp generates a random guid and tacks it on the end of your ClientBaseAddress, effectively creating a random url every time you run the client app. So if you shut down and restart your client app, you're now listening on a different url than the one the service is going to send messages to and the connection is broken. Oops.

The issues don't end there. On the service side of a duplex contract, you get an object you can use to call back to the client via OperationContext.Current.GetCallbackChannel. This works fine, as long as you don't have to shut down your service. There's no way to persist the callback channel information to disk and later recreate it. So if you shut down and restart your service, there's no way to reconnect with the client, even if they haven't changed the url they're listening on. Oops.

So in other words, WCF can do long running services using the wsDualHttp binding, as long as you don't restart the client or service during the conversation. Because that would never ever happen, right?

This is part of the reason why I'm sold on Service Broker. From where I sit, it looks like WCF can't handle long running operations at all - at least, not with any of the built in transports and bindings. You may be able to build something custom that would work for long running services, I'm not a deep enough expert on WCF to know. From reading what Nicholas Allen has to say about CompositeDuplex, I'm fairly sure you could work around the client url issue if you built a custom binding element to set the ListenUriBaseAddress. But I have no idea how to deal with the service callback channel issue. It doesn't appear that the necessary plumbing is there at all to persist and rehydrate the callback channel. If you can't do that, I don't see how you can reliably support long running services.

The Other Foundation Technology

I mentioned last week that WF "is one of two foundation technologies that my project absolutely depends on". Sam Gentile assumes the other foundation technology is WCF. It's not.

As a quick reminder, my day job these days is to architect and deliver shared service-oriented infrastructure for Microsoft's IT division. These services will be automating long running business operations. And when I say long running, I mean days, weeks or longer. While there will surely be some atomic or stateless services, I expect most of the services we build will be long running. Thus, the infrastructure I'm responsible for has to enable and support long running services.

The other foundation technology my project depends on is Service Broker. Service Broker was expressly designed for building these types of long running services. It supports several capabilities that I consider absolutely critical for long running services:

• Service Initiated Interaction. Polling for changes is inefficient. Long running operations need support for the Solicit-Response and/or Notification message exchange patterns.
• Durable Messaging. The first fallacy of distributed computing is that the network is reliable. If you need to be 100% sure the message gets delivered, you have to write it to disk on both sides.
• Service Instance Dehydration. It's both dangerous and inefficient to keep an instance of a long running service in memory when it's idle. In order to maximize integrity (i.e. service instances survive a system crash) as well as resource utilization (i.e. we're not wasting memory/CPU/etc on idle service instances), service instances must be dehydrated to disk.

In addition to these capabilities, Service Broker supports something called Conversation Group Locking, which turns out to be important when building highly scalable long running services. Furthermore, my understanding is that Conversation Group Locking is a feature unique to Service Broker, not only across Microsoft's products but across the industry. Basically, it means that inbound messages for a specific long running service instance are locked so they can't be processed on more than one thread at a time.

Here's an example: let's say I'm processing a Cancel Order message for a specific order when the Ready to Ship message arrives for that order arrives. With Conversation Group Locking, the Ready to Ship message stays locked in the queue until the Cancel Order message transaction is complete, regardless of the number of service threads there are. Without Conversation Group Locking, the Ready to Ship message might get processed by another service thread at the same time the Cancel Order message is being processed. The customer might get notified that the cancellation succeeded while the shipping service gets notified to ship the product. Oops.

There's also an almost-natural fit between Service Broker and Windows Workflow. For example, a Service Broker Conversation Group and a WorkflowInstance are roughly analogous. They even both use a Guid for identification, making mapping between Conversation Group and WF Instance simple and direct. I was able to get prototype Service Broker / WF integration up and running in about a day. I'll post more on that integration later this week.

Last but not least, Service Broker is wicked fast. Unfortunately, I don't have any public benchmarks to point to, but the Service Broker team told me about a private customer benchmark that handled almost 9,000 messages per second! One of the reasons Service Broker is so fast is because it's integrated into SQL Server 2005, which is is pretty fast in it's own right. Since Service Broker is baked right in, you can do all your messaging work and your data manipulation within the scope of a local transaction.

Service Broker has a few rough areas and it lacks a supported managed api (though there is a sample managed api available). Probably the biggest issue is that Service Broker has almost no interop story. If you need to interop with a Service Broker service, you can use SQL Server's native Web Service support. or the BizTalk adapter for Service Broker from AdapterWORX. However, I'm not sure how many of Service Broker's native capabilities are exposed if you use these interop mechanisms. You would probably have to write a bunch of application code to make these capabilities work in an interop scenario.

Still, I feel Service Broker's unique set of capabilities, its natural fit with WF and its high performance make it the best choice for building my project's long running services. Is it the best choice for your project? I have no idea. One of the benefits of working for MSIT is that I get to focus on solving a specific problem and not on solving general problems. I would say that if you're doing exclusively atomic or stateless services, Service Broker is probably overkill. If you're doing any long running services at all, I would at least give Service Broker a serious look.

Lip Service on Long Term Planning

Long term readers know my liberal political leanings. So it should come as no surprise to them that I read liberal blogs like Washington Monthly. But this isn't a post about politics, it's a post about planning:

This kind of long-term planning — in politics, in business, in nearly every walk of life — is something that nearly everyone says they support, but when push comes to shove very few people are willing to back it up. There's always something this week, or this month, or this year that seems uniquely crucial and demands our attention. Next year there will be something else, and the year after that something else again. The long-term stuff simply never gets done unless someone like Dean is willing to go to the mat for it.
[Building a Better Movement - Kevin Drum]

I don't have much to add to this, except that planning is a big part of architecture, especially architecture in the enterprise (which may or may not be "Enterprise Architecture"). Who "goes to the mat" for the long-term stuff at your company? Or does the long-term stuff simply never get done?

Thoughts on the SOA Workshop

Last week, I attended an SOA workshop presented by SOA Systems and delivered by "top-selling SOA author" Thomas Erl. It was two SOA-jammed days + the drive to Vancouver and back primarily discussing SOA with Dale. In other words, it was a lot of SOA. I went up expecting to take Erl to task for his "Services are Stateless" principle. However, that turned out to be a misunderstanding on my part about how Erl uses the term stateless. However, while Erl and I agreed on optimizing memory utilization (which is what he means by stateless), that wasn't much else when it came to common ground. As I wrote last week, Erl's vision of service-orientation is predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse.

Erl spends a lot of time talking about service reuse. I think it's safe to say, in Erl's mind, reuse is the primary value of service orientation. However, he didn't offer any reason to believe we can reuse services any more successfully than we were able to reuse objects. Furthermore, his predictions about the amount of reuse you can achieve are completely made up. At one point, he was giving actual reuse numbers (i.e. 35% new code, 65% existing code). When I asked him where those numbers came from, Erl admitted that they were "estimates" because "there hasn't been enough activity in serious SOA projects to provide accurate metrics" and that there is "no short term way of proving" the amount of service reuse. In other words, Erl made those numbers up out of thin air.

This whole "serious" or "real" SOA is a major theme with Erl. One the one hand, I agree that SOA is a horribly overused term. Many projects labeled SOA have little or nothing to do with SO. On the other hand, it seems pretty convenient to chalk up failed projects as not being "real" SOA so you can continue to spout attractive yet completely fictional reuse numbers. I asked about the Gartner's 20% service reuse prediction and Erl responded that low reuse number was because the WS-* specs are still in process. While I agree that the WS-* specs are critical to the advancement of SO, I fail to see how lack of security, reliable messaging and transactions are holding back reuse. If anything, I would expect those specs to impede reuse, as it adds further contextual requirements to the service.

While I think Erl is mistaken when it comes to the potential for service reuse, he's absolutely dreaming when it comes to the organizational structure and behavior that has to be in place for this potential service reuse to happen in the first place. I'm not sure what Erl was doing before he became a "top-selling SOA author," but I find it hard to believe it included any time in any significantly sized IT shop.

Erl expects services - "real" services, anyway - to take around 30% more time and money than he traditional siloed approach. The upside for spending this extra time and money is the potential service reuse. The obvious problem with this is that we don't know how much reuse we're going to see for this extra time and money. If you spend 30% more but can only reuse 20% of your services (as Gartner predicts), is it worth it? If you end up spending 50% more but are only able to reuse 10% of your services, is it worth it? Where's the line where it's no longer worth it to do SOA? Given that there's no real way to know how much reuse you're going to see, Erl's vision of SOA requires a huge leap of faith on the part of the implementer. "Huge leap of faith" doesn't go so well with "corporate IT department".

Furthermore, the next IT project I encounter that is willing to invest any additional time and money - much less 30% - in order to achieve some theoretical organizational benefit down the road will be the first. Most projects I've encountered (including inside MSIT) sacrifice long term time and money in return for short term gain. When asked how to make this 30% investment happen, Erl suggested that the CIO has to have a "dictatorial" relationship with the projects in the IT shop. I'm thinking that CIO's that adopt a dictatorial stance won't get much cooperation from the IT department and will soon be ex-CIO's.

In the end, I got a lot less out of this workshop than I was hoping to. As long as SO takes 30% more time and money and the primary benefit is the same retread promises of reuse that OO failed to deliver on, I have a hard time imagining SO making much headway.

PS - I have a barely used copy of "Service-Oriented Architecture: Concepts, Technology, and Design" if anyone wants to trade for it. It's not a red paperclip, but it's like new - only flipped through once. :)

Stateless != Stateless

A while back, I blogged that Services Aren't Stateless, in response to some stuff in Thomas Erl's latest book. At the time, I mentioned that I was looking forward to discussing my opinions with Erl when I attended his workshop. I've spent the last two days at said workshop. I'll have a full write-up on the workshop later this week, but I wanted to blog the resolution to this stateless issue right away.

At the time, I wrote "I assume Erl means that service should be stateless in the same way HTTP is stateless." Turns out, my assumption was way way wrong. When he addressed this topic in his workshop, he started by talking about dealing with concurrency and scalability, which got me confused at first. Apparently, when Erl says stateless, he's referring to minimizing memory usage. That is, don't keep service state in memory longer than you need to. So all the stuff about activity data, that's all fine as per Erl's principles, as long as you write it out to database instead of keeping it in memory. In his book, he talks about the service being "temporarily stateful" while processing a message. When I read that, I didn't get it - because I was thinking of the HTTP definition of stateless & stateful. But if we're just talking about raw memory usage, it suddenly makes sense.

On the one hand, I'm happy to agree 100% with Erl on another of his principles. Frankly, much of what he talked about in his workshop seems predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse. So to be in complete agreement with him on something was a nice change of pace. Thomas is much more interested in long-running and async services than I originally expected when I first flipped thru his book.

On the other hand, calling this out as a "principle of service orientation" hardly seems warranted. I mean, large scale web sites have been doing this for a long time and SQL Session State support has been a part of ASP.NET since v1. Furthermore, using the term "stateless" in this way is fundamentally different from the way HTTP and the industry at large uses it, which was the source of my confusion. So while I agree with the concept, I really wish Erl hadn't chosen an overloaded term to refer to it.

Feasible Service Reuse

Yesterday, I posted about services and reuse. More to the point, I posted why I don't believe that business services will be reusable, any more than business objects were reusable. However, "can't reuse business services" isn't the whole story, because I believe in different kinds of reuse.

The kind of reuse I was writing about yesterday is typically referred to as "black box reuse". The idea being that I can reuse the item (object or service) with little or no understanding of how it works. Thomas Beck wrote about colored boxes on his blog yesterday. Context impacts reuse - the environments in which you plan to reuse an item must be compatible with what the item expects. However, those contextual requirements aren't written down anywhere - at least, they're not encoded anywhere in the item's interface. Those contextual requirements are buried in the code - the code you're not supposed to look at because we're striving for black box reuse. Opaque Requirements == No Possibility of Reuse.

As I wrote yesterday, David Chappell tears this type of reuse apart fairly adeptly. Money quote: "Creating services that can be reused requires predicting the future". But black box reuse this isn't the only kind of reuse. It's attractive, since it's simple. At least it would be, if it actually worked. So what kind of reuse doesn't require predicting the future?

Refactoring.

I realize most people probably don't consider refactoring to be reuse. But let's take a look at the official definition from refactoring.com:

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring

Two things about this definition imply reuse. First, refactoring is "restructuring an existing body of code". It's not rewriting that existing body of code. You may be making changes to the code - this certainly isn't black box reuse - but you're not scrapping the code completely and starting over. Second, refactoring is making changes to the code "without changing its external behavior". You care about the code's external behavior because somewhere, some other code is calling the code you're refactoring. Some other existing piece of code that you don't want to change - i.e. that you want to reuse.

When you refactor, you still reuse a significant amount of the code, but you’re not having to predict the future to do it. Refactoring is the kind of reuse I believe in.

In his article, David talks about types of reuse such as business agility, adaptability and easily changeable orchestration. These look a lot more like refactoring than black box reuse to me. Unfortunately, David waves these away, saying  "Still, isn’t this just another form of reuse?". Reconfiguration hardly qualifies as "predict the future" style reuse that he spends the rest of the article arguing against. It's just one paragraph in an otherwise splendid article, so I'll give him a pass this time. (I'm sure he's relieved.)

A Question of Context

A couple of weeks ago, David Chappell posted a great article on SOA and the Reality of Reuse. When someone mentions the idea of using SOA for reuse, I cringe. David does a great job blowing the "SOA for Reuse" argument out of the water. In the future, I will just send that link rather than spending the time arguing out the point.

But something nagged at the back of my brain about that post. David starts by talking about object reuse before making the parallel to services. The problem with that comparison is that object reuse hasn't been a failure. When was the last time you wrote a String class? A Linked List? A Button? There's been support for Buttons in Windows since at least the 3.x days (probably longer, but that's before my time). Whatever your OO language of choice, there's a big collection of reusable objects to go with it.

Given his position as "famous technology author", I'm assuming David is well aware of successes of object reuse. Furthermore, I doubt it was an accident that in his article he writes that "reuse of business objects failed" (emphasis added). While there has been success around object reuse, essentially none of those successes have been in a business scenario. In fact, there have been some high profile projects such as Microsoft's Business Framework and IBM's San Francisco Project that have crashed and burned been significantly less than successful.

So here's the question: given that general object reuse has seen some success, what's so different about business objects that causes reuse to fail utterly? Since we're really interested in service reuse, knowing why some object reuse succeeds and other reuse fails will help us understand which services are likely to be reusable and which wont. I would say that success of object reuse hinges on context.

Wikipedia gives this definition of context: "The context of an event, word, paradigm, change or other reality includes the circumstances and conditions which surround it." (emphasis in original) For example, the word "order" is ambiguous. If you're using a procurement system for the military, you could conceivably be given an order to place an order. (OK, that's silly. But you get the idea.) The word "order" has two different meanings. However, the words that surround the ambiguous term make the meaning clear. An order that you place is different that an order that you give. That's context.

A string or a linked list or a button has very little in the way of contextual needs. That is to say you can use it the exact same way in a wide variety of environments. A business object on the other hand has significant contextual requirements, which makes reuse difficult or impossible. For example, a Purchase Order object from the above-mentioned military procurement system sounds like it might be reusable. At least until you take into account the differences between branches of the military, between ordering tanks and ordering uniforms, between active units and reserve units, etc. Once the generic Purchase Order has been specialized to the point of practical usability for a given scenario, it's no longer reusable in the general case.

Taking this back to the service realm, likewise I figure the reusable services will be the ones with little or no contextual needs. A good example is the identity and directory services such as Active Directory and its competitors. Sure, you use LDAP not SOAP to access it, but AD is certainly both reusable and a service plus it's in wide usage. Other candidates for reusable services my team is looking at are service directory, management and operations, business activity monitoring and provisioning.

I actually think there will be less reuse in services than there was with objects. The value of reuse of services has to exceed not only the contextual issues but also the overhead of distributed access. Calling across the network is an expensive operation - whatever's on the other side better be worth the drive. I'm guessing for services, more often than not, reuse won't be worth the trip (if it's possible at all).

Update - David pointed out to me that the last paragraph of his article begins:

Object reuse has been successful in some areas. The large class libraries in J2EE and the .NET Framework are perhaps the most important examples of this.

Doh! I guess my "assumption" that David is aware of successful object reuse was correct.

SOA Sample Scenario

So now that I'm back, we're beginning work in earnest on my project. For those not following along at home, my project is to deliver shared service oriented infrastructure for Microsoft's internal IT department. We've spent the time since I moved over working on our business justification, and now we're moving into specifications and prototyping. That is, I get to starting getting my hands dirty.

As part of the prototyping efforts, we're looking to build some sample business services on top of our prototype infrastructure. The idea being to both illustrate what we're building as well as have a playground where we can experiment with new ideas. During the prototyping, I'll be pretty involved with the development. But once we start writing production code, the dev team will take that over but I will continue to own the service playground. I've been kicking a playground idea like this around for several years, so I'm pretty excited about it.

The question is, what kind of business scenario should we build? I want the sample business services to be something interesting and real-world-esque. But of course, it can't be too complex since I wasn't hired to build a playground as my primary job function.

So far, we have two primary ideas:

• Enterprise Management System: AdventureWorks is the primary sample database that ships with SQL Server. They have business scenarios around Sales & Marketing, Product Management, Purchasing and Manufacturing. This sounds suspiciously like an ERP/CRM/SFA type enterprise system. On the plus side, MS is an enterprise so things like ERP/CRM/SFA are the types of solutions we need/use/buy/build internally. On the negative side, it's complex to do real-world and teams that actually do ERP/CRM/SFA inside Microsoft might dismiss the infrastructure if the playground isn't real-world enough.
• Prediction Market: If you've ever seen Hollywood Stock Exchange (HSX) or Tradesports, those are prediction markets. The basic idea is that you trade on predictions, rather than companies like you would in a stock market. Using HSX as an example, you get 2 million "Hollywood Dollars" (i.e. play money) to invest in upcoming movies and / or movie stars. Those movies and stars pay out based on the money they make at the box office. They also have derivatives for opening weekend, blockbusters and the Oscars. HSX even sells forecasting and prediction services based on the HSX market. Of course, I can't build something quite so extensive, but we could get pretty far with this idea. The upside is that it's relatively simple (compared to enterprise management systems) and there's little conflict with existing systems inside Microsoft. The downside also is that it's relatively simple and not like anything we're building inside Microsoft.

I'm leaning towards the prediction market, as it sounds more fun to build and experiment with. What do you think?

Cut Out The Middle Man

Nick Malik is an architect in MSIT's Enterprise Architecture group. He's been blogging a while, though I only discovered his blog a couple of weeks ago. Yesterday he posted about OMG's SOA SIG's Draft RFI on EDA and it's relationship to SOA and BPM. That's a veritable alphabet soup of acronyms! To translate, the Object Management Group's Special Interest Group on Service Oriented Architecture has posted a draft Request for Information on Event Driven Architecture and it's relationship to Service Oriented Architecture and Business Process Management. Here's the summary from the actual document:

The EDA Sub-group of the OMG SOA SIG seeks information from members of the EDA, BPM and SOA community as well as anyone interested in promoting standards in this area. Requested information will be evaluated by the EDA Sub-group, resulting in the development of Requests for Proposal(s) (RFP) for standardization of Event definition, relationship between EDA, BPM and SOA that will ultimately allow development of standards for Complete Life Cycle of Events -Ontology of Events, Sense and Respond Services, Events Metrics and processing of complex events. Please note that it is our intent to develop modeling standards for the EDA/SOA and EDA-Business Process interaction and provide standards for the implementation of that interaction as well.

First off, I'm a bit wary about this part: "it is our intent to develop modeling standards". Of course, OMG is responsible for UML and long time readers should be well aware of my opinion of UML. I don't want to set the bozo bit on an entire organization, but I am skeptical that any new modeling "standards" from OMG will be any more effective than the Unwanted Modeling Language.

Secondly, EDA seems to be vaguely defined, if at all. Wikipedia has this to say about EDA:

An event-driven architecture (EDA) defines a methodology for designing and implementing applications and systems in which events transmit between loosely coupled software components and services. An event-driven system is typically comprised of event consumers and event producers. Event consumers subscribe to an intermediary event manager, and event producers publish to this manager. When the event manager receives an event from a producer, the manager forwards the event to the consumer. If the consumer is unavailable, the manager can store the event and try to forward it later. This method of event transmission is referred to in message-based systems as store and forward.
[emphasis in original]

Assuming events are encoded as messages, then you can rewrite "event consumers / event producers" as "message receivers / message senders" and you really blur the line with SOA? 

The big difference in EDA seems to be the use of an "intermediary event manager". The problem I have with this approach is that the "intermediary event manager" works fine if you have a small number of endpoints, but how will it scale to handle hundreds of systems? Thousands? Tens of thousands? I don't see how the centralized event manager approach can possibly scale to handle tens of millions of events delivered between tens of thousands of systems. The management of such a system would be a nightmare? If a business process went south, you would obviously look in the central event manager as the source of the problem, but I would think that would be like finding a needle in a haystack. You could federate the event managers, instead of attempting to scale out the center. But a federated event manager approach would seem to defeat much of the purpose of an EDA in the first place. If you're going to federate your event managers, why not cut out the middle man and make each event producer it's own event manager as well? What is the benefit of separating these capabilities?

I guess fleshing out EDA isn't a bad idea, but it seems more like a solution looking for a problem to me.

Business Processes Are Services Too

I've been having a conversation with Piyush Pant over on his blog that started as a comment he left on my Services Aren't Stateless post. He thinks that I'm "missing the crucial point here by implicitly conflating business process and service state". While Piyush hasn't really defined what he means by these terms, I think I understand what he's getting at. Yes, process and service state are different in many ways, but they are also similar in that they are both service private data.

Pat Helland (side note - I wish Pat would start blogging again) wrote an article some time ago titled Data on the Outside vs. Data on the Inside where he talked about the differences between service private data and data in the space between the services. For example, data on the outside is immutable, requires an open schema for interop, doesn't need encapsulation and is representable in XML. Service private data is not immutable, doesn't need an open schema for interop, requires encapsulation and is typically stored in a SQL RDBMS. So on this front, process and service state are both service private data so conflating them makes some sense.

However, what's not in the article is the idea of Resource and Activity data. Not sure why Pat didn't include this in the article, but he was talking about it as far back as PDC 2003. Stu Charlton described the difference between resource and activity data in his Autonomous Services article:

Activity Data - This is "work in progress" data for any long-running business operation, and is usually encapsulated by business logic. A classic example is a shopping cart in any e-commerce system. This data is mutable, but typically has low concurrency conflicts, as it is not widely shared. Typically activity data retires after a long running operation completes, and may be archived in a decision support system for later analysis.

Resource Data - This is "state of the business" data, which represents the resources of an organization, and is usually encapsulated by business logic. Examples are: room availability in a hotel, inventory levels in a warehouse, account statuses, employee and customer information. Some resources have a small life span, others may last a very long time (years). Resource data is usually volatile with potential for high concurrency conflicts.

So I'm fairly sure that when Piyush says "process state" I should hear "activity data". Similarly "service state" is "resource data". The differences between activity and resource data lead to some interesting implementation artifacts, which I assume he getting at when he says I'm conflating the two. For example, since activity data like shopping cart has low or no concurrency issues, using an optimistic concurrency scheme is entirely appropriate, which you would never use for highly volatile resource data like warehouse inventory levels. In fact, since activity data doesn't have concurrency issues, you could even store it inside an instance of workflow or orchestration, which gets serialized to a persistent store when it's in an idle state.

However, the fact that activity and resource data is handled differently doesn't mean that most services won't have activity data. When Thomas Erl says that that stateless services is a "common principle of service orientation", essentially what I think he's saying that services should only have resource data. And as I said before, this seems wrong to me. Sure, some services will be stateless. But all services? Services implement business capabilities. Most business capabilities are long running processes. Doesn't that imply that most services in the enterprise will need to be long running workflows or orchestrations?

So for the most part, Piyush and I just seem to have different names for the same concepts. The one issue I have with Piyush's descricription of process and service state is that he seems to implicitly assume that processes aren't services. Why not? Again, not all services will be processes, but if you're not exposing processes as services, how exactly are you exposing them?

Is SML Another Unwanted Modeling Language?

Piyush Pant wonders if SML solves a problem that nobody has? He also points out SysML project, which recently got folded into the OMG. Well, that explains why we called it "Service" instead of "System" modeling language.

Now that I work in IT, I can definitely say that SML will eventually solve a problem that I have. Most people agree that operations today is way to dependent on manual processes to scale effectively. Now SML doesn't solve that issue directly - as Piyush pointed out SML is a meta-modeling specification. However, SML is the foundation for the next generation of operational modeling tools like what we see in Visual Studio Team System for Architects. As I wrote several years ago, VSTS:A solves a very common problem - developers lack of understanding about the deployment environment. Piyush, haven't you ever had a long weekend going back to the drawing board because the solution you had built was undeployable and you didn't discover that fact until the operations team attempted to deploy it? If you haven't, I envy you.

On the surface, I agree with Piyush when he says that "history of software is littered with unsuccessful attempts to impose monolithic modeling constructs". However, the fact that it keeps happening indicates the problem hasn't been solved. Wanting to solve a problem and being able to solve a problem are two different things. Furthermore, the history of software is also littered with very successful attempts to raise the level of abstraction by the introduction of new programming languages: C, C++, VB and Ruby are all examples of this. Given that Code is Model, what we have is a history of software littered with some successful and some unsuccessful modeling constructs. I would argue that the successful modeling constructs have taken a bottom up approach - build a language a small abstraction step above something that actually runs and compile down. These unsuccessful modeling constructs (*cough* UML *cough*) take a top down approach - build a language way above anything that actually runs and hope a miracle happens to keep it in sync with the stuff you actually build.

The question is whether SML will be top-down (i.e. a failure) or bottoms-up (i.e. a success). So far, it's to early to tell, but I have high hopes.

Service Modeling Language

On the one hand, this seems like a somewhat arbitrary name change - from System Defintion Model to Service Modeling Language. And the use of the term "Service" instead of "System" seems sketchy to me.

On the other hand, you gotta love this line from the press release: "The group plans to submit the draft specification to an industry standards organization later this year." Given the companies involved in "the group" - MSFT, IBM, BEA, Sun, Dell, BMC, Cicso, Intel, HP and EMC - you gotta think SML has a bright future.

Here's hoping that publishing the SML spec is the first step in a public-workshop-based revision process, a la the Web Services Protocol Workshops.

deVadoss Down on SOA

My old boss's boss seems like he was in a downer mood yesterday. First, he blogged about the "Myth of Reuse in SOA", then the "Achilles Heel of SOA". Actually, truth be told, I agree with him on both counts.

I slam the door on the reuse argument every time it comes up in my new job. Actually, I slam the door on what I call "Naive Reuse". When John talks factoring for agility, he's talking about a form of reuse - similar to how use "reuse" code when you refactor. What does it mean to refactor service? How about refactoring your enterprise?

As for the Achilles Heel "data problem", I think that's an artifact of the prevailing stateless request/response mindset most people have about services that I touched on yesterday. I think Pat Helland described a very good approach for dealing with data in an SOA, but I haven't seen it implemented broadly. Rest assured, many of the concepts Pat described are at the forefront of my thinking as my new project takes shape.

Services Aren't Stateless

My teammate Dale and I are going to an SOA Workshop in Vancouver in mid September. The workshop is put on by SOA Systems, which was founded by "top-selling SOA author" Thomas Erl. I have a copy of his first book, but I've never really opened it. Dale let me borrow Erl's second book. I figured since I was going to see him speak, I should at least flip through his books.

I was looking thru the chapter 9 "Principles of Service-Orientation". Most of them are spot on, if not exactly news. Services are loosely coupled, autonomous and share a formal contract. Yep, with you so far. But then I got to this one:

Services are Stateless
Services should not be required to manage state information, as that can impede their ability to remain loosely coupled. Services should be designed to maximize statelessness even if that means deferring state management elsewhere.

This seems way wrong to me on several levels. Now I'm really looking forward to going to Erl's workshop, so I can discuss this with him face-to-face.

First off, his terminology is confusing. I have a hard time believing he really think services in general should have no state at all. I'm sure there are some examples of completely state-free services, but I believe they are both very rare and fundamentally uninteresting. A simple calculator service has no state, but why would you actually build or use one (except as a demo)? I assume Erl means that service should be stateless in the same way HTTP is stateless. IMO, stateless is poor description of HTTP. Connectionless or sessionless would be more accurate.

Regardless of my opinions on poor terminology, the problem with stateless services is that many - perhaps most - business operations aren't stateless. And while HTTP is stateless, as soon as you use cookies, it becomes a stateful protocol. If you don't believe business operations are stateful, try buying something on your favorite ecommerce site with your cookies disabled. Most sites will give you a "Your computer must have cookies enabled" error message. Sites that still work are embedding a session ID in the URL instead of in a cookie (ASP.NET has built in support for this type of Cookieless Session State). Either way, state is required for even the simple task of ordering something from a web site.

If most business operations aren't stateless, why should services that implement business operations be stateless? This seems like a violation of the "but no simpler" part of Einstein's famous paraphrased quote.

Slides from Gartner EA Conference

My old team decided to post the slides from my sessions at the Gartner EA Summit last month. Well, they've posted PDF versions of said slides and the fonts are somewhat off, but you'll get the idea.

• Beyond SOA: Understanding the Role of Users in Architecture
• The Dell Integrated Desktop: A Case Study of the Role of Users in Architecture

If you're interested in more info on the Dell Integrated Desktop, you can check out the full case study.

FYI, Gartner recorded audio for these sessions and supposedly it's on its way to me. I'll try and post the audio with synced slides as soon as I can.

New Teammates Blogging

I'm settling in to my new job. One way to tell, read Dale Churchward's blog. Dale's a teammate of mine. He only joined Microsoft a few months ago. Apparently, he used to blog at his old job, but either way we've now doubled the number of bloggers on my new team, with hopefully more to follow.

In addition to his opinions of political discource and the Seahawks chances next season, Dale's got some interesting posts on data integration and system diagrams. Check it out.

Gartner EA Summit Day Two

I didn't post a day two wrap up of the Gartner EA Summit because I only made it to my session and booth duty right afterwards. That wasn't the original plan - the pipes in the bathroom of my hotel room kept banging so I didn't get much sleep.

My session went well. I heard from several people afterwards that it was their favorite session or that it was the highlight of the conference. Nice anecdotal evidence, but I still want to see the scores. They recorded the session, hopefully I can get it so I can publish it here. I had lots of great conversations afterwards (as expected). Maybe Gartner will have me back next year with twelve months of my new project under my belt.

One suggestion for the Gartner folks. Next year, don't pick a logo with an arrow in it. I got a little confused when I first showed up because I followed the arrow and ended up on the wrong side of the hotel from the event. My friend Scott snapped this picture of a sign with two arrows pointing in different directions.

Architecture by Powerpoint

Adam Stanley left the following comment about the MS booth @ the Gartner EA Summit and I just had to share.

One of the true revalations of being an enterprise architect is in the books that Microsoft was giving away. One on enterprise patterns, one of powerpoint. I do a lot of powerpoint! :)

The book in question is Beyond Bullet Points, which I highly recommend to architects and non-architects alike.

Gartner EA Summit Day One

I'm in sunny San Diego for the Gartner Enterprise Architecture Summit. I'm presenting a sponsor session and case study tomorrow (MSFT is a platinum sponsor for the event) but I came in yesterday so I could attend a few sessions, meet a few customers and work the MS booth in the Solution Showcase. It's my last event and deliverable for my old team before switching to the new role full time. Details of today's sessions below the fold.

Moving On...

It's not the biggest job change news this week (or the day), but after three years on Architecture Strategy and six years total as an evangelist, I'm moving on to a new role. After six years, I decided it was time for me to put my money where my mouth is as well as get my hands dirty building something more substantial than buzz.

I'll be moving over into Microsoft's IT division as a member of the Integration Center of Excellence Architecture Team. Integration, as you might guess, is a euphemism here for service-orientation. My team is tasked with architecting and delivering the shared service-oriented infrastructure for four of the biggest projects Microsoft IT will be delivering in the next year. Last time I changed jobs, I lamented that "With each job I take at MSFT, coding seems to become less a part of the job description." Happily, this is NOT the case this time.

About a year ago, Microsoft hired Stuart Scott to run the business apps side of IT as one of our two CIOs (our other CIO Ron Markezich oversees the IT infrastructure). Stuart was kind enough to spend about an hour with me last week explaining his vision for how he sees MSIT evolving under his leadership. Here's what he said in a recent interview:

PressPass: How do you see Microsoft IT evolving?

Scott: There is a broader role for IT to play at the front end of the development of products and services. Our IT organization knows a lot about the challenges that other IT organizations face because we build and maintain the IT backbone of a massive worldwide enterprise. IT must become future-thought leaders in the development of the product roadmap for our enterprise products.

By using our internal applications and experiences to build better products for our enterprise customers, we have the potential to solve the challenges that other IT organizations face. We’re heavily involved in dogfooding our products once they’ve been developed, but we also see a role closer to the front end of the product development cycle. Business Intelligence is one area where we will be partnering with the product groups and Finance as we build out our internal capability. I want to ensure that any product we develop to meet the needs of Microsoft, also meets the needs of the marketplace.

He was also very frank about the current state of affairs in MSIT relative to the vision. He was quoted in China Information World (no link, sorry) as saying that "The systems Microsoft now uses are already 14 years old and based on previous versions of windows, so from a systems capability perspective, they cannot support the needs of the growing business."

All in all, I was pretty impressed with what he's setting out to do and the opportunity not only from a business perspective from from an industry perspective as well. Hence the whole "going to work in his division" thing. Of course, "Thought Leadership" is one of the things Architecture Strategy works on very diligently, so in some ways this isn't as big a change as it might be. On the other hand, giving advice to people solving hard problems is a lot different than solving those hard problems yourself.

I'll be starting this new role pretty much immediately, so expect the less-than-usual blogging to continue for the time being. But my external visibility on my blog and presenting at conferences and executive briefings is one of the things they hired me for. So after I get my bearings things should be back to normal. At that point, I'll hopefully be able to talk in more specific terms about what we're tackling on my new team. I hope to shake things up quite a bit over there and deliver the play by play here on my blog.

Massive thanks to John deVadoss and the rest of the Architecture Strategy Team. Back when I started, I think Simon Guest was the only other blogger on the team. Now there are only three non-bloggers on the whole team. It's been a great three years and I've been a part of so many great accomplishments:

• Making Microsoft the first (I think!) global sponsor for the International Association of Software Architects
• Delivering the CRM Integration for Outlook Sample Application
• Launching the Solution Architect and Infrastructure Architect MVP programs
• Delivering the first-ever Architecture Track at Tech-Ed 2004
• Taking Arvindra's amazing JOURNAL The Architecture Journal worldwide
• Launching the Architecture Resource Center and the MSDN Solution Architecture Center websites
• Presenting at PDC, worldwide Tech-Eds, p&p Summits and a host of other events
• Meeting tons of customers in user group meetings, at executive briefings and in architecture design sessions

I always joke that if I ever left Microsoft, I wouldn't want to go work for another technical company. Now, I get the chance to go affect the business of a Fortune 50 business while not having to leave Microsoft. Pretty sweet.

See you on the other side.

June DSL CTP

Congrats to the team for their latest version of the DSL Toolkit, integrated into the June CTP of the VS SDK. According to the published product plans of the VS SDK, they're suppoesd to ship their next release - including the final DSL toolkit - next month. Looking forward to it.

Thinking About Object Models

I'm doing some experiments with Amazon's S3 Service. Very cool service, I might add. Anyway, the sample C# REST code basically wraps the network requests with a single connection class that has individual methods for each type of service interaction (list all my buckets, list all objects in a bucket, create a bucket, create an object, you get the idea).

However, S3's service is a natural hierarchy. The Service contains many Buckets, which in turn contain many Objects. So another way to wrap the service interaction is with a series of objects that are related to one another and only implement the service interactions relevant to that class. (Service would implement List My Buckets and perhaps Create Bucket. Bucket would implement List Objects and Delete Bucket. Again, you get the idea.)

For an interface as relatively simple as S3 (the SOAP interface has a grand total of 13 operations) it probably doesn't matter one way or the other. Furthermore, it's probably a question of personal preference. My question: What's your personal preference? A single object with many methods or a hierarchy of objects each with fewer methods?

Enterprise 2.0 ARCast

Ron just posted his latest ARCast featuring yours truly talking about Enterprise 2.0. Some of the same stuff I blogged about last month, but in a conversational style. Check it out.

Architect Connections Conference

For any readers in LA, I'm keynoting the MS Architect Connections conference in LA next week. They have a few spots still open, so if you're interested you can sign up here.

TechEd Iron Architect Contest

For a change, I'm not involved with TechEd at ALL this year. It's all Simon and Marty. Looks like they've got some cool stuff cooking, particularly the Iron Architect contest. Almost makes me wish I was going this year.

SPARK Weblog

In preperation for SPARK later this month, we (i.e. the Architecture Strategy Team) has set up a SPARK Blog. So far it's mostly links to a few people talking about SPARK, but it's also appears to be an opportunity to use the work "SPARK" whenever possible, such as "SPARKs Fly".

I'm just waiting for someone to blog about SPARKitecture. :)

Tim's Wannabe Five

I don't know how I missed this before, but Tim Bray blogged about not inventing XML languages over a month ago. This comment is right on the money: "The value of a markup language is proportional approximately to the square of the number of different software implementations that can process it." Conceptually, I agree with him - my primary argument against XSPF is that it has the same basic semantics as RSS, but RSS is much more widely used. But his list of the "big five" markup lanugages seems more like the "wannabe five". How many different siftware implementations process any of the things on his list?

I'm not arguing the technical quality of these lanugages - frankly I'm not that familiar with any of them but Atom. But if you're arguing the network effect, none of these formats Tim lists qualify. I'm sure he wants them to be popular, but wishing doesn't make it so.

Redundant Specifications

So after my two posts on XSPF and some public discussion in the comments, I took the conversation with Lucas offline in hopes of getting a better understanding about the thought process that went into the spec. Unfortunately, Lucas has reacted as if I called his baby ugly and we got nowhere. Needless to say, I still believe that XSPF is completely redundant because it is nearly semantically identical to RSS.

Yet Another AJAX Toolkit

Hot on the heels of my post about the AJAX toolkit spectrum comes news that Yahoo! has released their own AJAX toolkit as well as a very cool web page design pattern catalog.

Some non-surprises:

• The Yahoo! toolkit has it's own animation library, it's own drag and drop library and it's own XmlHttpRequest wrapper, just like other libraries like Dojo and prototype.
• The Yahoo! UI Controls depend on Yahoo's Core Utilities. The libraries are modular, but not polymorphic with other these libraries.
• The code was released under an open source license (BSD).

All these different implementations of the same core set of capabilities at best means learning lots of ways of doing the same thing and at worst means incompatibility between components from different libraries.

BTW, last week I wrote that using a text-based high-level scripting language like Javascript in the browser "encourages business models where the in-browser code has little if any value." This release from Yahoo! supports that point. While their mapping component is subject a strict Terms of Use, their UI components were released with a liberal open source licence. I doubt that's a coincidence.

Thoughts on the AJAX Toolkit Spectrum

Last week, Dion wrote about the spectrum of AJAX tookits. He ended with a question, wondering which end of the spectrum will dominate? Will it be lightweight composable toolkits like prototype, script.aculo.us or Dojo? Or a more comprehensive toolkit like Atlas?

This came up in a chat I had w/ Jimmy Nilsson today. Well, not specifically about AJAX toolkits. Rather, we were talking about what he called technicalities vs. semantics:

I have noticed that there seems to be a focus first on the technicalities and then on the semantics. Take Indigo (or WCF) for example. There has been sooo much talk about its technical aspects, but very little talk about how to design for it. I'm pretty sure that when the technicalities have been sorted, it's time for the semantic side. I'm not thinking about technical semantics, but rather business semantics.

On more than one occasion, I've had a head-beating-wall conversations with WCF folks who are completely obsessed with the secure, reliable and transactional delivery of messages, but have given exactly zero thought to the actual contents of said message. So I know where Jimmy is coming from.

With respect to AJAX toolkits, the question becomes just how easy will these lightweight toolkits compose? Because while Dion describes Google Maps as "a simple JavaScript include", that's just the technicalities, it doesn't begin to deal with the semantics. For example, Dojo has Dictionary object, prototype has a Hash object. Dojo extends the Javascript Array, so does prototype. Both libraries wrap the XmlHttpRequest object. In each of these cases, it appears to me that the library authors have focused on the technicalities, but not thought about the semantics. These implementations are all semantically similar, but incompatible. So I don't buy that these lightweight toolkits will compose well. What do I do if I'm using prototype but want the rich text editor in Dojo?

The network effect that Dion doesn't consider is the component ecosystem phenomenon that Microsoft has a ton of experience with. Old school VB, COM/ActiveX and .NET have all had large ecosystems of components and controls evolve that extend the functionality of the baseline development platform. There's no reason to believe that won't happen with Atlas. I think it's wrong to describe Atlas as a monolith or self-contained or enclosing. It's an extensible baseline platform - i.e. the baseline functionality is set down once at the development platform and the ecosystem can extend it from there. Sure, overlapping extensions happen (how many rich text editor components are there for ASP.NET?) but at least they all have basic compatibility.

Update - Fixed link to Dojo Toolkit in the first paragraph.

Web 2.0 Evolution

In his now-famous talk, Dick Hardt describes Identity 2.0 as inevitable. As in "coming for sure, but not here yet". I wonder how much of Web 2.0 is here now, and how much is inevitable? And furthermore, how much can we generalize about the future of Web 2.0 from what is happening now? As in many things, I think the answer isn't black and white.

For example, I think we can generalize about the bright future of peer-to-peer based technologies from looking at systems like Skype and FolderShare. Naturally, with the power shifting to the edge, I believe it's inevitable for more edge machines to communicate directly with each other rather than being mediated by a service in the center. In fact, in many cases I believe were going to want to shift a significant percentage of social computing to the peer-to-peer model. It scales better and doesn't have centralized privacy concerns. Furthermore, I think there may be be specific peer-to-peer capabilities that are difficult or impossible to replicate with a centralized model, though so far, I haven't them yet.

However, I'm not sure we can generalize about the future of mashups the same way. This isn't to say I think mashups are going away - far from it. I just think that mashups a year from now will look very different than they do today.

First off, I don't think we can  generalize the success of Google Maps. In the Programmable Web how to guide, they mention that "Plotting markers on maps is probably the easiest place to start". Apparently, many people are taking that advice because 297 of the 411 mashups listed use one of the three major (i.e. GYM) mapping services. However, maps are unique because of the massive amount of data, the extremely simple API and the ubiquity of location information. They are also one of the few mashup API's that runs in the browser - the vast majority of mashup API's are back end data type services like Amazon's E-Commerce Service. How many more in-browser mashup API's are out there waiting to be built? I'm not sure, but as I wrote in Browser as VM, the problem with these in-browser mashup API's is that you can't protect your IP.

As for back-end service mashup APIs, there needs to be a way for these service providers to make money. Even if the software they use to build the service is free, things like hardware and bandwidth are not. For an Amazon or Ebay, making money on thier services is relatively easy since they are facilitating sales transactions. In the end, they probably won't care much if a sales transaction originated on their site or on a site leveraging their APIs. However, if the service provider is ad-funded, the service API effectively routes around the site's revenue mechanism. Take, for example, a site for tracking events like Zvents, Eventful or Upcoming. They need to drive users to the actual site in order to drive revenue. So it remains to be seen exactly how the API based access is going to work out. Today, these API's are specifically provided for "non-commercial use only" so one way would be to charge for access via the API (either flat-rate subscription, a per-use charge or a combination of the two). Alternatively, they could be bought up by a larger company who could then afford to run the business at a loss. Yahoo already bought Upcoming and Google Base already has an event item type, but the other big companies in this space (I'd guess Microsoft, Amazon, Ebay and maybe Apple) might be interested. Again, I'm not sure how this evolves either, but it's got to evolve beyond "non-commercial access".

Latest Architecture Journal

It's been a while coming, but the print subscriptions of The Architecture Journal are underway. I got my copy of Journal 6 in the mail today. It's not online yet, but you can get the back issues and sign up for your own free print subscription on the website.

SPARK is Out of the Bag

As part of the new job, I'm involved in the planning a workshop called SPARK, which Dion Hinchcliffe blogged about this morning. (Dion also writes a blog here - bringing the total to three - so I created a combined feed just to keep track of all the places he writes). My new boss Mike also mentioned SPARK this morning. In the hopes of sparking futher interest (pun intended), here's the overview of SPARK:

SPARK is the first in a series of high-level forums hosted by Microsoft that use a workshop setting to examine “the issues that matter most” in the practice of strategic architecture and produce guidance for the industry as a whole.

Today, new social movements, advances in technology, and forces within business are overlapping to create a landscape glutted with challenges and opportunities. In many cases, these forces have driven the deployment of new technologies and the adoption of new behaviors, adding multiple layers to an already complex set of issues that must be navigated. Architects are searching for a solution that helps manage this complexity.

SOA, Software as a Service, Web 2.0, and Edge are all elements of the solution, but are they the complete picture? Are they a sufficient answer to the issues?  Can they be used together in a productive and efficient fashion? What matters most?

New DSL Toolkit Drop

I have been so focused on Web 2.0 stuff that I've been reading a bunch of new blogs and disregarding the old ones I used to read. So I didn't realize until today that the DSL Tools team released a new drop last week. According to Gareth, the highlights include:

• Integration into the Visual Studio SDK. According to the site, they are shooting for an April release for v2 of the VS2005 SDK, so does that mean the DSL Tools will be done in April?
• Single file format and complete visual designer for all aspects of a DSL. I'm guessing this mean we no longer have to edit the designer definition by hand. That's a good thing. But I liked the seperation of domain model and designer, so I'll be interested to see how I like what they've built.
• Domain-specifiic model serialization. This is huge - previously, the domain model dictated the XML serialization format. Now, if you can customize this, you can provide a clean model syntax and even possibly read in other syntaxes as well
• Port Shapes and a revised modeling API

Update: Apparently, I can't read. Only the VS SDK integration is done in this build. improvements to the file format and model serialization will be in the next drop.

Flash, the Other White Meat

When I wrote yesterday about the Browser as VM, I made the point that extensibility is difficult as we have four major browsers and multiple OSes to deal with. What Web 2.0 company is going to be willing to bet on a proprietary extension implemented in only one of those combinations? Not many if any I would guess. However, there is one option that works across all those browsers and OSes: Macromedia (now Adobe) Flash Player.

Unlike the browser, where AJAX is a relatively new idea, Flash has been positioning itself as a platform for nearly four years. Instead of AJAX, Macromedia coined the term Rich Internet Application or RIA. RIAs share a lot in common with AJAX in that they are downloaded on demand, execute arbitrary script code and can retrieve data across the network. But the most interesting commonality that Flash has with the browser is that runs across multiple browsers and OSes.

In platform portability, Flash has succeeded where Java failed. I haven't done enough research to know exactly why yet, but I suspect that it's because Java tried to be a complete portable environment on day one where Flash focused on specific functionality that weren't possible any other way - so called "skip intros" - and grew up from there. In other words, Java tried the top-down approach and Flash tried the bottom-up approach. I'm not surprised bottom-up worked and top-down crashed and burned.

While the modern browser has evolved to make it a capable platform, it still lacks some capabilities that Flash has. Most notably support for rich media. Thus, sites like Pandora, Google Video and MTV Overdrive need the capabilities provided by Flash.

While it's hard to imagine enhancements to the browser due to the difficulties across four browsers and multiple OSes, improvements to Flash are easy to imagine. According to Macromedia, Flash has 98% penetration. Even more impressive is that Flash reaches 80% penetration with new versions of the player within 12 months.

Check out this post from Kevin Lynch for more on Flash for Web 2.0 companies. So far, the only Web 2.0 company I know about (which is to say I'm sure there are more out there) is Goowy. Which ones am I missing?

Browser as Virtual Machine

Note: this is the first in a series of Web 2.0 entries. I know I’m on record as hating the term Web 2.0, but as I wrote in that post, I do belief there is a fundamental shift underway in computing. The industry is calling this Web 2.0, and I can either spit in the wind or go with the flow. Furthermore, for the more Web 2.0 savvy among my readership, much of what I write about in this series may be old news. But I want to blog what I learn as I learn it, so bear with me.

Just as the dumb terminal was eventually replaced with more sophisticated personal computers, the dumb browser has been replaced on the modern desktop by something significantly more versatile. When the ability to process arbitrary script code was added to the browser, it became a virtual machine in its own right. Perhaps unique and special-purpose when compared to environments such as the .NET CLR, but a VM all the same. And while its unique nature makes the browser unusable for entire genres of applications – you’d never use the browser to build a server application for example – it makes it well tailored for user-centric, software as a service style applications that have become commonplace. While the browser’s scripting capabilities have been around since the mid 90s, the industry has only recently started to leverage those capabilities to build applications that run on the client inside the browser. Jesse James Garrett coined the term “AJAX” – Asynchronous JavaScript and XML – to describe this style of application.

If the browser is a virtual machine, that makes JavaScript the “assembly language” of the browser. That is, JavaScript is the lowest level of abstraction you can program the browser with.  This has pretty dramatic implications on the applications you build for the browser VM. For one, JavaScript is at a sufficiently high level of abstraction that you can use it directly and be productive. Writing an entire application in IL or Java byte code is unthinkable, but isn’t really a big deal for JavaScript. Furthermore, Because JavaScript is a text-based scripting language, protecting your code as intellectual property is extremely difficult. While obfuscators exist, in the end they can only delay the reverse engineering of your code, not prevent it. This encourages business models where the in-browser code has little if any value.

For example, the big mashup functionality these days is mapping. There are three big mapping services out there: Google Maps, Microsoft Virtual Earth and Yahoo! Maps. 266 of the 368 mashups listed on ProgrammableWeb as I write this include mapping functionality from one of those services. That's nearly three out of four. Mapping is interesting because of the sheer amount of data involved. In fact, the code is pretty useless without the back-end data. So while I can get the code for Google Maps, it does me no good without access to the data for which I need the API key. Contrast this with the complete lack of market for browser-based rich text editors. Sure, there are various open-source script libraries like Dojo, Web Wiz RTE and Kevin Roth's RTE. But no companies offering a rich text editor service like they offer map services. Why is that? I would think the value of rich text editing would be even more widely applicable than mapping. The problem is that, unlike the map service, there's no back end associated with a rich text editor. There's no way to protect a client-side-only solution such as these rich text editors. The only people who do sell rich text editor components are ones who have integrated into some back-end programming environment such as Richer Components' RichTextBox for ASP.NET.

The browser as a VM also has broad implications with regard to extensibility. Similar capabilities are delivered by the four major browsers (IE, Firefox, Opera and Safari) across the major operating systems (Windows, MacOS, Linux, FreeBSD). So the question is, how will new capability evolve in the browser? Will the growing number of Web 2.0 companies looking to provide compelling features and differentiate themselves in the marketplace demand new functionality in the browser VM? Will one of the browser vendors be willing to take the heat of building proprietary extensions to their browser? I realize that many people have a dim view of proprietary extensions, but many features we take for granted today are de facto standards that arose from Microsoft's proprietary extensions to IE. Most notable of these of course is XMLHttpRequest, without which "AJAX" would just be "J". And JavaScript itself started life as a proprietary extension to Netscape before eventually being turned over to ECMA for standardization.

Developer 2.0 ARCast

FYI, I sat down for an ARCast with Ron Jacobs last month to talk about Developer 2.0. It's not he full presentation I did at the p&p summit, but it covers much of the same ground. Catch the show over on Channel 9.

Developer 2.0 at VSLive!

FYI, for those going to VSLive! in San Francisco, I'm a last minute addition to the schedule. I'm presenting a talk titled Developer 2.0: Finding Your Way in the Future of Software Development. I wrote and delivered this presentation originally for the patterns & practices Summit back in December. It was the second-highest rated talk of the summit (after Anders' LINQ keynote) so I'm excited to be delivering it again. I'm hoping to get a high-quality recording so I can publish it (I recorded the p&p summit version with my laptop. You can hear it but I wouldn't say it's "high quality"). The session is at 5:45pm on Tuesday. I've been told it's in room 2016/2018, but you should double check when you get there if you're interested in going.

There's a solid showing from the Architecture Strategy Team at the VSLive! Software Architecture Summit this year. In addition to my last minute addition:

• John deVadoss's day two keynote on Software Architecture on the Edge: Moving Beyond SOA.
• Ron Jacobs on Integration, Service Orientation and Pattern Driven Design .
• John Evdemon on Workflow Enabled Services.
• Mauro Regio on Collaboration Software Factory.

See you in San Fran!

Hating the Term Web 2.0

Now that I’m an Architect on the Edge (I’m thinking of putting that on my business card. Good idea or bad idea?) of course the first order of business is taking a closer look at “Web 2.0”. One thing leaps out at me right away – I hate the name “Web 2.0”.

First off, it’s a pure marketing buzzword. It was originally coined as a conference name. In a way, the fact that is has no underlying meaning is a good thing, because it gives people argue whether it really exists or not. In a way, it’s like the word “multimedia” back when we were first putting CD-ROMs into computers. There used to be lots of discussion if one thing or another truly was “multimedia”. Now, we don’t really worry about categorizing it as the marketing buzz around the term is long gone.

Secondly, I think it’s wildly arrogant to claim we’re only on version 2.0. The Internet has been around for 36 years. So everything before mid 2004 was Web 1.0 or earlier? And people are already talking about Web 3.0. Come on, let’s get real. The technologies that are driving the current revolution have been percolating for more than one major version of the underlying technology.

Finally, what’s with the version number anyway? One of the core principles Tim O’Reilly outlined was the “End of the Software Release Cycle”. Why are we using a holdover from the software release cycle days to indicate the end of the software release cycle?

Don’t get me wrong, I strongly believe that there is dramatic change happening in this industry. The way I explain my new job is to consider that one of the most basic axioms of distributed computing has been overturned.

From day one, all the computing power has been focused in the center. At first, the machines on the edge had no power at all – they were just dumb terminals. Slowly but surely, those machines at the edge started to become powerful in their own right. However, it’s only in the last seven to ten years that commodity hardware that was pervasive on the edge grew powerful enough to power the center. And it’s only in the last two or three years that the connection between the center and the edge grew fast and pervasive enough to make that edge power relevant.

The rules have changed. The power has shifted from the center to the edge. And we’re only just beginning to see the effects.

Maybe we should call it WebNT? :)

Architect on the Edge

So for the fourth time in seven months, I have a new manager. Way way way back and the end of June, I left marketing to be a solution architect. In doing so, I traded in Norman for John as my manager. Things were looking shiny but then Microsoft had a major reorg back in September. These big reorgs often cause small ripples, like the director of Architecture Strategy deciding to move back to a product group. John was promoted to head the team and Gianpaolo was hired to take over John's previous role as head of solution architecture. Finally, it seemed like things were calming down on the management front, so I was caught off-guard by what happened next.

Last Monday, John offered me a chance to switch roles. We have spun up an Edge Architecture team to focus on the architectural impact of the next generation of computing - what some call "Web 2.0" - and he thought I would be a perfect fit for the team. I agreed and jumped at the chance. So now I work for Michael Platt. Don't let the infrastructure focus of Michael's blog fool you - he incredibly deep on Edge Architecture and I am currently playing catch-up. The strangest thing so far is having to re-orient my thinking for consumer focused systems and away from the enterprise world where I have I've gotten the vast majority of my experience and where I have spent my entire Microsoft career to date.

In the immediate future, I'm been dumped into the MIX06 planning process. We have other stuff going on in the MIX timeframe that I'll get to soon enough. I'm also re-thinking my personal presence. As my career progresses, the moniker "DevHawk" seems more and more outdated. Is it?

CNET Picks Up Outlook Integration Sample News

I couldn't blog about the Outlook Integration Sample last week because we were waiting to see if the mainstream media would pick up the news. CNET ran an article about the sample today. There's also a link to the Customer Explorer case study that was the original inspiration for the Outlook Integration Sample and VSTO Outlook in the first place.

Evolving Language Fidelity

I haven't seen much in the way of response to my Hi-Fi Models post, but I did come across this great article by Ted Neward on the history of the tumultuous marriage of objects and relational databases, primarily in the context of LINQ. In the context of Code is Model, the following passage from the summary was the most interesting:

While Project LINQ doesn't purport to be the "final answer" to all of the world's object-relational mismatch problems, it does represent a significant shift in direction to solving the problem; instead of taking an approach that centers around code generation, or automated mapping based around metadata and type inference, both of which are exercises in slaving the relational model to an object-oriented one, Project LINQ instead chooses to elevate relations and queries as a first-class concept within language semantics and library-based extensions.
[Comparing LINQ and Its Contemporaries - Ted Neward]

When Ted says relations and queries are elevated to "first class concepts" within the language, it makes me think of Stuart's comment about language fidelity. I'm not sure I would say C# 3.0 is at a higher level of abstraction than 2.0, but I would say that the inclusion of these new abstractions does improve the language's fidelity. This fidelity improvement does come at the cost of complexity (TANSTAAFL) but compared to the current alternatives, I'm willing to pay that price.

The problem with increasing the language fidelity like this is dealing with the outdated code it leaves behind. You see this today with the addition of generics in the 2.0 CLR. How many hand-coded or generated strongly typed collections are floating around out there from the 1.x days? Lots. (As if 1.x was so long ago!) How much database access code is floating around out there today? An astronomical amount. Every app that touches a database or processes XML will be outdated with the arrival of C# 3.0 and VB 9.0. But the price of converting this outdated code to use the new abstractions probably won't be worth the time or risk. That means you're left with maintaining the outdated code while also writing any new functionality with the new language features.

I wonder how DSLs will be impacted by this evolving language fidelity issue? On the one hand, the nature of DSLs is that they have much narrower usage (i.e. one domain) than something like generics or LINQ. On the other hand, I expect DSLs to evolve faster than general mainstream languages like C# can. So I'm thinking the impact will be about the same.

Outlook Integration Sample

For the past few months, I've been heavily involved in a project but I wasn't allowed to blog about it. Last week, it went live on MSDN so finally the gag is off.

About a year ago, word started to surface about something called Project Elixir which aimed to integrate back end CRM systems with Microsoft Outlook. Part of that effort resulted in the addition of Outlook Managed Add-ins to Visual Studio 2005 Tools for Office. However, the VSTO team's primary deliverable was an add-in loader that enforced security, enabled shutdown unloading and provided a better startup/shutdown developer experience that IDTExtensibility2. (Check out the VSTO Outlook Architecture document for more details.) While those are important fundamentals that needed to be gotten right, VSTO Outlook doesn't provide much in the way of tools or guidance for building Outlook add-ins that leverage managed forms and controls or integrate with your back end systems. That's where the CRM Integration for Outlook sample comes in.

What we've built is a sample application that surfaces CRM style data inside of Outlook. Outlook is the natural home for your calendar and your personal contacts. Why not make it the natural home for your customer contacts, activities and opportunities as well? As part of the demo project we've implemented:

• Using Windows Forms for editing custom items. Check out this screenshot. The Activity form is a standard managed Windows Forms form, not an Outlook custom form.
• Using a Windows Forms user control as a folder home page. Here's a screenshot of the "CRM Today" page. Again, that's a standard managed Windows Forms user control.
• A framework for adding menu items and toolbars. In Outlook, the developer has to manage adding the custom toolbars and menu to each explorer and inspector window themselves. With our sample, we built a framework to handle that for you.
• Using SQL Express as a local cache of CRM data. It turns out that for many scenarios, storing a copy of all the back-end data directly in Outlook is a bad idea. First, it increases the size of the users mailbox, requiring more storage on the Exchange server. Furthermore, any custom data in Outlook has to be synced twice - once from the back end system to Outlook on the desktop, then from Outlook back to Exchange. By minimizing the amount of back-end data stored in Outlook proper, we reduce the mailbox size and sync bandwidth needs. In both the above screenshots, the displayed data is coming out of the local SQL Express instance, not Outlook.
• Having two separate storage locations (Outlook & SQL Express) means having to sync between them. We've built a local sync engine that can sync both individual items between Outlook and SQL Express as well as a collection of items between SQL Express and a given Outlook folder.
• Finally, there are some utility classes to make it easier to deal with Outlook folders and items. Of primary note is the ItemAdapter class which provides a pseudo base class for Outlook items (appointments, emails, tasks, etc). Those items all have a set of similar properties and methods, but don't have a common base class so they can't be treated polymorphicaly. ItemAdapter uses runtime reflection to implement those common operations without needing to cast to the concrete Outlook item type.

Check out the Architecture Design Guide, as well as the Outlook Customization Guide and the Local Sync Engine Guide up on the Solution Architecture Center. You can also pick up the source code. Also, I spun up a GDN Workspace so we can have a discussion forum and to track bugs and requests.

Going forward, I'm going to be focusing on the remote data sync story for this scenario. Among other responsibilities, I "own" the Data pillar of our Connected Systems model so this dovetails nicely. You'll note above that while we have a local sync engine in the sample, we don't have any way to move the data back and forth between the local copy in SQL Express and the remote copy in the CRM back-end. We are working on some guidance around this right now, but we didn't want to hold up publishing the rest of the sample.

Frankly, it's been nice to be involved with something so technical after spending time on the marketing team. I'm pretty proud of the project and I look forward to your feedback.

Update: Removed the link to the running demo as it's been taken off the download site for reasons I am not aware of. If you want the binary and you don't know how to compile it, drop me a mail.

My New Boss is Blogging

Gianpaolo took over the Solution Architecture team a few months ago and he rebooted his blog a couple of weeks ago. And he's active on the afore mentioned Architecture Forums. Subscribed.

Architecture Forums

Simon just emailed a bunch of internal architects about the new Architecture Forums on Microsoft.com. So far there's a general architecture forum as well as one for modeling and tools, with more to come I would guess.

What other forums do you think we need?

Hi-Fi Models

I'm slowly but surely working through my holiday backlog of email and weblogs. Slowly being the operative word here.

Anyway, Stuart has a great post on the process by which we build models. And he's not talking theoretically here, he's working on a model for the designer definition file in the DSL toolkit. (Which is good news in and of itself as hand-writing the XML dsldd file is a pain in the butt. Though until then there's the great Dm2Dd tool from Modelisoft). The iterative process he describes certainly looks a lot like development, in the same way that C# development looks like C development. Similar steps taken on different concepts. Additionally, he's working bottom up - the output of the model will eventually be a working program (a designer in this case) which is the point I made in Abstraction Gap Leapfrog. There are existing abstractions that work now (i.e. the code generated from the existing dsldd file) and he's trying to building something one level up from there.

I also like Stuart's use of "fidelity" instead of my use of "complete". Stuart uses it as an indication of how correct a given model is. That's what I was implying when I said "complete" but "fidelity" captures the idea much better. I could imagine both lo-fidelity and hi-fidelity models for a given domain, though I would imagine you would always want to use the highest fidelity model available. The difference would be the amount of custom code you have to write - the higher the model fidelity, the less code you write by hand. And I would imagine the model's fidelity would evolve over time, which introduces interesting questions regarding language evolution as well as the evolution of projects built with those languages.

I hope Stuart keeps blogging about this project.

Abstraction Gap Leapfrog

One of the cool things about having a blogging conversation with someone on the other side of the world is that while you sleep they are thinking of a good response to your post. The only downside? Having to deal with rampant misspelling like "artefacts". :)

Anyway, Gareth responds to my post:

Until we get models that are perfectly aligned with our business domains, we'll have people who want to create models but who get them slightly wrong from a precision point of view - usually in the places where the imperfect models interact with other aspects of the system across or down the abstraction stack.

With code, you'd likely not want to have people check in sources that don't even compile and then hand them off to other folks who do make them compile, but I think that's exactly the type of process we'll see emerging in modelling for a while. I feel this way because I don't foresee us getting modelling languages of pure business intent 100% right for some time yet - we're simply not close enough to formal enough descriptions of systems as intensely human as a business yet. However, I hope we won't want to try and keep modelling as locked away with the techies as traditional development has been. (Hope I'm not talking myself out of a job here…)

[Pseudomodels and intent]

I keep saying incomplete and Gareth keeps saying imprecise, but I think we can both agree on the term "imperfect". There's a massive difference between having an precise language that is imperfect versus a language that is inherently imprecise like UML.

However, I think the primary disconnect here has to do with Gareth and my views on how higher abstracted languages will evolve. Gareth's comments about modeling "pure business intent", having "models that are perfectly aligned with our business domains" and not "keep[ing] modelling as locked away with the techies" imply to me that Gareth wants to work down from the high level business abstractions into implementable technical abstractions. Frankly, I don't think that's very likely. Leapfrogging a few levels of abstraction hasn't worked well in the past (CASE and UML/MDA) and I don't think it will work well now.

I find it much more likely that we will build higher level abstractions directly on top of existing abstractions. Again, this is similar to the way C++ built on C which in turn built on ASM. Sure, that could keep modeling "locked away with the techies" for a while, but we're already beginning to see the light at the end of that tunnel. Windows Workflow Foundation is a significant leap in abstraction while also being something than non-techies can use. Reports about about Sharepoint "12" embedding the WF engine and FrontPage "12" providing a Workflow Designer for building SharePoint workflows. While I imagine (and I haven't used any of the new Office "12" suite so this is pure conjecture) these WF tools are targeting the "power user", they certainly aren't only for developers.

Believe me, I would love to be wrong about this. I would much rather work down from or business user intent than up from the technical foundation. I just don't think it's feasible. The process Gareth describes breaks the "Model Transformation must be Deterministic" tenet of Code is Model, though the word "must" may be to strong to allow for language evolution.

Imprecise vs. Incomplete

Gareth responds to the first tenet of Code is Model:

[A]lthough as an industry we desperately need to drag models kicking and screaming from the far left of pretty-picturedom a good long way to the right in the direction of precision, I don't want to throw the baby out with the bathwater.

I'm going to take it as a given that folks believe that precise models are valuable development artefacts. Why do I think imprecise models are also valuable? Here are three things that tools for imprecise models help you to do:

1. Communicate with people about design
2. Think out loud in a way that's more shareable that your whiteboard
3. Start with an imprecise model and progress gradually toward precise models

Hopefully the first of these is obvious - there is value in model as communication device - it's just not enormous.  I've talked before about the value of the second - I draw pictures on my whiteboard and when I'm on a conference call to Redmond, they're effectively useless.

The third is something I've only recently become a convert to.  I'm happy to have models which are not precise so long as I can still reason about them programmatically.  This allows me to have development processes that are about a quantitive process of iterative refinement.

Here's an example - in some infrastructure modelling tool, I have a node type which specifies a logical machine group.  One of its properties is the number of actual machines required to suit the proposed scale of the application to be deployed.  I'd like to be able to put "Depends on outcome of Fred's scalability investigation" into that numeric field, or perhaps "4->8".  I can still generate a pretty good report from this model, but I can't really provision a set of physical servers from it.

But here's the kicker - it's vital that I can write tools that programmatically assess this model and tell me what work needs to be done in order to make it precise.  I want to know exactly what must be done on this model before, for example, it is suitable for feeding into some kind of provisioning tool.  You might say it needs to be precisely imprecise; I prefer to think of it as quantifiable imprecision.

[Imprecise Models and Killing Hippies]

The only thing I disagree with Gareth about is terminology. Like the term architecture, model has become a catch-all for things that aren't code. Regular readers of this blog know I like to be more precise than that. As such, I think items #1 and #2 from Gareth's list aren't actually models at all. I think of them as pseudomodels, similar to the concept of pseudocode. Actually, I like the name pseudomodel - it also applies well to Grady's scaffolding. Like psuedocode, pseudomodels have tons of value in communication and reasoning about a problem but they can't be used as development artifacts.

As for the third, I think what Gareth is describing is an incomplete model, rather than an imprecise one. If the model is imprecise, there's no way to programmatically reason about it. But if we look to code as an example, obviously, there are many cases where code is incomplete. Every compiler error you've ever seen is an example of incomplete code. And because the language is precisely specified, the compiler can tell you what needs to be done in order to make it precise, exactly as Gareth requested. I don't think of writing code as "progressing gradually towards precision" and I doubt anyone else does either. And while I do see development as an "iterative process", I don't think of it as "iterative refinement". Modeling shouldn't be any different.

One area where I do see refinement being critical is in the development of the modeling language itself. Traditionally, the language stays stable while the program written with it changes. But with the introduction of DSLs, it becomes possible for both to vary independently. I would assume that a DSL would evolve over time to have better "coverage" of a given domain. For example, if I was building a CAB DSL, I would implement support for WorkItem right off the bat, but supporting WorkItemExtension would be much lower on the priority list. This represents language refinement, but I would argue it's a refinement of coverage not a refinement of precision.

Scaffolding Isn't a Model

Grady Booch @ the OOPSLA 05 Structured Design and Modern Software Practices Panel at (according to the transcript)

The most important part is the executable code. I typically throw my models away, but I always save my source code. I design because I need abstractions to help me reason out my projects.

Grady Booch ranting on his blog:

It's sad how one can be misquoted and then for that misquote to be picked up by someone else with both then making a spin of the events to support their position. How silly is that.

Juha-Pekka Tovanen quoted me from my OOPSLA panel as saying that "when the project gets closer to the delivery you normally throw away UML models."

<snip>

Let me be excruciatingly clear: Over the years I have been consistent in saying that in a) the most important artifact of any software development organization is executable code and yet b) modeling is essential in constructing such executables. This is because c) models help us reason about, specify, construct, and document software-intensive systems at levels of abstraction that transcend source code (and the UML is the accepted open standard for doing so). That being said, it is a pragmatic reality that d) some models are essential (and should be retained) while others are simply scaffolding (and should be discarded). I have never said and do not say now that one should throw away all models, as Juha-Pekka then Harry then Steve imply.

First off, Grady claiming to be misquoted his pretty disingenuous. Juha-Pekka's account of what Grady said on the panel is pretty spot on. Furthermore, Grady claming that I implied he said "throw away all models" is also disingenuous. I specifically wrote that I thought Grady was being taken out of context:

I've gotta believe that this comment was somehow taken out of context and that the Grand Poobah of the Common Semantic Model doesn't actually believe that tossing the model at the end of the project is a good thing.

But he-said/she-said nitpicking aside, where's the guidance on which models are essential and which are "simply scaffolding". Last time I checked the UML spec, none of the models are labeled as disposable. How about Rational Rose? Any dialog boxes that pop up reading: "Don't worry keeping this model up to date, it's just scaffolding"? I don't think so.

Obviously, working at a higher level of abstraction helps you reason about a project. But reasoning at high levels of abstraction doesn't mean you're modeling. When a building architect sits down with a prospective customer and a sketchpad, they may be working out great ideas but no one is going to call the result a blueprint. Grady's scaffolding "models" break nearly every tenant of Code is Model. Scaffolding isn't precise or deterministic. And if it ends up in the recycle bin, I guess it's not intrinsic to the development process.

Actually, it's good that scaffolding isn't a model. It means Grady is specifically not suggesting to throw away models. He just needs to get his terminology right.

As for Grady's request for "just one DSM out there in production", Don lists a few: Workflow languages (XLANG and WF), Business Rules Engine Languages and Build lanugages (MSBuild, Ant and NAnt). Juha-Pekka pointed to this list of DSM case studies. I'd also add UI Designers such as the Windows Forms and ASP.NET designers in Visual Studio. In the Window Forms case, the code is stored in a seperate file (yay partial classes) and are specifically marked: "do not modify the contents of this method with the code editor." In the ASP.NET case, the code for an ASPX file isn't even generated until runtime. And how about HTML itself? I'm thinking HTML qualifies as "in production"?

JavaScript Utilities

Can you tell it's a slow day in the office? :)

Speaking of raising the level of abstraction as well as browser based applications, check out Nikhil's JavaScript Utilities project:

The project introduces the notion of .jsx (extended JavaScript) and .jsa (JavaScript assembly) files. JSX files provide the ability to include conditional code via familiar preprocessor directives such as #if, #else, #endif and so on...The tool processes these directives in order to produce a standard .js file. JSA files build on top of .jsx files. While they can include normal JavaScript and preprocessor directives, they are primarily there for including individual .jsx and .js files via #include directives. This allows you to split your script into more manageable individual chunks.

Now, that's not raising the level of abstraction much, but here's another example of working in a higher abstracted environment (jsx and jsa) and then compiling down to something the underlying platform can execute (js). Nikhil provides three ways of doing this compliation:

1. A set of standalone tools that output standard .js files that you can then deploy to your web site. Command line parameters are used to control the behavior of the tools.
2. A couple of IHttpHandler implementations that allow you to dynamically convert your code into standard .js files. This is the mode where you can benefit from implementing per-browser code in a conditional manner. AppSetting values in configuration are used to control the conversion behavior.
3. As a script project in VS using an msbuild project along with an msbuild task that comes along with the project. MSBuild project properties are used to control the conversion behavior.

If you're going to raise the level of abstraction to do implement a preprocessor, you could also go all the way and implement an entirely new language that gets compiled down to JavaScript for execution in the browser. For example, I'm not as familiar or comfortable with JavaScript's prototype based approach. But I could imagine a more class based language that compiles to JavaScript. That's the same way early C++ compilers worked - they were a preprocessor pass that converted the C++ into C, which could then be compiled with traditional C compilers.

I wonder what JavaScript++ would look like

As Simple as Possible, But No Simpler

Chris Bilson left the following comment to my Thoughts on CAB post

I hesitate to agree that raising the abstraction level of tools is a good idea. That's just hiding the complexity that's already there inside of more complexity. If you ever need to look under the hood, it's even harder to grok. I think it would be better to go the other way. Try removing stuff to combat complexity.

Given that the software industry has been raising the level of abstraction of tools since the start, I found this comment surprising. Assuming Chris doesn't write in assembly code, he's leveraging something at a higher level of abstraction that's "just hiding the complexity that's already there". As I wrote in Code is Model:

The only code that the CPU can understand is machine code. But nobody wants to write and debug all their code using 0’s and 1’s. So we move up a level of abstraction and use a language that humans can read more easily and that can be automatically translated (i.e. compiled) into machine code the CPU can execute. The simplest step above machine code is assembly language. But ASM isn't particularly productive so work with, so the industry has continuously raised the level of abstraction in the languages they use. C is a higher level of abstraction than ASM, adding concepts like types and functions. C++ is a higher level of abstraction than C, adding concepts like classes and inheritance. Each of these levels of abstraction presents a new model of the execution environment with new features that make programming more productive (and sometimes more portable). [emphasis added]

I feel like Chris has mischaracterized what I wrote. Here it is again:

If you can't lower the complexity of your framework, it's time to raise the abstraction of your tools.

Note I'm not advocating raising the level of abstraction for abstractions sake. Believe me, I'm hate overly complex code. The project I've been on (and can blog about soon I hope) had a tone of overly complex code that wasn't particularly germane to solving the problem. I yanked all that code and have reduced the framework to a quarter it's original size while adding functionality. But the reality is, simplification can't always be achieved by removing code. To paraphrase Albert Einstein, solutions to problems should be as simple as possible, but no simpler.

CAB is the simplest solution to the problem it addresses, and no simpler. Since we can't make CAB simpler and still solve the problem, the only alternative we have is to have the tools hide that complexity. Given how well this has worked in the past, I see no reason why it can't work again in the future.

He's Bad at Taking Advice, but Man is He Smart

He in this case is Tim Mallalieu from the data programmability team. He and I used to work on the .NET Adoption Team in the field together before we both moved to campus. Tim, if I recall correctly, was the first outside hire we made on that team. We used to constantly argue about who was the second smartest on the team. He thought it was me and I thought it was him. We both agreed the Jeromy was the smartest.

Anyway, after literally years of haranguing, Tim finally started a blog. But he's still not really listening to me. I told him he should start with a series of smaller posts and he didn't. 

I'm back working with Tim in a fashion as both my team and his team try and get a handle on how data is going to evolve as we move to service orientation. Most of the existing data paradigms and semantics such as table joins and transactions assume everything is together in one database. Obviously, as we break down large systems into smaller services, having everything in one database just isn't practical.

Browser Based Applications in the Enterprise

I asked Scoble for his thoughts on my post about the term mashup, and he decided simply to link to it. While I appreciate the traffic, I'd appreciate opinions even more. Luckily, in the same post he linked to Sam Ramji writing about what he called "enterprise mashups". Again, love the idea but hate the name mashup.

According to Sam, Scoble said:

[W]hat's to prevent mash-ups from being the main way that departmental apps are built in most enterprises 3-4 years from now?

Again, it depends on the definition of the term "mashup". Are we're talking about browser component based apps that leverage what Scoble calls ICC's? If so, Scoble is only wrong about the timeframe. My team has a browser based application leveraging Virtual Earth running on our intranet server right now.

Quick terminology sidebar: I use the term browser based application to refer to these AJAX and mashup style apps, since they download code and run in the browser itself. By comparison, I use the term web based application for the more traditional browser apps that did all the processing on the server and used the browser essentially as a dumb terminal.

While I think these browser based applications will go mainstream in the enterprise soon, there are a couple of things holding up adoption: availability of tools and of the components themselves.

On the tools side, the AJAX programming model is just too difficult for most programmers. Rudimentary debugging support, late bound script languages, no compiler to help you find errors. Sounds like web development circa 1997, doesn't it? I'm hoping Atlas will be to AJAX what ASP.NET was to ASP.

As for existing components, most of the ones Scoble calls out have little relevance to the enterprise. There's little point to a Feedmap or a Flickr bar in my enterprise app. And I'm not sure I even want to go into the implications of serving up ads in an enterprise app. That pretty much leaves mapping components like VE and Google Maps as the only browser components of interest to the enterprise developer. So far anyway.

I wonder if Scoble knows of any other enterprise relevant browser components out there?

Thoughts on CAB

As I wrote earlier, I'm really impressed with p&p's latest release - the Composite UI Application Block. I had to fly to Atlanta for a customer meeting last week and I spent most of my spare time (flying, in hotel, etc) digging thru CAB. Well, digging through CAB and reading State of Fear by Michael Crichton. IMO, it's his best book in quite a while, perhaps second best he's ever written after Jurassic Park.

Back to CAB. First off, I want more information about ObjectBuilder. p&p's dependency injection framework is very impressive. However, with the exception of the code, the code reference in the help files, the unit tests, and a single article on how it was designed there just isn't much info on it. For example, I know I can use it to inject a concrete type where a class is expecting an abstract type. But I couldn't figure out how so I went and asked Peter and Brad. (Short answer - use TypeMappingPolicy). Do me a favor, go contact Peter and David and tell them you want more info on ObjectBuilder. Sample code especially. Go on, drop them a line right now while you're thinking about it. I'll wait.

Back already? Cool. One thing you can't help but notice about CAB (and OB for that matter) is the heavy use of attributes, which I feel is an extremely elegant solution. Remember the first time you looked at NUnit? How sensible it seemed to use attributes like [TestFixture], [Test] and [ExpectedException] compared to what other xUnit frameworks provide? Get ready to experience that all over again when you look at OB and CAB. Now you're looking at attributes like [CreateNew], [EventPublication] and [CommandHandler]. There's a reason why Sun cloned attributes for J2SE 5.0 - it's powerful (in the right hands). The attributes both drive human readability - when you're looking at a property adorned with [CreateNew] or [Dependency] it's obvious that these are injected - as well as the implementation. Win-win as far as I'm concerned.

CAB does a great job of codifying standard patterns in smart client design, such as events, commands and controllers, as well as implementing completely new ideas such as workspaces, work items and smart parts. And they've done it with an eye on the future. I love the isolation between the windows forms specific parts of CAB and the underlying infrastructure of CAB. Check out this picture of the layers of CAB. By isolating all the WinForms specific stuff in a separate assembly, they are well set up to support WPF while minimizing redundant effort. I can't wait to see a Microsoft.Practices.CompositeUI.Avalon project (because Microsoft.Practices.CompositeUI.WindowsPresentationFoundation is just to long a namespace).

However, I can't shake the feeling of how complex this stuff is. Yeah, you use CAB for solving complex problems, but there is a significant learning curve here. I imagine debugging a CAB app will be significantly non-trivial. Take a look at the final step of the walkthru app. Here's the sequence that gets "hello world" painted on the screen:

1. Main entrypoint creates and runs the ShellApplication.
2. ShellApplication creates a ShellForm window, which in turn contains the tabWorkspace1 workspace.
3. ShellApplication dynamically loads MyModule because it's listed in the ProfileCatalog.xml file.
4. CAB creates an instance of the MyModuleInit class from the MyModule assembly.
5. MyModuleInit creates and runs an instance of MyWorkItem.
6. MyWorkItem creates a MyView and MyPresenter.
7. MyWorkItem adds the MyView instance to the tabWorkspace1 workspace, hosted in the ShellForm
8. MyPresenter handles the MyView load event and sets the message property of MyView to "Hello, World"

Of course, the point of a demo app like the walkthru is comprehension. You would never use CAB to build this Hello World app. But I worry that the level of complexity will put CAB beyond the reach or inclination on many potential users. I imagine this single line of code will scare off many would-be CAB developers:

MyWorkItem myWorkItem = parentWorkItem.WorkItems.AddNew<MyWorkItem>();

Given that most people are used to writing "new MyWorkItem()", the line above represents a significantly rise in complexity. Of course, CAB is trying to solve a complex problem. I'm sure CAB's designers would rather the solution wasn't so complex, but that's the reality of the problem space their dealing with.

If you can't lower the complexity of your framework, it's time to raise the abstraction of your tools.

I wonder what CAB specific tooling would look like? At a minimum, it would like built in support for the primary concepts in CAB - WorkItem, SmartPart, Workspace and Service to name a few. Another opportunity would be to move from embedded strings, used to identify events, commands, UIExtensionSites and the like to true variable-style names that could be validated at compile time, sort of like how LINQ is extending C# to get rid of embedded database query commands. There's lots of possibilities and the more I work w/ CAB the better idea I'll have about them.

I Hate The Term Mashup

Wikipedia has two definitions of mashup:

• Mashup (music): a musical genre consisting of the combination of the parts of more than one song
• Mashup (web application hybrid): a website or web application that combines content from more than one source

I was originally introduced to the term mashup in the musical sense by Daily Source Code. In this usage, mashup implies mixing songs together. Given that songs are typically stand along entities designed to be enjoyed as is, mashup means (to me anyway) combining stuff that was never meant to be combined in the first place. Even Wikipedia includes musical mashup under the general heading of Bastard Pop.

So given that, by definition, mashups combine stuff that wasn't designed to be combined, I don't understand why an application like Zvents or Virtual Places is considered a mashup. Apps like these use browser based components (Scoble calls them Internet Connected Components) that are well defined, have public APIs and are designed to be used together. Not exactly "bastard pop" now is it?

In reality, a site like Zvents could have used a server side mapping component and provided a similar experience. Of course, a client side mapping solution is both sexier and more practical (no need for dedicated map data files or assemblies on your own machines) but semantically they provide the same information. Same thing goes for Virtual Places, except that Virtual Places is also pulling both functionality (i.e. the mapping component) as well as data (i.e. blogs and photos) from other sites across the Internet. Could that be done on the server side? You betcha. Would it be as cool or functional? No. Does that make it a completely different type of application that deserves a new name. IMO, no. These are component based apps - they just use the browser as the platform and the components are coming across the web (as you would expect when you use the browser as a platform).

The higher order bit for me is who controls the experience. For apps like Zvents and VirtualPlaces, it's the application developer. For something like Live.com, it's me. I decide what to put on my Live page. Not that one is more important than the other or that they aren't compatible experiences - I could easily imagine a Zvents gadget that lived in Live.com. Or consuming the RSS feed from Zvents in Live.com. Or a generic iCal gadget that could consume a Zvents iCal feed. But the point is that there are large differences between browser component based application like Zvents and a user managed composite browser application like Live.com.

To me, composite apps like Live.com fit better to the original definition of mashup than something like Zvents or Virtual Places. They both have their place and their value, but I like to call things that are different by different names in order to reduce confusion.

Of course, composite apps aren't limited to the browser. p&p just shipped their Composite UI Application Block last week. I dug into it a bit last week and it's awesome. More on that later.