The SOA Manifesto – Pointlessness Manifested

You know what the Agile Manifesto doesn’t have? Video of a “very formal ceremony” announcing said manifesto. Instead, we just have artistically rendered picture of what sure looks like Martin Fowler pointing at a white board while some of the other original signatories look on. Sure, it’s a cool picture, but wouldn’t it have been much cooler if they had captured that moment on video instead? Especially if it was video of them all standing around looking vaguely uncomfortable while photographers took their picture and someone gravely read the manifesto to give it artificially inflated importance.

I only watched ten seconds of the SOA Manifesto announcement video before I realized there’s nothing to see here, move along, just a bunch of navel gazing from the usual SOA suspects.

Seriously, if you having a big announcement about how cool, earth shattering, significant or, hell, even interesting your manifesto is, then it’s not any of those things. It’s a waste of my time.

Then I noticed that my previous manager and personal friend John deVadoss is one of the signatories. I have metric tons of respect for John, so I gave the SOA Manifesto a second chance.

It lost me at the second sentence.

Service orientation is a paradigm that frames what you do. Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation. We have been applying service orientation to help organizations consistently deliver sustainable business value, with increased agility and cost effectiveness, in line with changing business needs.

Are you frakking kidding me? “Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation.” Who the hell came up with that? I didn’t realize the International SOA Symposium had a Department of Redundancy Department.

When you define SOA in terms of SO, then you can’t possibly score well on the practicality quotient.

The rest of the manifesto isn’t much better. What made the Agile manifesto great IMO was that it ran counter to “common” beliefs like “changing requirements late in the process is bad” and “shipping software cycles should take years”. Sure, we all realize how right those Agile manifesto guys were now, but at the time it was the next best thing to heresy for many organizations. Those guys were agents of change. And I mean real change, not “this will help me sell books” change.

SOA manifesto on the other hand is basically repackaged common sense. Stuff like “Recognize that SOA ultimately demands change on many levels” and “The scope of SOA adoption can vary. Keep efforts manageable and within meaningful boundaries” Any help figuring out what that change is or what the scope should be? Nope. Thanks for the advice guys, but I had your so called “Guiding Principles” figured out long ago.

But hey, I’m sure the manifesto will help Thomas Erl et. al. sell more SOA books. So, I guess from that perspective it’s mission accomplished.

Architecture Astronauts and Over Engineers

Since it’s apparently Architecture Week™ [1] here at DevHawk, here’s another of my favorite Dilbert cartoons of all time – relevant to the discussion at hand.

Two interesting comments on yesterday’s post:

Architectural thinking is a necessary (and very important) part of software development - but beyond the systems level (which is systems administration and not software architecture) I have a hard time seeing divorcing architectural thinking from the actual development as anything but a terrible thing. Although I see that your definition of architecture (at the functional level) does not match my caricature of the 'architecture astronauts' which I do think can be endemic in languages that encourage additional layers of architecture. [Michael Foord]

So based on the definition of architecture I'm reading into your post, you wouldn't consider the choice of object-oriented versus functional programming styles from an architectural perspective? I'm trying to understand what level of architecture you mean here. Like Michael, I usually think of architecture even down into the implementation patterns level (hence the architecture astronauts), but that seems to be included in what you might be calling an engineering concern. [Ryan Riley]

Let me be very clear. Using my definition, there is no such thing “architecture even down into the implementation patterns level”. I’d argue that the implementation patterns level is engineering, not architecture. From what I’ve seen, the terms “architecture” and “engineering” tend to be used interchangeably in the software industry, and frankly I think that’s a mistake. I said as much in yet another post I wrote four years ago:

Architecture is the intersection between business and IT.

If a decision doesn't effect a business person, it's not an architecture decision. I'm not saying it's not important - I think the role of the software engineer is critical in large-scale enterprise system design and construction. And I will readily admit that often a single person is responsible for both architecture and engineering. But that doesn't make them the same activity. As long as we continue to confuse the two disciplines, we hold them both back.

Michael and Ryan (or anyone else for that matter) are welcome to disagree with my definition of architecture. I often joke that if you asked ten architects to define “architecture”, you’d get twelve answers. But that’s my definition and I’m sticking to it.

But what of the Architecture Astronauts? Both Michael and Ryan mentioned them. Unsurprisingly, I think that term is used too broadly as well. If you go back and read Joel’s original post of Architecture Astronauts, there wasn’t much reference, if any, to the implementation layer at all.

The Architecture Astronauts will say things like: "Can you imagine a program like Napster where you can download anything, not just songs?" Then they'll build applications like Groove that they think are more general than Napster, but which seem to have neglected that wee little feature that lets you type the name of a song and then listen to it -- the feature we wanted in the first place. Talk about missing the point. If Napster wasn't peer-to-peer but it did let you type the name of a song and then listen to it, it would have been just as popular

[Joel on Software, Don't Let Architecture Astronauts Scare You]

I feel that my definition fits very well with the way Joel writes about architecture in this paragraph. The Architect Astronaut is trying to solve a real business problem - people need access to information besides music. But the mistake they make is thinking they can solve multiple problems with a single solution. So they abstract higher and higher until they’ve lost sight of the original problem and can only focus on the abstractions. If you look at what Joel has to say about technologies like Hailstorm and Jini, you see the same pattern emerge.

This isn’t to say that similar problems of over-abstraction don’t happen at the implementation layer – they do. But they happen for very different reasons. Astronaut Architects are trying to solve multiple problems with a single solution. But when over-abstraction happens at the implementation level, it because someone thought they could predict the future.

We’ve all seen our fair share of over-engineered systems that introduce significant unneeded complexity on the off chance that the development team can successfully predict the kind of change likely to come in the next version of the product. Invariably, the team’s precognitive abilities are revealed to be as poor as everyone else's, so they’re left with a bunch of extra layers of software cruft that has to be maintained but provides zero additional value to the system. I’ve blogged about that problem before as well: Kitchen Sink Variability.

Since I’m big on keeping the terminology of architecture and engineering separate, then I’d argue that we need a different term than Architecture Astronaut for people who want to introduce additional layers of abstraction at the implementation layer on the off chance that they don’t suck at precognition. Since we call such systems over-engineered, wouldn’t that make the people who build them “Over Engineers”?

[1] It’s like Shark Week, but with white boards and even more terrifying.

Dynamic Languages in Architecture

In the comments from yesterday’s post, IronPython MVP and author extraordinaire Michael Foord asked:

Has your view on architecture as a discipline separate from coding changed since working with dynamic languages?

In a word:“No” (though as always, I reserve the right to be wrong and/or convinced otherwise.)

When I was an architect, I tried very hard to treat it as a “discipline separate from coding”. To use my last post as an example, building a central repository of system audit information is an architectural decision. A bad one IMHO - at least the way Dilbert’s PHB described it - but an architectural decision all the same. It was a decision about what kind of system to build, part of an overall application portfolio, as opposed to a decision about how to build the system.

I’ve held this opinion of architecture for a long time. Four years (and three jobs) ago, I wrote the following:

IMO, building a system that has a set of functional requirements (track customers, process orders, etc) and non-functional constraints (sub-second response time, support 10,000 concurrent users, use Microsoft Windows platform, etc) is an engineering problem. Coming up with the lists of functional requirements and non-functional constraints is the architecture problem.

Working with dynamic languages has dramatically changed my view of engineering and design of individual systems. But from the pure architecture perspective, I want to be able to treat individual systems as black boxes as much as possible. That means the programming language is an implementation detail that shouldn’t matter to the architect.

Note the significant bet-hedging language in the paragraph above. I’m using phrases like “shouldn’t matter” and “as much as possible” because we all know that there’s no such thing as a “pure architecture perspective”. Unlike building architecture, software architecture is in constant flux at every level. At the enterprise level, there are always new regulatory obligations, new competitors and new partners to consider. At the end-to-end process level, there are always new systems or new version of existing systems coming on line. And at the individual system level, there are always new – or at least new versions - of tools, frameworks and languages being released.

Once you introduce time into your architecture perspective, individual system engineering will affect the overall architecture, since system engineering affects the rate of change. Language choice will certainly have some engineering impact. However, in my experience language choice is rarely high on the list of concerns relative to things like project scope and team experience.

So my “No” answer to Michael’s question is predicated on the following:

• As an architect, I want to consider individual systems as black boxes where implementation details like language choice are completely irrelevant.
• As a practical architect, I realize that some system implementation details are relevant – especially over time - but in my experience language choice isn’t one of them.

On the other hand, most IT shops try to standardize on one programming language – certainly MS IT did – so maybe language choice would be more architecturally relevant in a mixed language shop. I’d love to hear from folks who have multiple standard languages in their IT shop – especially if you have both static and dynamic languages on your standards list.

Probably Wrong Info Is Worse Than No Info At All

Like many geeks, I love Dilbert. However, I rarely identify with it as well as I did Sunday.

I kid you not, I’ve had almost exactly this conversation back when I worked in MS IT. They have this big repository of information about deployed applications. Technically, you’re not supposed to deploy an application without listing it in the application repository. Like Dilbert, I never really understood what people were going to do with this information, but the projects I was on dutifully collected the relevant information and put it into the repository.

And never thought of it again. Ever.

And therein lies the problem. Populating the application repository was an artificial step on the critical path of the deployment process. Writing the software, acquiring the physical hardware to run it on, stuff like that really is on the critical path. Populating the application repository was extra busy work legislated by someone (I forget if it was the central architecture team or management) that didn’t benefit the project in the slightest. As such, it was given the minimal about of attention and effort, meaning there was little quality or consistency in the data. Worse yet, when the application changed or was decommissioned , updating the application repository just didn’t happen. I mean, it was supposed to, but rarely did.

So you ended up with a repository of information that was worse than useless. I had a colleague who insisted that the repository had some value because “not all of the data was wrong”. Of course, he couldn’t tell me with any consistency which data was accurate and therefore valuable and which was not. Hence, my argument that it was “worse than useless”.

The only way an application repository is going to be of any value at all is if you can collect the data automatically. My old teammate Buzz coined a phrase we used often: “The Truth Is On The Edge”. You should always regard any central repository of information with a very critical eye since it’s rarely going to be the truth.

(Ed. Note – Man, it’s been a long time since I’ve written about Architecture. My last Architecture post was almost a year ago. I don’t miss the job but I do miss my old teammates – in particular Buzz, Rick, Dale and of course Nick Malik.)

Morning Coffee 171

• Big news for IronRuby out of OSCON. John and Jim have the details. Congrats to the IronRuby folks on reaching these milestones and paving the way for others (i.e. IPy) to follow some of the same paths.
• One of those OSCON announcements, is a project my teammate Jimmy Schementi has been working on: Silverline, which "let's you write Rails code that can run on the client".
• Shri Borde - the dev manager for IPy, IRuby and F# - tackles a tricky subject of static compilation of dynamic Python code. This came up on the mailing list recently as one of the outstanding requests for IPy to do is support custom attributes, which requires static compilation. Shri lays out some of the big issues with this approach. However, the community has been fairly clear on this, so it's obviously something we need to look at.
• I met someone from MS Research at the MS Product Fair who pointed me to the Institute for Personal Robots in Education, a joint effort between Georgia Tech and Bryn Mawr College and sponsored by Microsoft Research. Their Myro software (myro == my robot) is written in CPython, but there's an effort underway (aka Miro 3.0) to build a .NET version that uses IronPython. Must investigate.
• Seshadri shows how easy it is to extend C# types in IronPython. It's also shows how simple it is to host DLR code in your app - it's like 6 lines of code!
• Early reviews of IronPython in Action are good.  
• If you want to run an IronPython IDE in your browser with Silverlight, check out SilverShell from Dan Eloff.
• The XNA team has announced their business plans for community games. Basically, you set a price point between 200 and 800 points (aka between $2.50 and $10) and receive a "baseline" of 70% of the revenue the game generates. More details are available in the FAQ. This is pretty excited. I'd like to build some co-op kids games.
• Speaking of XNA, Caligari is now offering TrueSpace 7.6 for free . David Weller and Glenn Wilson provide an XNA viewpoint on the announcement, Chris Pendleton shows how to upload your models to VirtualEarth.
• Congrats to the CodePlex team on their latest drop, which features that a cool new feature - Mailing Lists! IronPython has had a Mailman mailing list for years, so I'm not sure we'll use this feature on IPy, but I'll investigate it
• Two PDC notes: First, Rick Rashid - VP of MS Research - will be delivering a PDC keynote. Second, the PDC team has put up a video podcast on Producing a Ginormous Conference in 10 Minutes or Less! It's the "inaugural episode" so watch for more Countdown to PDC video podcast episodes in the future.
• I recently discovered Chris Smith's F# blog. He's got recent posts on Mastering F# Lists and Guidelines for Readable F# code. For the F# novice, check out his F# in 20 Minutes posts (part one, part two)
• Pat Helland is moving to the SQL team. Good luck Pat!
• I like Nick Malik's formal definition of use cases, but I can't help be reminded of Charlie Alfred's Value-Driven Architecture article in Architecture Journal 5 where he said use cases were "easy to teach and explain" but that "if simplicity were the only goal that counted, we'd all still be walking or riding horses to get from one place to another."

Morning Coffee 166

Yes, I realize it’s been a while. I tried in vain to catch up with my blog reading after my Hawaii vacation and finally just gave up and hit “mark all as read”.

Dynamic Languages

• There's a new version of the DLR hosting spec available (doc, pdf). The DLR implementation is still in motion, so there are some inconsistencies between the spec and the code, but the spec should give you the high level overview you need if you want to host DLR languages inside your app.
• Oleg Tkachenko recently joined the dynamic languages team. He's the creator of the Interactive IronRuby Web Shell, an IronRuby version of Try Ruby. Of course, it’s not as cool as using SL2to execute the code directly in the browser. Michael Foord has his Python in the Browser and my teammates John and Jimmy demoed a Silverlight version of Try Ruby @ TechEd.
• Jim Deville, also of the dynamic languages team, recently started blogging.
• I have a new boss, Dave Remy. He doesn't have a blog - yet - but you can follow him on Twitter as daveremy. When Twitter is actually working that is.
• There's a new homepage/wiki for IronRuby though I’m not sure why there's a picture of Matz wearing a Python shirt on the home page.
• My teammate Jimmy Schementi provides some "continued hope" for a better (heck, I'll take current) ASP.NET and ASP.NET MVC story for DLR languages.
• Via Michael Foord, sounds like IronClad is making good progress. V0.4 can run the bz2 module "in its entirity" (maybe run a spellcheck on your site, guys?) and now apparently, it's now able to load numpy.core. Very exciting!

Other Stuff

• Pat Helland, who has blogged even less than me for the past few months, has a post up about controller and doers in the IT department. After 18 months in MSIT, put me in the doer camp, please.
• The F# team has pushed out a spec for v1.9.4 of the language. Don Syme says it's not official, but it's a huge improvement over the old informal spec
• Speaking of F#, my friend Matthew Podwysocki recently published FsTest, a testing DSL for F#. I wrote about F# unit testing as part of my PEG parsing series, and I really like the direction Matthew has taken this project. You can pull it down from CodePlex.
• When I did my PEG talk @ Lang.NET, Gilad Bracha mentioned I should check out oMeta. It looks really cool, though with the job change I haven’t had the time to play with it. Now I discover that Jeff Moser is working on a version for CLR called oMeta# that I’ve got to spend some time with. And in the comments to that post, I discovered pyMeta from Allen Short, though it apparently doesn’t work on IronPython (must investigate why).
• James Kovacs introduces psake, a PowerShell based build automation tool which uses a rake-inspired internal DSL syntax similar to one I blogged last year. I'd love to see this take off, but given MSBuild's tool integration, I wonder if that's feasible.
• I upgraded my home wireless network almost exactly a year ago. I've been happy with the range and coverage, but not so happy with the Buffalo Tech firmware. The built-in DHCP server is pretty flaky. So I upgraded to the open-source Tomato firmware. Upgrade was smooth, though I did need to reset my cable modem. But even that was smooth - Comcast has an automated service for that now,

Kitchen Sink Variability

Nick Malik forwarded the last ZapFlash newsletter to me. I gave up on analyst newsletters like this long ago, but Nick shared it with me because it "hit directly on what [Nick] thinks an ESB is and does, and why an ESB is not a hub." I'm not a fan of the whole ESB concept and frankly this article didn't do much to change my opinion. However, this passage did sorta jump out at me.

[T]he main concept of SOA is that we want to deal with frequent and unpredictable change by constructing an architectural model, discipline, and abstraction that loosely-couples the providers of capability from the consumers of capability in an environment of continuous heterogeneity. This means that we have to somehow develop capabilities without the knowledge of how they might be used...[T]his means that we need to have significant variability at a variety of aspects including the implementation, infrastructure, contract, process, policy, data schema, and semantic aspects. Having this variability allows companies to have stability in their business and IT even though the IT and business continue to change. Agility, predictability, reliability, visibility, and cost-effectiveness all become that much more realistic when we can achieve that sort of abstraction.

My reading of this is that the author, Ronald Schmelzer, is advising organizations to introduce "significant variability at a variety of aspects" in their services in order to deal with what he openly admits is "unpredictable change".

This sounds like a mind-boggling awful idea to me.

At it's heart, any practical design - including a service-oriented one - needs to be an exercise in tradeoff analysis. You can't add "significant variability" without also adding significant complexity, effort, time and cost. So the real question is: Is the significant variability Ronald describes worth the inevitable tradeoff in significant time, effort, cost and complexity?

I seriously doubt it.

Since unpredictable change is - by definition - unpredictable, you have no way of knowing if you will actually need any specific aspect of variability down the road. Ronald's strategy - if you can call it that - seems to be let everything he can think of vary except the kitchen sink. That way, when said unpredictable change happens, you might get lucky and have already enabled the variability you need to handle the change with a minimum of effort.

Getting lucky is not a strategy.

Chances are, a specific aspect of variability won't ever be needed. In other words, most of the the time, effort and money you spent building these aspects of variability will be wasted. And remember, this isn't just a one time cost - the increased complexity from including this significant variability means you'll pay the price in additional time, effort and money every time you have to change the system.

I wonder if Ronald is familiar with the term "You Aren't Gonna Need It". He talks about increasing business agility, but he eschews many of the principles of agile development. I realize they aren't the same thing, but I have a hard time believing that they are so diametrically opposed that a core principle of agile development should be readily violated in order to enable business agility.

Maybe it's cliche, but I try to always come back to "What's the simplest thing that could possibly work?". I would think that building a ton of currently-unnecessary variability into your system on the off chance that someday you'll need it fails the "simplest thing that could possibly work" test spectacularly.

Personally, given the choice of taking advice from Ward Cunningham or pretty much any enterprise analyst on the planet, I'll pick Ward every time.

What is the ROI on EA?

Nick Malik took me to task for my suggestion that Enterprise Architecture provides no value.

You implied that I could not answer the question, "How does EA demonstrate value." That is not true. I can readily answer the question, from my viewpoint, but I chose instead to *ask* the question to see if my answer matches the various answers that I may hear back. I got a lot of valuable input, both on the blog and on the forum on Shared Insights where I asked the same question.

You are the ONLY person to reply and say that EA provides no value.

Perhaps you should read about the role and value of Enterprise Architecture from established sources before you bash the entire profession.

EA is real, my friend. It is as real as city planning. The only major city in the US without city planning is Houston. I have visited a few times, and I can honestly say that without city planning, they are a mess.

Nick also provided a link to an article in Architecture and Governance magazine. I was going to read it, however their web site is down as I write this. I feel comfortable interpreting that as a sign that I'm right... :)

Actually, Nick's got a point. It was wholly unfair of me to say that EA provides no value. However, I do believe the return on investment of enterprise architecture is fairly low, perhaps even negative. In other words, I shouldn't have argued that EA doesn't deliver any value, but instead that I don't think it delivers enough value, given what we spend on it.

Architecture ROI is hard enough to calculate on a project by project basis. I would argue that measuring it at the EA level is probably impossible, but I think that's both a blessing and a curse. It's a curse because EA can't justify their existence in terms the business can understand. It's a blessing because if EA is running as deep in the red as I suspect they might be, the company would cut EA entirely in a heartbeat.

Since Nick started my asking a question about value, let me turn it around and ask some questions of my own:

1. How much do you think your organization spends on EA per year?
2. What do you think your organization's EA ROI is?
3. What can you do to improve your organization's EA ROI?

Morning Coffee 150

• Yesterday was the NHL trading deadline, and the Capitals were very busy. They obtained Huet from Montreal, Federov from Columbus and Cooke from Vancouver. Given they are fighting just to make the playoffs, going for three soon-to-be unrestricted free agents seems like an odd choice. However, the consensus (among my parents anyway) was that it's critical to get this very young Caps team some playoff experience. Even if all three walk at season's end, it'll be worth if the Caps make a playoff run. Besides it's not like we gave up much: an extra second round pick in '09, a 19 year old defensive prospect (who was apparently 14th on the depth chart) and an underachieving winger.
• Speaking of the Caps playoff chances, they are currently one and a half games back of the division leading Hurricanes and two games behind the current eighth seed Flyers. Yes, I rank hockey teams using baseball's standings system. Otherwise, you have to talk about games in hand (i.e. the Caps are five points behind Carolina with two games in hand).
• The writer's guild ratified the new contract, so Hollywood labor strife is now officially behind us. At least until July when the the actors may go on strike.
• It seems like a slow week for Microsoft geek news, which is odd since WS08, VS08 and SQL08 all launch today. I'm guessing it's the calm before the Mix storm next week.
• After going dark for six months, Linq to XSD has been re-released to work with the RTM version of VS08. Scott Hanselman demonstrates Linq to XSD by applying it to OFX, an XML Schema he calls "goofy" but apparently helped develop. OFX uses derivation by restriction, which has no direct corollary in C#, but Linq to XSD's  is able to translate between XML and objects without loosing any of that type fidelity. Nice to know Linq to XSD can tolerate OFX's level of goofiness, though I'm guessing most people use much more straightforward schemas.
• Speaking of Linq, I discovered LINQPad via a comment on Rob Conery's blog (which I found via DNK). It's basically a code snippet IDE for C# 3.0 and VB9, with it also has built in database connection support, so it can fulfil much the same role as SQL Management Studio. I only played with it for a few minutes, but I was really impressed.  This is definitely going in my utilities folder. I wonder if they're interested in supporting F#?
• Not sure how I missed this, but you can get MSDN Magazine via same Syndicated Client Experience as Architecture Journal. Unlike AJ which is divided into issues, the MSDN magazine client is divided into topics which is harder to square with the physical magazine. On the other hand, since MSDN Mag has been around longer, perhaps topics + search is a better discovery mechanism.
• Soma announces the Visual Studio Gallery, a repository of VS Extensions. It's kinda cool, but the whole discovery mechanism is clunky. I might like to experiment with some free or even free trial products, but there's no way to filter on cost so finding them is a hassle. Also, there's no way for community members to vote, rate or comment on the products in any way.
• Nick Malik can't answer the question "how does Enterprise Architecture demonstrate value?" I could be snarky and say "it doesn't", but that's only half the answer. It doesn't, but it should. My opinion, since you asked Nick, is that EA fails to deliver value because it tries to control the uncontrollable. Trying to gain efficiency thru establishing standards and eliminating overlap via reuse are pipe dreams, though literally millions of $$$ have been poured into those sink-holes. There are a few areas where centrally funded infrastructure projects can solve big problems that individual projects can't effectively tackle on their own. EA should focus their time there, they can actually make a difference. Otherwise, they should stay out of project's way.

Morning Coffee 147

• My son Patrick turns five today. The big treat was his cousin Jack coming up for a visit. Here's a picture of the two of them at Patrick's party on Saturday. My wife has all the details on her blog. Update: My wife just posted a whole slew of Early Patrick Pictures.
• If my son is five, it means this blog is also five - I started this blog about a month before Patrick was born. I never remember to mark the occasion until Paddy boy's big day comes around.
• Major props to the House of Representatives for growing a backbone and not caving to President 30% Approval on telecom immunity...yet. Personally, I'd like to see the House bury the measure completely, though I'm not holding my breath. But given that even the right-wing Washington Times reports "Analysts say FISA will suffice", maybe the House Dems will do the right thing.
• After tearing it up since Thanksgiving, the Caps have gone a little cold. 5-4-1 in their last ten and 2-2-1 in their last five. In the month of February, they're 1-3-1 against SE division opponents. Good news is that they're still even with Carolina (two points behind with two games in hand), half a game up on Atlanta, a game and a half up on Florida and two and a half games up on Tampa Bay.
• Bill Gates announced a new program called DreamSpark to provide college students access to all of Microsoft's developer and designer tools, including Visual Studio, Expression, SQL Server, Windows Server and XNA Creators Club membership. This looks like an outgrowth of the MSDN Academic Alliance program. I think it's a great idea. Update: Looks like high-school students will be able to access the DreamSpark program too. However, since they're minors, they have to get the software via their teachers. (via LiveSide)
• The winners of the XNA Silicon Minds contest have been announced. Of the five winners, Specimen looks the coolest to me. I wish I had more time to get into game development. (Via LetsKillDave)
• Speaking of game development, this week is the Game Development Conference, so be on the lookout for lots of game-related news. Xbox Live VP John Schappert is giving a keynote on "Unleashing the Creative Community". XNA GM Chris Satchell said last year they would "announce full details on, and ... vision for, opening XNA creations to the community" sometime this year. I'm guessing this is said announcement.
• Speaking of Xbox, there's a rumor that Microsoft and Netflix will announce this week that Netflix is bringing their Watch Instantly service to Xbox 360. If true, sign me up!
• Grigori Melnik announces the GAX/GAT February 2008 final release. Key feature is VS08 support. Is it just me, or does calling it the "final release" make it sound like they won't be upgrading GAX/GAT further?
• Speaking of p&p, Grigori also announces the Feb 2008 CTP of Unity, p&p's new IoC container. I've seem lots of folks echoing the announcement, but not much in the way of specifics on Unity itself. For example, Chris Brandsma describes IoC and mentions Unity, but he doesn't cover any Unity specifics. :(
• MSIT EA Nilesh Bhide has started blogging. His first post is on Customer perception of Service Quality in S+S/SaaS. I've worked closely with Nilesh in the past two years, so I'm excited to see him take to the blogosphere. (via Nick Malik)
• I don't know how I missed it, but the MSDN Code Gallery launched last month. As Charlie Calvert explained, this is logical successor to GotDotNet's user samples area. Between Code Gallery and CodePlex, GotDotNet has finally been shuttered for good.
• Telligent, makers of the very popular Community Server, have released Graffiti CMS, which looks like a more flexible content platform than Community Server. (via DNK)
• In somewhat unexpected news (at least, unexpected by me) Microsoft has released specs for the Office binary file formats. I'm not sure why this is happening now, rather than say when we released the specs for the Open Office XML file formats. (via DNK)

Nick's Flawed Vision of a Shared Integration Model

Of all the things you might say about Nick Malik, "thinks small" is not one of them. He takes on a significant percentage of the .NET developer community over the definition of Mort. He wants to get IT out of the applications business. He invents his own architecture TLA: SDA (aka Solution Domain Architecture). He's a man on a mission, no doubt. And for the most part, I'm with him 110% on his ideas.

However, when he starts going on about a shared global integration model, I start to wonder if he has both hands on the steering wheel, as it were.

Nick's been talking about this for over a year. As he points out, SaaS integration layer is the new vendor lock-in. One of the attractions of SaaS is that you could - theoretically, anyway - switch SaaS application providers easily which would drive said SaaS companies to constantly innovate. However, if the integration layers aren't compatible, the cost to switch goes up dramatically, leaving the customer locked-in to whatever SaaS company they initially bet on - even if that bet turns out to be bad.

OK, I'm with him so far. Not exactly breaking news here - we've seen the same integration issues inside the enterprise for decades. SaaS adds new wrinkles - if your ERP vendor goes belly-up, they can't take your data with them or worse sell it to your competition - but otherwise it sounds like the same old story to me.

However, where Nick loses me is when he recommends this solution:

"To overcome this conflict, it is imperative that we begin, now, to embark on a new approach. We need a single canonical mechanism for all enterprise app modules to integrate with each other. If done correctly, each enterprise will be able to pick and choose modules from different vendors and the integration will be smooth and relatively painless."

Yeah, and if a frog had wings, it wouldn't bump its ass when it hopped.[1] There are so many things wrong with this approach, I'm not sure I can get them all into a single web post.

First off, it won't, in fact, be done correctly - at least, not the first time. I realize everyone knocks MSFT for never getting an application right before version 3.0, but I believe it's actually systemic to the industry. Whatever you think you know about the problem to be solved, it's at best woefully incomplete and at worst wrong on all counts. So getting it right the first time is simply not possible. Getting it right the second time is very unlikely. It isn't until the third time that you really start to get a handle on the problem you're really trying to solve. Getting an effort like this off the ground in the first place would be a Herculean task. Keeping it together thru a couple of bad spec revisions would be impossible.

Meanwhile, the vendors aren't going to be waiting around twiddling their thumbs waiting for the specs to be done. We've seen efforts to unify multiple completing vendors around a single interoperable specification. By and large, those efforts (UNIX, CORBA, Java) have been failures. The technologies themselves haven't been failures, but the idea that there was going to be "relatively painless" portability or interoperability among different vendors never really materialized. If it didn't work for UNIX, CORBA or Java, what makes Nick think it will work for the significantly more complex concept of a shared global integration model? Not only more complex in terms of spec density, but the mind-boggling number of vendors in this space.

Nick is worried that either "we do this as a community or one vendor will do it and force it on the rest of us." But if you look at how specifications evolve, retroactive realization of defacto standards is the way the best standards get created. For example, I could argue that TCP was forced on us by the US Military, but I don't hear anyone complaining. I realize there's a big difference between having a vendor force a spec down our throat vs. a single big customer, but either way it's not designed by committee. Besides, if we do see get an enterprise integration standard forced on us, I don't believe it will be the vendors doing the forcing. If I were a betting man, I'd bet on Wal-Mart. Business leverage trumps IT leverage and Wal-Mart has more business leverage than anyone in this space these days.

BTW, would design-by-committee be an extreme example of BDUF? Do we really want to develop a this critical integration model using the same process that produced the XSD spec?

Finally, Nick thinks that this model will improve innovation, where I think it will have the exact opposite effect. Once you lay a standard in place, the way you innovate is to build proprietary extensions on top of that standard. However, by definition, these extensions aren't going to be interoperable. If someone has a good idea, others will copy it and eventually it will become a defacto standard.

A recent example of the process of defacto standardization is XMLHttpRequest. Microsoft created it in 1999 for IE 5, Mozilla copied it for their browser a couple of years later, followed by the other major browser vendors. Google innovated with it, Jesse James Garrett coined the term AJAX, everyone else started doing it and then finally - nearly a decade later and still counting - a standards body is getting around to putting their stamp of approval on it.

However, if you're worried about painless integration and not having something forced on you by some vendor, then you're not going to embrace these innovations - which means, you won't embrace any innovation. Well, there may be some innovation in the systems themselves that doesn't affect the interface, but once that interface is cast in stone, the amount of innovation will go way down. How do vendors differentiate themselves? There's only two ways: price and innovation. Take away innovation with standardization, and you're left with a race to the rock bottom price with no incentive to actually improve the products.

I get where Nick is going with this. He looks around our enterprise and sees duplication of effort and massive resources being spent on hooking shit together. It sure would be nice to spend those resources on something more useful to the bottom line. But standardizing - or worse legislating - the problem out of existence isn't going to work. What will? IMO, applying Nick's ideas of Free Code to interop code. If we're going to get IT out of the app business, can't we get out of the integration business at the same time?

[1] It's exceedingly rare that you get to quote Wayne's World or Raising Arizona in a discussion about Enterprise Architecture, much less both at the same time. Savor it.

Morning Coffee 134

• Bill de Hora responds to a few of my Durable and RESTful ideas. He points out that relying on a client-generated ID can be troublesome, and recommends using multiple identifiers - one created by the sender, one by the receiver and one representing the message exchange itself. However, the sender ID is vulnerable to client bugs & tampering as Bill points out, and neither the receiver ID nor the exchange ID can be used to determine if a given message is a duplicate. If you don't trust the sender, is it even possible to determine if a given message is a duplicate?
• Pablo Castro confirms that there are "practical limits" to what ADO.NET Data Services can do with respect to idempotence. Nothing in his post was surprising, though I hope it will be more explicitly called out in the final docs. Developers used to the comforting protection of a transaction may be in for a rude awakening.
• Dare Obasanjo has a great post comparing the new features in C# 3.0 to dynamic languages like IronPython. I believe many of the productivity aspects of dynamic languages have little to do with being dynamic.
• Pat Helland noodles on durability and messaging, two topics near and dear to my heart (probably from working with him for a couple of years). I'm not sure where he's going with this - his conclusion that "Basically, big, complex, and distributed system are big, complex, and distributed" isn't exactly ground-breaking. But his point that "durable" isn't a binary concept is worth more consideration. Also, his description of IMS only looking at the effects of a committed transaction is very similar to how web sites work, though obviously HTTP isn't durable so you can't make event horizon optimizations like IMS did.
• Tangentially related, Werner Vogels discusses the idea of eventually consistent distributed databases. Today, that's a problem mostly only Internet-scale sites like Amazon deal with. In the near future of continued data explosion + manycore, we'll all have to deal with it.
• Nick Malik argues that categorizing enterprise applications by lifecycle is much less useful than categorization based on organizational impact. He might also need a new chair.
• Jesus Rodriguez digs into one of SSB's new features in SQL 2008: conversation priorities.
• Arnon Rotem-Gal-Oz and Sam Gentile are mixing it up over the definition of SOA. Sam thinks SOA has to include business drivers and Arnon doesn't. I'm with Sam on this, defining "SOA" independently from "Applying SOA" seems pointless. Then again, rigorously defining SOA - much less arguing about said definition - seems like a waste of time in the first place IMHO.
• Wow, this guy Zed is mad at the Ruby community.
• Andrew Baron has 8 Reasons Why The TV Studios Will Die. Personally, I think reason #2 - Expendable Middle-Person - is the most important. If content producers can reach consumers directly, what value-add will the networks provide? (via United Hollywood)

Morning Coffee 130

• Michael Klucher announces the release of XNA Game Studio 2.0 and Major Nelson points to the press release announcing the release. You can get the bits from XNA Creators Club Online (the XNA dev center has yet to be updated).
• Speaking of XNA, David Weller points out the warm-up challenge for Dream-Build-Play 2008. I assume networking will be a big part of this years' entries, but the warm-up challenge is to "Create a new and innovative use of Artificial Intelligence in a game".
• Still speaking of XNA, Gamasutra has an interview with XNA GM Chris Satchell where he hints at a publishing channel for XNA games on the Xbox 360, with "full details" coming sometime in the new year.
• The Capitals beat the Rangers in overtime last night. Since changing coaches on Thanksgiving, they're 6-3-1. That's great, but they're still five games under .500. The good news is that even though the Caps tied for last in the league, they're only six points out of a playoff spot with about fifty games left in the season.
• My old team puts on an event every year called the Strategic Architects Forum. It's invite-only, but they've posted some of the videos, slides and transcripts from this year's event.
• J.D. Meier discusses the new Guidance Explorer release. They're now up to 3,000 "nuggets" of guidance and they've moved the guidance store to MSDN. (via Sam Gentle)
• Arnon Rotem-Gal-Oz explains further why arbitrary tier-splitting is bad. I'd also suggest reading Chapter 7 of PoEAA which provides another version of the same story: You can't take an object that's designed for fine-grained local access and make it remote without really screwing yourself up.
• Eric Lippert thinks immutable data structures are "the way of the future in C#" so he's written a series on immutability. Posts include kinds of immutability, an immutable stack, an immutable covariant stack and an immutable queue. As I've discussed, immutable data structures are HUGE in functional programming. Eric's immutable stacks and queues are similar to F#'s native list type. (via Jomo Fisher)

Morning Coffee 127

• I'm back at the office today after almost two weeks away. So "catch-up" is the official hyphenated word of the day.
• Big news last week was an update on Silverlight. The next version (the one with the embedded cross-platform CLR in it) was rebranded Silverlight 2.0 and will include higher-level UI framework features and richer networking support. Look for a go-live beta Q1 next year.
• According to Scott Guthrie, there should be a CTP of the ASP.NET 3.5 Extensions this week, including the new ASP.NET MVC framework that many folks await with baited breath. Scott posted the second in new series on this framework, this time covering URL Routing.
• After a long blog silence, Pat Helland posted four presentations that he delivered @ TechEd Barcelona. They're pretty much required reading as far as I am concerned. Here's hoping the TechEd Barcelona folks recorded at least the audio of these sessions.
  • Metropolis: Interchangeability of Operations
  • Life Beyond Distributed Transactions: an Apostate's Opinion
  • Data on the Inside versus Data on the Outside
  • The Irresistible Forces Meet the Movable Objects
• There's a new drop of F#, v1.9.3.7. Don Syme calls it "a release candidate for F# 1.9.3". Full release notes are over on his blog.
• Speaking of F#, Douglas Stockwell explains how to roll-your-own F# Express by combining the F# download above & the VS08 shell.
• Still speaking of F#, Robert Pickering has written an article on Active Patterns as well as a series on concurrency in F#. So far he's covered Async Workflows (twice), Erlang style messages and message queuing.
• Speaking of concurrency, Soma announced the CTP of ParallelFX last week. This includes Parallel LINQ (aka PLINQ) and Task Parallel Library (aka TPL). Also check out their whitepaper, team blog and dev center.
• I'm a big fan of the Architecture Journal. Now you can get all that great content in a great WPF-based app. Simon Guest has the details.

Morning Coffee 118 - ITARC SoCal Edition

I'm not back on blog sabbatical, but between finishing my presentation and attending ITARC SoCal earlier this week - not to mention being sick - I didn't have time to write anything. Normal Morning Coffee resumes tomorrow, here's a summary of my notes from on my two days at ITARC.

• Scott Ambler did the opening keynote on agile enterprise architecture strategy.
• He claims that success is more prevalent in the industry that people think, because the industry has a narrow definition of success. If you change (aka widen) the definition, the success rate goes way up! That's not exactly useful, but he referred to an as-yet-unpublished survey on project success rate that should be up on DDJ "soon". I'd like to see that raw data.
• While I agree with most of his points, Scott's presentation style is very abrasive. For example, he makes the point that there is no one-size-fits-all process, which I couldn't agree with more. But does he say it like that? No, he says "Repeatable processes? What an incredibly stupid idea!" even though the room is full of folks who probably think repeatable process is actually a good idea.
• Scott suggested that unit tests are the best way to specify requirements. I've heard this before from agile practitioners, but something nags at me about it. Certainly, having executable requirements is a huge plus. But how can you be sure they're the right requirements if the stakeholders can't read them?
• This keynote setup what turned out to be a major theme for the conference - traditional vs. non-traditional enterprise architecture. Or as I would characterize it: Industrial vs. Post Industrial architecture.
• Simon Guest presented on user experience in architecture, which is his specialty these days. He lays out a UX model that was very compelling. I'm not sure if there's a whitepaper version of this model (there should be) but you can see the model as he lays it out in powerpoint. I've seen Simon's UX decks, but never actually seen him present it, so that was a treat.
• I skipped Ted Neward's session in order to take in something new. So I went to see Daniel Brookshier of No Magic talk about DoDAF - the Dept. of Defense Architecture Framework. I had met Daniel the night before at dinner and while No Magic primarily sells UML modeling tools, we seemed to agree that UML is most useful (in my opinion "at all useful") when you imbue the vanilla models with custom semantics - aka you turn them into a DSL. So while I liked hanging out with Daniel, his DoDAF session did nothing except ensure I never work for the DoD. There's no amount of money that's worth dealing with the two dozen or so bureaucratic models that are all wholly isolated from anything that actually executes. Daniel kept saying how easy these models are to build. I'm sure they are, but that's not the problem. Since they're not an intrinsic part of a construction process, they won't stay up to date. This was a very industrial approach - Daniel even stated at one point that he was "anti-Ambler".
• David Chappell did the second keynote on grid-enabled SOA.
• When did David join Oracle? I guess I haven't been paying much attention to competitors since I moved to MSIT.
• There's an article version of this presentation available, but I haven't read it yet.
• For me, the best part of this presentation was him acknowledging that there's a need for non-stateless services, something he has blogged about recently. I'm not sure I agree with his framework for stateful interaction, but at least he's admitting that it's needed. Now if I could only convince the Connected Systems Division...
• The rest of his talk was basically a sales pitch for the Coherence product Oracle recently bought. Basically, it's a huge, multi-node, redundant, in-memory database. While I'm sure there are a few high-end problems out there - my immediate thought was travel and David mentioned SABRE is one of their customers - this is not a good general purpose solution, though David was positioning it as such.
• My talk on "Moving Beyond Industrial Software" was after the second keynote. It was good, if sparsely attended. I'm doing it again @ the p&p Summit so I'll post the slides and hopefully a recording after that.
• I skipped the last session of the day to decompress, so the next session I went to was the day two opening keynote by Fred Waskiewicz, OMG's Director of Standards. His talk, unsurprisingly, was on the value of standards - in particular, OMG's standards. This was about as anti-Ambler, anti-agile, pro-industrial a presentation as you could make. I'd heard this spiel before, so I mostly tuned out. I did challenge Fred on his point that the UML models are at a higher level of abstraction than code. They're not - they're a visualization and they're very useful, but they're at the exact same level of abstraction as code. That's why you can automatically generate the visualization in tools like Visual Studio's class designer. Fred didn't have much of a response to my question, though he did point out that some models like Business Process Models are, in fact, higher levels of abstraction.
• Next was what I thought was the best presentation of the entire show, IASA Founder Paul Preiss on what architects need to know. Note, I'm not brown-nosing Paul here - I'm the guy that first decided to commit Microsoft as an IASA sponsor, so he has to like me even if I thought his session was crap. Paul talked about architect as a career, comparing it to doctors. He worries that he's over-using that analogy, but software architect has much more in common career wise than it does with building architects IMO. I wonder where one might do their architecture residency? He also thinks of architects as "living governance", saying that project managers answer to the stakeholders while architects are beholden to the stockholders. I like that approach to governance.
• Finally, I attended Vince Casarez's session on Web 2.0 in the enterprise. Vince is an Oracle VP and this turned into a sales pitch like David Chappell's keynote did. I'm not sure what product it was, but it reminded me of QEDWiki from IBM that I saw at ETech last year, which isn't a complement. If you're going to build an enterprise mashup designer, is it just me or is "lots of code spew" a poor model. Why not go for something like Popfly or Pipes?
• I left early the second day in order to get home before my kids went to sleep (which I failed at due to lack of naptime). Overall, the conference was pretty good, though a bit sparsely attended in part I think because they held it in San Diego. The Orange Country IASA user group is very popular, so I don't understand why they didn't just hold it around there somewhere. Live and learn, I guess. They did have to postpone the DC event until next year sometime. Here's hoping I get invited to that as well as well as ITARC SoCal '08 (note, that *is* brown-nosing a bit)

The DevHawk 2007 World Tour

After spending almost all of fiscal year 07 (July '06 thru June '07) not traveling and not presenting, I'm going to be doing a few public talks to finish out the year. If you, dear reader, are going to one of these please drop me a line. Invariably, it's the side meetings and discussions that are the most valuable at these conferences.

IT Architect Regional Conference 2007
October 15th - 16th, San Diego, CA

I'm a huge fan of IASA, so I'm thrilled to be doing their west regional conference. I've presented to a packed house for the local chapter before, so I think these folks will put on a good conference. They sure have a good selection of topics and speakers.

My session is called "Moving Beyond Industrial Software". Here's the abstract:

Computers have been instrumental in ushering in the post-industrial age. Yet, most enterprises today are run with an industrial mindset and the IT department is organized like a factory. This creates a tension between the forces of industrialization that define the organization and the forces of post-industrialization that define today’s marketplace. For example, our post-industrial world is becoming more decentralized by the day. Yet many organizations believe the key to a successful service oriented architecture – a very decentralized system design – is to have a central service repository.

In this session, Harry Pierson will examine this tension, get you thinking outside the industrial mindset and help you think about software development in a post-industrial way.

I'm very excited about this talk.

MS SOA & Business Process Conference
October 29th - November 2nd, Redmond, WA

I'm not presenting at MSSOABPC (that's a mouthful) but looks like most of my team is going. So if you're going and want to hang out with the guys who are doing this stuff in the trenches @ MSIT, let me know. Also, I put out the call for anyone interested in a geek dinner. From the agenda, looks like they're keeping us busy until 8pm every night Mon-Wed, so we can either a) have geek dinner Thursday or Friday or b) have geek beers after one of the receptions in the early part of the week.

patterns & practices Summit USA West
November 5th - 9th, Redmond, WA

I did the p&p Summit back in 2005, a very successful debut of my Developer 2.0 talk. (I'm doing that talk at a different conference this year, details below.) This year, I'm not 100% sure what I'm going to talk about yet. I'm currently slated to talk about the Rome project that I'm doing in MSIT, but given our current slow progress on that project, I'm probably going to talk about something else. I'm thinking either the "Moving Beyond Industrial Software" talk described above or the "Facing the Fallacies of Distributed Computing" talk described below. Any other suggestions?

DevTeach Vancouver 2007
November 26th - 30th, Vancouver, BC

This is a brand new experience for me. Frankly, I'd never heard of DevTeach before my friend Mario Cardnial suggested I submit a couple of sessions. Since it's only a few hours drive away, I'm bringing the family along. We'll see how that goes. And when I'm not doing my sessions or hanging out with the family, I might take in a session or two in the XNA track.

Here are the sessions I'm doing:

Developer 2.0
Finding Your Way in the Future of Software Development
The one constant in software development is change. Software development in 2007 is dramatically different than it was in 2000, which was in turn dramatically different than in 1993. You can be guaranteed that the platforms, languages, and tools will continue to evolve. Learn how Harry Pierson, Architect in Microsoft IT, believes software development is going to evolve in the next five years and what you must do today to remain competitive.

Facing the Fallacies of Distributed Computing
Sun Fellow Peter Deutsch is credited with authoring "The Eight Fallacies of Distributed Computing". These are near-universal assumptions about distributed systems that “All prove to be false in the long run and all cause big trouble and painful learning experiences.” In this session, we will examine these fallacies in depth and learn how to avoid them on the Windows platform by leveraging Web Services, WCF and SQL Service Broker.

The Integration Business Case, continued

Nick responds to my visceral thoughts on the integration business case. There's no point in excerpting it - go read the whole thing. I'll wait.

It looks like for case #2, he added the ability to "change readily and inexpensively", which is to say he made it overlap even further with #4 than it used to. He also changed #3 to make it clear that he was collecting metrics to give us "awareness of process efficiency". That makes #3 overlap with #4 on efficiency instead of #1 on BI, but either way it's still redundant.

So we're still left with the business cases of Business Intelligence, Efficiency and Agility. Nick conflates Efficiency and Agility both in his original post and his follow-up, but I think it makes sense to separate them. I still stand by my original point that the business is only interested in directly funding Business Intelligence.

Nick is willing to bet a nice lunch that MSFT has invested more in improving operational efficiency that we have on BI in the past four years. He's probably right, but he missed the point I was making. The business will readily invest in improving a specific process they can measure the ROI on improving. MSFT has lots of processes, I'm sure most of them have significant room for improvement.

But Nick's list isn't about specific improvements. He's explicitly wrote that he's describing a scenario where "our systems are all optimally integrated". Selling the business on generally improving efficiency is very different that selling the business on improving the efficiency of a specific process. I'd bet the same nice lunch that the vast majority - if not all - of integration infrastructure running at MSFT was originally deployed as part of a specific business scenario that needed to be solved.

My point here is most businesses are better at funding projects to meet specific business needs than it is at funding pure infrastructure projects.

As for agility. Martin Fowler pointed out once that adding flexibility means adding complexity. But chances are, you'll be wrong about the flexibility you think you'll need. So you actually end up with the additional complexity but none of the flexibility benefit. Martin recommends "since most of the time we get it wrong, just don't put the flexibility in there". Instead, you should strive for simplicity, since simpler systems are easier to understand and thus easier to change.

Does the same philosophy apply to process? I think so, though there is one thing I'd be willing to risk being wrong on. We all expect the steps in a process to change over time, so moving to a declarative model for process definition sounds like a good idea. Luckily, there's existing platform infrastructure that helps you out here. But beyond that, I can't think of a flexibility requirement that I'm so sure of that I'm willing to take on the additional complexity.

Again, I'm not saying efficiency or agility (or integration for that matter) are bad things. I'm saying they're a tough sell to the business in the absence of specific scenarios. Selling the business on automating the ordering processing is feasible. Selling the business on building out integration infrastructure because some future project will leverage it is much tougher. If you can sell them on it, either because the company is particularly forward thinking or because you can sell ice to Eskimos, then more power to you. But for the rest of us, better to focus on specific scenarios that the business will value and keep the integration details under wraps.

Morning Coffee 115

• Scott Guthrie has two new posts in his series on LINQ to SQL. The first covers updating the database using stored procs instead of dynamic SQL. I was somewhat surprised that there wasn't the capability to auto-generate vanilla Insert, Update and Deleted procs, but I guess DBA's probably hate that anyway. The second shows how to use ExecuteQuery to execute arbitrary SQL instead of using the cool LINQ query syntax. I'm doing a bunch of loosely-typed SQL work right now, so I'm going to take a deeper look at this.
• Speaking of LINQ, I just discovered this great series on IQueriable by Bart De Smet. It's four months old, but takes an incredibly detailed look at what happens under the hood with LINQ. Bart also has a reference implementation of LINQ's standard query operators as well as LINQ to Sharepoint.
• Dan Maharry has pulled together what looks like the definitive guide for really slimming down and speeding up your VPC. It's XP specific, but I'd bet most of the guidance would also apply to WS03, which is what I mostly use in my VPCs. (via Larkware)
• Jimmy Nilsson thinks it's the operations department that holds the power in today's IT world. I agree 100% That's why I value Dale's input so much.
• Nick Malik wonders if it's time to translate the Federal Enterprise Architecture for use in the commercial sector. My dad just retired from 5 years in the FAA and he thinks FEA is too high level to be particularly useful.
• The 2007 edition version of Scott Hanselman's ultimate tool list is now available.
• A bunch of XNA Gamefest sessions are now available for on-demand viewing.

Morning Coffee 114 - MoMAAB Edition

• We spent all day yesterday discussing four topics: SaaS, Tools for Scrum, Web 2.0 and Domain Specific Languages. Even though it was just a day, my brain is full. These were deep and challenging discussion. I need to let the discussions stew a bit before posting anything about them here. But I will.
• Next time we do one of these, I'm bringing a video camera. I took notes, but looking over them the next morning they seem woefully incomplete. OneNote's integrated audio/video recording capabilities would nicely augment my notes.
• We ran this meeting using Open Space, and it worked very well. Of course, we only had 8 people, so we didn't need a lot of process to self organize. However, it did whet my appetite for having a larger Open Space style un-conference for architects. Is that something other folks might be interested in?
• Major thanks to the folks at Clarity Consulting who graciously gave us space to meet and fed us yesterday. Their CTO Jon Rauschenberger sat in on most of our meeting, and drove our Web 2.0 discussion. I said I wanted to stew a bit on the discussions, but Jon's slides are available on line if you're interested.
• Scott Colestock showed me Diigo, a social annotation tool. Where del.icio.us lets you tag and annotate individual pages, Diigo lets you annotate and highlight specific parts of the page. They also have blogging tools, where these annotations and highlights become blog posts, but they don't support dasBlog. However, since FeedBurner doesn't support Diigo for link splicing, I'm afraid my use of it will be limited.
• Jim Wilt introduced me to Virtual PC's command line. He recommends using "-pc <vpc name> -launch -singlepc" which launches a single virtual environment without the VPC console. I rarely run more than one VPC at a time and I hate stuff cluttering up my taskbar and notification area, so I like this a lot.
• Loren Goodman demonstrated the SharePoint Explorer Client. SharePoint & MOSS came up several times in all of our topics, so this is going to get a second look. I always thought it was strange that MSFT ships a smart client for editing WSS & MOSS, but not viewing it. SP Explorer looks like it fills that gap nicely.
• Shannon Braun sent us all a link to the 50/70 rule, which seems like a good rule of thumb. Of course, assuming that things won't progress linearly is almost always a good rule of thumb. But the 50/70 rule has reasoning behind the assumption.
• Chicago is nice, but the weather has been a little freaky. It's either been hot & humid, downporing thunderstorms or tornados. Keith Powell showed me FlightAware, which shows you flight departure and arrival history. My flight hasn't left within an hour of scheduled departure in a week. I'm going to try and grab an earlier flight, but I have a feeling it's going to be a long trip home.

Morning Coffee 113

• I'm in Chicago today and tomorrow for a reunion of sorts. In my last job, I managed a group of external architects called the Microsoft Architecture Advisory Board (aka the MAAB). We discontinued the program a while back, but the core of the group found the program valuable enough they have continued to meet anyway. I found the MAAB meetings incredibly valuable and insightful, so I'm really excited to be invited to continue my involvement with the group.
• I picked up Bioshock Tuesday (Circuit City had it on sale) on my way to my bi-weekly campus excursion. My meetings were over around 2pm so I headed home early, expecting to surprise the kids. But Jules had decided to skip naps and go shopping with them. Her cell phone was dead, so I ended up at home with a couple of hours to myself and a brand new copy of Bioshock. Wow, is that a good game. Certainly deserving of the amazingly good reviews it's garnered.
• Speaking of reviews, this transparently biased review of Bioshock over at Sony Defense Farce Force is frakking hilarious. Somehow, I doubt their dubious review will stem the tidal wave of Bioshock's well-deserved hype. Can't wait to read their Halo 3 review.
• Pat Helland writes at length on master-master replication. I reformated it into PDF so I could read it - the large images were messing up the text flow on my system. As usual for Pat, there's gold in that thar post. His thoughts on DAGs of versions and vector clocks as identifiers are very exciting. However, I think he glosses over the importance of declarative merging. I would think programmatic merge would likely be non-deterministic across nodes. If so, wouldn't you end up with two documents with the same vector-clock identifier by different data?
• Joe McKendrick points to a few people who predict the term "service-oriented" will eventually be subsumed under the general heading of "architecture". Not to brag, but I made that exact same prediction almost three years ago.
• Erik Johnson thinks that SOA 2.0 centers on transformational patterns. The idea (I think) is that if systems "understand each other more deeply", then we can build a "smarter stack" and design apps via new constructs to promote agility and simplicity. Personally, I'm skeptical that we can define unambiguously system semantics except in the simplest scenarios, but Erik talks about using "graph transformation mathematics" to encode semantics. I don't know anything about graph transformation mathematics, but at least Erik has progressed beyond hand waving to describing the "what". Here's looking forward to the "how".
• New dad Clemens Vasters somehow finds time to implement an XML-RPC binding for WCF 3.5. I was encouraged that it didn't require any custom attributes or extensions at the programmer level. Of course, XML-RPC fits semantically into WCF's interface based service model, so it shouldn't be a huge surprise that it didn't require any custom extensions. But did it need WCF 3.5? Would this work if recompiled against the 3.0 assemblies?
• Phil Haack writes a long post on Duck Typing. VB9 originally supported duck typing - the feature was called Dynamic Interfaces - when it was first announced, but it was subsequently cut. I was really looking forward to that feature. Between it and XML Literals, VB9 was really stepping out of C#'s shadow. I guess it still is, even without dynamic interfaces.
• Since I've been doing some LINQ to XML work lately, I decided to go back and re-write my code in VB9 using XML literals. While XML literals are nice, I don't think they're a must have. First, LINQ to XML has a nice fluent interface, so the literals don't give you that much cleaner code (though you do avoid writing XElement and XAttribute over and over.) Second, I find VB9's template syntax (like ASP <%= expression %>) clunky to work with, especially in nested templates. Finally, I like the namespace support of XNames better. As far as I can tell, VB9 defines namespaces with xmlns attributes just like XML does. So I'm not dying for literal XML support in a future version of C#. How about you?

Morning Coffee 112

• The Lee Holmes over at the Powershell Team Blog writes about alternatives to the "decades-old" Windows console host. Powershell Plus looks awesome. PoshConsole also looks pretty cool (though far from finished yet) and is free.
• WL ID Web Authentication SDK has been released. Details on the WL ID team blog. It looks like what Passport SDK provided for quite some time, but now it's free. There's also a client auth SDK in development. (via Dare Obasanjo)
• Libor Soucek leaps to the wrong conclusion about not differentiating enterprise & support systems. Of course, different systems will have different availability requirements. But what happens when we connect them together? We can't let the support system effect the availability of the enterprise system, right? To me, that implies either a) the support system now needs to conform to enterprise system availability requirements or b) we need some other mechanism (like async durable messaging) to act as a buffer between them. Personally, I like "b".
• Nick Carr points to an article The Trouble with Enterprise Software by Cynthia Rettig. Cynthia writes that while the massive complexity of enterprise software, especially large-scale ERP systems like SAP, significantly hinder it's value. It's a must read. Choice quotes:
• "It is estimated that for every 25% increase in complexity in the tasks to be automated, the complexity of the software solution itself rises by 100%."
• "The notion of reusable software works on a small scale. Programmers have successfully built and reused subroutines of standard functions. But as software grows more complex, reusability becomes a difficult or impossible task."
• "Hope, unfortunately, has never been a very effective strategy."
• "Is enterprise software just too complex to deliver on its promises? After all, enterprise systems were supposed to streamline and simplify business processes. Instead, they have brought high risks, uncertainty and a deeply worrying level of complexity. Rather than agility they have produced rigidity and unexpected barriers to change, a veritable glut of information containing myriad hidden errors, and a cloud of questions regarding their overall benefits."

Morning Coffee 111

• I'm not sure if I should laugh or cry at Nick Malik's definition of politecture. I mean, it's funny so I'm laughing, but it's so true that it makes me want to cry.
• Don Box comments on retiring the tenets. It's good to see him say "please God tell me we can do better" than CLR interfaces or WSDL.
• Looks like the P2P APIs are finally getting the managed treatment in .NET FX 3.5. A long time ago, John deVadoss asked me what an enterprise system like CRM might look like if it used a peer-to-peer approach instead of client-server. If I had any free time, I'd prototype one out on this API. (via Mike Taulty)
• Scott Guthrie goes back to his LINQ to SQL series to tackle Stored Procs and UDFs. Being able to use UDFs inline with LINQ queries is very cool. However, it seems to me that LINQ discourages the use of stored procs. As a developer, I'd rather write LINQ queries than stored procs, if I can. The probably puts me at odds with DBAs who'd rather all DB access be via stored procs they control.
• Soma writes about new MSBuild enhancements in VS08: multi-targeting and parallel build.
• I just discovered Vista Battery Saver. Basically, it turns off Aero and Sidebar when you're on battery. I'm traveling to Chicago next week, so we'll see if it has much impact on my battery life. (via Plenty of Code and Larkware)

The Durable Messaging Debate Continues

Last week, Nick Malik responded to Libor Soucek's advice to avoid durable messaging. Nick points out that while both durable and non-durable messaging requires some type of compensation logic (nothing is 100% foolproof because fools are so ingenious), the durable messaging compensation logic is significantly simpler.

This led to a very long conversation over on Libor's blog. Libor started by clarifying his original point, and then the two of them went back and forth chatting in the comments. It's been very respectful, Libor calls both Nick and I "clever and influential" though he also thinks we're wrong on this durable messaging thing. In my private emails with Libor, he's been equally respectful and his opinion is very well thought out, though obviously I think he's the one who's wrong. :)

I'm not sure how much is clear from Libor's public posts, but it looks like most of his recent experience comes from building trading exchanges. According to his about page, he's been building electronic trading systems since 2002. While I have very little experience in that domain, I can see very clearly how the highly redundant, reliable multicast approach that he describes would be a very good if not the best solution.

But there is no system inside Microsoft IT that looks even remotely like a trading exchange. Furthermore, I don't think approaches for building a trading exchange generalize well. So that means Nick and I have very different priorities than Libor, something that seems to materialize as a significant amount of talking past each other. As much as I respect Libor, I can't shake the feeling that he doesn't "get" my priorities and I wouldn't be at all surprised if he felt the same way about me.

The biggest problem with his highly redundant approach is the sheer cost when you consider the large number of systems involved. According to Nick, MSIT has "over 2700 applications in 82 solution domains". When you consider the cost for taking a highly redundant approach across that many applications, the cost gets out of control very quickly. Nick estimates that the support staff cost alone for tripling our hardware infrastructure to make it highly redundant would be around half a billion dollars a year. And that doesn't include hardware acquisition costs, electricity costs, real-estate costs (gotta put all those servers somewhere) or any other costs. The impact to Microsoft's bottom line would be enormous, for what Nick calls "negligible or invisible" benefit.

There's no question that high availability costs big money. I just asked Dale about it, and he said that in his opinion going above 99.9% availability increases costs "nearly exponentially". He estimates just going from 99% to 99.9% doubles the cost. 99% availability is almost 15 minutes of downtime per day (on average). 99.9% is about 90 seconds downtime per day (again, on average). 

How much is that 13 extra minutes of uptime per day worth? I would say "depends on the application". How many of the 2700 applications Nick mentions need even 99% availability? Certainly some do, but I would guess that less than 10% of those systems need better than 99% availability. What pretty much all of them actually need is high reliability, which is to say they need to work even in the face of "hostile or unexpected circumstances" (like system failures and downtime).

High availability implies high reliability. However, the reverse is not true. You can build systems to gracefully handle failures without the cost overhead of highly redundant infrastructure intended to avoid failures. Personally, I think the best way to build such highly reliable yet not highly available systems is to use durable messaging, though I'm sure there are other ways.

This is probably the biggest difference between Libor and me. I am actively looking to trade away availability (not reliability) in return for lowering the cost of building and running a system. To someone who builds electronic trading systems like Libor, that probably sounds completely wrongheaded. But an electronic trading system would fall into the minority of systems that need high availability (ultra high five nines of availability in this case). For the systems that actually do need high availability, you have to invest in redundancy to get it. But for the rest of the systems, there's a less costly way to get the reliability you need: Durable Messaging.

Morning Coffee 102

Seems like a slow week.

• Jules and I went to see the latest Harry Potter movie this past weekend. It's easily the weakest of the six HP stories so far. The first two stories were about discovering this magical world, the next two about discovering Harry's past, and the last two about confronting said past. That leaves OotP as the odd-story-out, mostly bridging from the end of the fourth story to the start of the sixth.
• Speaking of movies, the new movie feature of Mobile Search v2 rocks, though I have two quick suggestions. First, it would be nice to have a time-sorted view of when a given movie is playing. So if it's playing at 4pm at one theater and 4:30pm at another, you'd see them in a list ordered that way. Second, how about an option to buy tickets directly from the phone?
• If you're interested in WPF and 3D, Eric Sink has a series for you.
• Old news, but Windows Home Server RTMed on Monday. I'm really looking forward to this product.
• I was looking for some information on how WCF pumps messages in the service host and I found this post from Maheshwar Jayaraman. Between that post and Reflector, I think I've got a good handle on how ChannelDispatcher works.
• Larry O'Brein calls out three MS Research Projects. Microsoft Research Accelerator is a high-level data-parallel library that targets GPUs. Graph Layout Execution Engine (aka GLEE) is a library for graph layout and viewing. VirtualEarth MapCruncher converts existing maps (PDF and bitmaps) to work with Virtual Earth.
• Ted Neward weighs in on the David Chappell's Korean War REST vs. WS-* analogy. Skim the history lesson, but make sure you read his points about security and reliability interop. WS-* has addressed these areas, so if you need those capabilities, why wouldn't you use WS-* to get them rather than re-invent the wheel? As for the history lesson, Ted does say he thinks software development is more analogous to making war than building a house. He expands on that idea and recommends Robert Greene's The 33 Strategies of War. I want to read the book and mull it over a bit, but I certainly see where Ted's coming from.

Morning Coffee 99

• Mladen Prajdic has a great post on handling a database in your unit tests. He mentions NDbUnit but seems mostly to favor SQL 2005's database snapshot feature. He's got sample code for creating and restoring a snapshot. (via DNK)
• Microsoft Robotics Studio 1.5 released yesterday. Tandy Trower - GM of the Robotics group - has the details on what's new.
• Herb Sutter has a new column in Dr. Dobbs on concurrency. First up, "building a consistent mental model for reasoning about concurrency". Sounds like a must read column. (via LtU)
• Scott Hanselman describes "Sez You Architecture". I wonder, do architecture ninjas get to wear a Shinobi shozoku?
• From the Not Everyone Agrees With DevHawk Dept.: Libor Soucek disagrees with me and thinks that durable messaging should be avoided. I had a hard time following Libor's logic but needless to say, I disagree with his disagreement. He writes that one of the reasons to use DM is for "Cooperating on transaction with external system". While multiple systems may be cooperating on a business transaction, in no way do I believe they are going to cooperate on a database transaction. But since he started talking about the DTC, I suspect we're talking past each other. Libor, drop me a line and we can discuss further.

Morning Coffee 92

• Brad Wilson blogs about SvnBridge, a tool that lets you use Subversion clients like TortoiseSVN to talk to Team Foundation Server. While I think that's cool, I wonder is anyone interested in subversion clients other than TortoiseSVN? For example, will people choose AnkhSVN instead of the Team Explorer Client?
• Speaking of TortoiseSVN, I wonder if those guys are interested in building a TortoiseTFS project? I did find two other TFS shell extensions projects: Dubbelbock TFS and Turtle, though neither appears as full featured as Tortoise.
• Scott Guthrie details VS08's multi-targeting support. Of course, the three versions of the .NET Framework VS08 can target all use the same underlying runtime, which probably made it easier to build.
• Michael Platt refactors Don Box's original tenets of service orientation so he can include some information about how these services get built.
• Scott Hanselman tackles the tricky question of assembly granularity.
• PowerShell Analyzer is now available for purchase. Among other things your $59 gets you, besides a 50% savings, is "Feature request priority". That's pretty cool. I wonder how many other micro-ISV's take the approach of "pay me now and you get to help me pick some of the new features."
• Brandon LeBlanc writes about dual monitor support in Vista. I'm loving the dual monitor support, though I have a somewhat strange setup. I keep my primary monitor rotated in portrait mode, which is great for reading and writing. I typically use my second monitor for blogs and mail. I even wrote a custom multi-mon wallpaper utility so I could easily generate new wallpapers for my non-standard monitor layout, including bitmap rotate support. If there's interest, I can post it. (via Sam Gentile)
• Nick Malik continues to write about Mort, with the usual response from the usual folks. I liked his point that "You cannot fight economics with education", but otherwise I'm staying out of this discussion.
• In the same vein, Martin Fowler writes about Technical Debt. I completely agree with his hypothesis that short changing design may save time in the short term but will cost much more in the long term. However, the problem is that the people who are making the tradeoff - i.e. the people paying for the project NOT the people building the project - either don't understand the tradeoff or are more than happy to sacrifice the long term cost for the short term gain. How are most projects measured? Being on time and on budget with the planned set of features. Very few projects - and none that I've ever seen - are goaled on long term maintainability. Until you can change that, this issue will continue to linger.

Morning Doughnuts 11

Harry will be back on Monday so I will returning to blogging on my website, while I will let the expert return to his normal posts here (not that he really took a break). I agree with Harry's post in that I really want to get something built so that we can talk about more than theoretical models. Like last time I appreciate the opportunity to sub for the master this last week. I hope that you found some of my entries interesting.

• Sam Gentile wrote the other day why it's great to be a Microsoft developer. I enjoyed that post because I just celebrated the end of my first year here at Microsoft. At this point I am not sure what I have contributed, but I have learned a great deal and want to apply that knowledge over the next year to help the company to succeed. We really do have great people and great technologies.
• The Seattle/Oklahoma City Sonics hired a GM who is only 30 years old. You know you must be getting old when the people running the sports teams are younger than you. He comes from the Spurs organization though so at least he has a background from a successful franchise.
• Ben Pearce listed out his top 5 questions about PowerShell this week at TechEd. He also recommends the book "PowerShell in Action" by Bruce Payette. I heartedly agree with this endorsement as the book is excellent.
• It looks like there are going to be more family friendly games for the XBOX360. I for one am glad to hear that. The other day as I was trying to find some games my 4 year old with the broken leg could play I realized how many games I have that wouldn't be appropriate for him. This is very good news in my opinion.

Morning Doughnuts 10

• I am a big fan of PowerShell, and I know Harry likes it as well. Of course I have aliased many of the commands so they appear more Unix like. I mention this because David Aiken mentions a new product for PowerShell called NetCmdlets produced by N Software. I downloaded a trial and have been impressed so far. If you use PowerShell it might be worth giving this a look.
• The New Yorker has an interesting article about feature choices in technology. Basically it comes down to customers choose products with more features and customization options if given the choice, but when they actually have to use the products they prefer simplicity. I think that is something lost on those of us who design things. Giving users every possible option can make our products seem more difficult to use over simpler less feature rich choices. (via Coding Horror)
• Well it took longer than some may have guessed, but Microsoft has been sued over the name Vista by a French TV company. Apparently they were going to launch a television station with the same name. I wonder what the courts will decide since apparently the TV company didn't register the name in the software category.

Making My Mark at TechEd

One of the things that's different about being in MSIT is that it's cut my travel dramatically. Basically, the only travel I've done since taking this job was Thomas Erl's SOA Workshop last September. So it came as a bit of a surprise when I got tapped to present at TechEd (at fairly close to the last minute).

The last year I ran the architecture track, all track owners were asked to include either a globalization or MSIT session in our track. Since then, MSIT has expanded it's role at TechEd dramatically, with 14 breakouts, 20 chalk talks and our own Technical Learning Center.

I'm doing a two chalk talks on my MSIT project, Rome. I mentioned the project back when I switched jobs, but we've never talked about the project by name before. We haven't accomplished quite as much as I'd hoped since then, but we've progressed to the point that we can talk publicly about what we're doing. Now that we've begun to open the kimono a bit, you should see more on Rome, Not only from me but also my teammates who blog: Dale, Rick and Dottie.

So if you're going to TechEd, make sure you stop by the MSIT Technical Learning Center and say hi. Unlike my last two trips to TechEd, I have very limited responsibilities this time - basically just show up on time and talk about what I do all day, twice - so that leaves plenty of time to attend sessions, chat up attendees and ride roller coasters. Hope to see you there.

ROME: Service Oriented Infrastructure for the Enterprise
Like most enterprises, Microsoft IT is adopting a service oriented approach for the development of their internal systems. However, in order to avoid projects building similar service infrastructures on a per application basis, MSIT realizes the need for a common service oriented infrastructure to build these systems upon. Inside MSIT, the Rome project is tasked with designing, building and maintaining the infrastructure for these services. Come chat with Harry Pierson, lead architect of the Rome project, and discuss the challenges of building a large-scale service infrastructure inside a large enterprise like Microsoft.

• MST02-TLC Monday (June 4th) 1:15-2:30pm
• MST14-TLC Thursday (June 7th) 1-2:15pm

Morning Coffee 64

• I took my son to the Pacific Science Center this past weekend to see the Grossology exibit. His favorite was the barfing machine, but we also got to play Urine: The Videogame, stand inside a giant nose, work the burping machine, climb the scab wall and slide down the throat into the stomach and come out thru the colon. We also checked out the dinosaur exibit (Patrick's favorite part: petrified dino poop), Kids Works and the Peter and the Wolf laser show. It was awesome.
• This week's Big News^tm is the rebranding of WPF/E as Silverlight. Tim Sneath has the rundown, including the news that more news about Silverlight is coming at MIX. Personally, I think Silverlight is a great name. I was worried it was going to be another W*F name. (I'm waiting for the day that MSFT marketing tries to rebrand Win32 as the Windows Technical Foundation).
• Gianpaolo Carraro writes about what happens when a SaaS company bites the dust - i.e. "what happens to my data?" I expect that this is one of the aspects of SaaS that you have to weigh, though I doubt most companies will explicitly think about what happens if their SaaS provider goes belly up. As the SaaS market expands and more companies will go belly up (I agree w/ GP 100% that this isn't SaaS specific, rather a natural force of any market) how much will that drag on SaaS adoption? I'm thinking it'll be a fairly significant drag, but the SaaS market will eventually rebound.
• Nick Malik picks up the decentralization meme is started last Friday and compares enterprise architecture to city zoning boards. In general, I agree with Nick's "not in a vacuum and not with a heavy hand" comment, but even more with his point that "we haven't figured all it out yet." Most EA efforts I've seen have been heavy handed and fairly divorced from reality (aka in a vacuum). More on this topic in the future.
• Kirk Evans closes the loop on city planning with a reference to Pat Helland's Metropolis work. Pat's work has been a huge influence on me. I often repeat Pat's point about cities being "interconnected yet independent". Services need to be interconnected yet independent also.
• Roger Wolter has a new Master Data Management whitepaper out, this one MDM hubs. I was literally talking about MDM on a conf call this morning, so Roger's timing is impeccable.
• I haven't read RADAR Architecture yet, but the fact that DAR in the acronym stands for "Dumb-Ass Recipient" made me laugh. (via Sam Gentile)
• MIT Media Lab has cointed the latest 2.0-ism: Human 2.0. I love Nick Carr's take on it: "We're definitely overdue for an upgrade - it seems like we've been stuck in Version 1.x for a few hundred thousand years, and that was after a beta that went on for freaking ever. Still, I think I'll probably hold off until 2.01 or 2.02. I don't want to be on the bleeding edge for this one."

Morning Coffee 61

• Nick Malik wonders if architecture is code or if it's data? Frankly, I have nothing to add to this, but thought I should link to something Nick wrote since he's letting me share his office for the next few months while I'm engaged with one of the teams he mentions, though he begged me not to disclose which one. :)
• Ted Neward's Five Minute Management Lessons for Developers made me snicker.
• Xbox.com is running a new contest called "My Mom's a Gamer". Mine is. These days, it's mostly casual games on MSN Games, but back in the day she played both the Atari 2600 and Colecovision. She would play Space Invaders for hours. And curse. A lot. Most kids learned to curse on the playground, I learned from my mother.
• Mark Cuban claims the HDTV is the new PC. TV and PC technologies are certainly evolving as they merge, but will that platform be as open as the desktop PC or the browser? It better be.
• According to Nick Carr, Citigroup is looking to cut $4.6 billion in spending over the next three years and that IT will be one of the "cornerstones" (i.e. hardest hit) of that effort. I had a chat with an Meta analyst in Australia a few years ago who suggested that IT spending was going to go thru an innovator's dilemma phase. Huge companies (like Citi) with huge IT budgets are facing significant competition from small companies that can't afford huge IT budgets. These smaller companies get used to running a tighter ship and tend to be more competitive as they grow and are able to directly face off against the big fish.

Morning Coffee 52

• I finally found a use for the free SOA book I got from attending that Thomas Erl workshop. I'm using it to prop up one end of my daughter's mattress while she's sick so she can sleep better.
• Jeff Tash states axiomatically that CASE has evolved into Enterprise Architecture. I agree with his points about why software construction isn't like manufacturing, but he seems to be describing BDUF rather than EA. I'm anti-BDUF too, but why blame EA? (via John deVadoss)
• Joe McKendrick comments on my SaaS/SOA post and wonders if SOA should stand for "SOA Oriented Architecture". He also writes that most organizations these days don't have an SOA, they have an AOS, "Agglomeration Of Services". So true, so true.
• JD Meier talks up the new VSTS guidance available on CodePlex. Looks like some good stuff in there. I like how the p&p guys are moving from documents to wikis to deliver their guidance.
• I've held off on getting the HD-DVD drive for my Xbox, but I think I'm going to cave soon, where soon == about two months. That's when The Matrix Trilogy is released on HD-DVD. Right around my birthday too, how convienent.

When is a Service Not a Service?

Conceptually, I like both Service Oriented Architecture (aka SOA) and Software as a Service (aka SaaS). However, I think we've done the industry a disservice by overloading the term "service".

John deVadoss likes the following definition of SaaS from Wikipedia. So do I.

Software as a service (SaaS) is a model of software delivery where the software company provides maintenance, daily technical operation, and support for the software provided to their client. SaaS is a model of software delivery rather than a market segment; it assumes the software is delivered over the Internet. Software can be delivered using this method to any market segment including home consumers, small business, medium and large business.

To paraphrase, SaaS is software that traditionally you might have bought, installed and run yourself but instead now can access over the network where someone else is responsible for installing and running it. For example, instead of buying, setting up and managing my own mail server to handle a single @devhawk.net email address, I use the WL Custom Domains service.

SOA on the other hand isn't a model of software delivery, it's a model of software segmentation. Again, here's the Wikipedia definition, this time for SOA:

There is no widely-agreed upon definition of Service-oriented architecture other than its literal translation that it is an architecture that relies on service-orientation as its fundamental design principle.

Err, that's not very helpful. Let's check out the OASIS definition (cribbed from Wikipedia).

[SOA is] A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

Well, at least it's not a self-referential recursive definition. But it is littered with committee-speak. (Who talks like that in real life?) Frak it, here's my definition:

SOA is a way of implementing IT systems as a web of interconnected yet independent loosely coupled subsystems (typically called services) instead of as big honking systems we have traditionally built that tend to be unwieldy, in-agile, difficult to change and probably obsolete by the time they were deployed.

We could argue about the language, but you get the point. There would be a ton of argument about the size of the subsystems (i.e. the service granularity), but I think most people can agree that SOA encourages building multiple smaller interconnected subsystems instead of one big (honking) system.

Which brings me back to my original point: Service, in the SOA sense, describes the approach to factoring parts of an software solution. Service, in the SaaS sense, describes a software delivery mechanism. Certainly, you can use both together and take an SOA approach to building a SaaS product. But you don't have to. So having the same term "service" used in both is very confusing.

How many SaaS products use SOA today? I would guess "not many" since there hasn't been much demand for it. When you're selling to the long tail of the LOB market, support for service-oriented integration isn't a critical selling feature. As SaaS becomes more attractive to larger companies (i.e. ones with dedicated IT staffs), using a SOA approach will be more important to SaaS product vendors. So they will converge in a way, but not in the way their naming suggests.

Of the two uses, SaaS seems closer to the dictionary definition of service. Maybe the S in SOA should stand for "Subsystem"? Nah, I like the term "connected systems" better than "service oriented" anyway.

Morning Coffee 49

• The eBay Architecture SD Forum presentation that spawned the whole Transactionless meme is available here. As I reported yesterday, it doesn't call for going completely transactionless as Martin suggested. It calls for going without distributed transactions, which I agree with 100%.
• More interesting than the transactional aspects, I found the data tier functional segmentation information facinating. Too bad those guys aren't using our platform, SSB was expressly designed for exactly this sort of segmentation. I also liked that step 1 for "massively scaling J2EE" is to "throw out most of J2EE".
• After going mostly dark since last august, the manager of my old team John deVadoss has been blogging up a storm since the beginning of March. So has my old boss Mike Platt. I wonder what happened at the begining of March? Here's hoping this blogging fever spreads on my old team.
• Joe McKendrick: "The bottom line is that ROI on SOA is an enterprise challenge, not an IT challenge." Truer words are rarely spoken.
• The rumor mill on the Black Xbox 360 "Elite" are coming fast and furious. I don't care about the HDMI port (my HDTV is five years old and doesn't have one) but I would like a bigger hard drive...

Morning Coffee 43

• This week is the MVP summit. Hopefully, I'll make it over there and see the Architect MVPs. Otherwise, things seem sort of quiet in the Microsoft wing of the blogoshphere.
• Saw 300 yesterday on a relatively rare day out with just Jules. Really enjoyed it.
• Nick Malik describes what goes into an enterprise platform roadmap.
• Joe McKendrick recaps a vendor SOA suites podcast. With SOA, you can revisit your platform decision on a project by project basis which allows you to avoid the dreaded "vendor lock-in". But supporting lots of platforms is an operational nightmare, so maybe lock-in isn't as bad as it sounds.

Morning Doughnuts 6

• The Wall Street Journal is reporting that Massachusetts lawmakers are considering a bill to punish retailers for leaks in personal data. I wonder how long it will take for a law like that to go nationwide? Looks like there may be some good jobs in retail IT data security opening up shortly.
• There is an interesting debate on the SAAS architecture in Dr. Dobb's Portal. The money quote for me was as follows:

"Ajax and Web 2.0 are great technologies for casual use, but for mission critical you need the capabilities of a desktop app," RightNow CEO Greg Gianforte says.

I have to admit I don't agree with that quote at all. It seems pretty shortsighted in minimizing the capabilities of web based applications.

• As a follow-up to yesterdays entry about the 12 steps to overcome email addiction here is a 12 step program to help you overcome being a SOAholic. There are also some symptoms you can look for to see if you are a SOAholic.
• Ram Ravishankar posts on if SOA requires web services. He makes pretty good arguments for an against a SOA requiring web services and ultimately doesn't answer the question. I would say that a SOA doesn't require web services, but it is very likely in the range of 90% plus that a SOA within a company is going to have at least some web services in it.
• Harry returns from his secret mission and will be back blogging on Monday. I have really enjoyed stepping in being a replacement blogger this week. While my take on technology is a bit different that Harry's I hope that my entries were interesting and offered a bit of a different perspective on IT.

Morning Doughnuts 3

• What does it take to be an architect? Skyscrapr.net attempts to answer this question by asking a bunch of architects.
• I have started teaching my children about astronomy. I found an open source product called Stellarium that is excellent for learning about the celestial objects visible in your area.
• A Methodology for SOA adoption? I read an interesting blog on this subject from a couple of weeks ago. It's not a long article, but the author makes some interesting points including an outline for SOA adoption.
• I finally picked up Gears of War on Friday. It really isn't a game I can see playing much, although I can see why it's popular. I guess the best and the worst part of the game is having to utilize cover so you don't die right away.
• Windows Live Writer is a great tool! I use it to author the blogs for my website, and this week I have been using it on these Morning Doughnuts posts. My favorite feature is that you can preview your post and see exactly how it will appear on your website. This has been particularly useful since Devhawk and my site look quite different.

Morning Doughnuts 1

Introduction

While Harry is out of the office on a well deserved vacation with his family I will be acting as a guest blogger for his website.

My name is Dale Churchward and I have been working at in Microsoft IT as an application architect since June 2006. Prior to coming to MS I worked in operations in both the telecommunications and healthcare industries. As Harry pointed out I tend to be very operations focused. While I love working on new technologies I am keenly interested in how a design will work in production, and try to ensure we have considered support considerations as part of our designs. Each member of our team comes in with a different background. This helps us as we are each strong in different areas. I also write to my own blog which can be found here. In my own blog I tend to write on a wide array of subjects, depending on my mood at the time. While Harry is away though, I will be posting on technology areas I am interested in, and staying true to his morning coffee vision, albeit with a slightly different take.

The Doughnuts

• The Build Master by Vincent Maraia is an excellent book if you are interested in the build process and how to make it as efficient as possible.
• We had a great meeting with the Patterns & Practices team the other day. Since I am still new to Microsoft it is still a bit overwhelming to meet the authors of documents you have read and used over the years.
• I recently have been spending some cycles working with System Center Operations Manager 2007. I believe that it provides some excellent tools to monitor and repair a system plus it's designed to be service focused.
• Francis Stokes has produced 6 episodes showing what would happen if heaven was being run like a company named God Inc. There are currently 6 episodes. No matter what your belief or lack thereof in a supreme being the videos are hilarious.
  • Episode 1
  • Episode 2
  • Episode 3
  • Episode 4
  • Episode 5
  • Episode 6 
• I have been spending a lot of time thinking about how heartbeat transactions between multiple services should operate. In the drawing below you can see 3 web services and a monitoring one. In the original design the monitoring service was sending heartbeats out to each of the web services to see if they were available. This seem inefficient to me as we really don't care if the monitoring service can reach the web service. What we need to know is if any dependent web services are able to connect. In the drawing we have a web service residing in the extranet (Web Service 1) that sends data to a web service in the corporate network (Web Service 2). We really don't care if the monitoring service can talk with web service 2, but we definitely want to know that web service 1 can get there. Once web service 1 realizes that is can't connect to 2 we then notify the monitoring system so that the owner of web service 2 can take action. Web service 1 still continues sending heartbeats though so that it is aware of when the second web service becomes available again.

Morning Coffee 28

• From the "Ask and ye shall receive" department: A couple weeks ago I wondered how good or bad my Gamerscore conversion rate is. MyGamerCard.net just launched a completion leaderboard where they rank you on your Gamerscore times your completion rate.
• Shane Courtrille pointed out that the prize you receive in from the Xbox Rewards program gets better if your Gamerscore is higher. With a meager 1090 points, I'm in level 1. But those with 10,000+ or more can get a copy of Fuzion Frenzy 2 for completing the challenge.
• Yesterday, I complained that code in my RSS feed looks awful. It appears to be a problem with dasBlog. In validating the HTML is actually XHTML, it screws up the white space. Of course, usually that's not a big deal, but inside a <pre> tag, it is. Until I get a chance to submit a patch to dasBlog to fix this, I'm using CodeHTMLer, which has a "convert white space" option that doesn't use the <pre> tag at all. As a bonus, it even support PowerShell! Note, you have to use the website, not their WLWriter plugin, if you want the convert white space option.
• There's a new beta of Ruby.NET available. Now that I've moved on to PowerShell, I'm only slightly interested in Ruby these days. If I can figure out how to create internal DSLs with PS, what would I need Ruby for? (via Larkware)
• My old team just shipped a single-instance multi-tenancy SaaS sample called LitwareHR. Details are on Gianpaolo's blog, code is up on CodePlex.

Morning Coffee 18

• I'm sure glad Heroes is back. And so far, I'm liking this season of 24 better than the last couple. I especially like how they're introducing Jack's family to the storyline - very cool.
• Bill Gates will be on The Daily Show with Jon Stewart next Monday. I rarely miss an episode (though I typically watch it a day or two later via my DVR) so I'm looking forward to this.
• Roger Sessions latest ObjectWatch newsletter is available. Roger does his usual brand of relating architectural concepts to everyday situations (this time, planning his daughter's wedding). This one was funnier than most - especially since I have a 1 year old daughter - though I'm still waiting for one of these that doesn't contain the words "doppio machiato". If you're reading this Roger, don't worry, I am enjoying every moment.
• Apparently, I am the keeper of obscure development knowledge on my team. My teammate Buzz was getting an error in shdocvw.dll when trying to open an XSD with the BizTalk editor. He's on an interim build of BTS06 R2, so bugs are to be expected, but he wasn't sure what shdocvw was, so he asked me. In case you're curious, shdocvw is the WebBrowser control.
• Did I mention that I left my laptop power cord at the office again on Tuesday? That's three times in four the last five trips to my office (not counting weekends, sick days and training). My boss actually got a spare from his boss that I can leave at the house 24/7.

Common Ground with My Conservative Teammates

I came in this morning to discover my boss and next cube neighbor Rick had decided to spruce up his cube with camo netting. He's ex-Army, so it's not like it's out of character for him. Of course, the camo netting has the exact opposite of it's indented effect, making Rick's cube very easy to find in the farm.

Unlike my last team, most of my teammates are conservatives. But apparently we can find common ground in our opinions of Defense Secretary Rumsfeld. Rick called him an abysmal failure. I couldn't agree more. Dale joked that Rumsfeld was joining our team and moving into Rick's newly camo festooned cube. Rick countered that Rumsfeld was actually joining the Enterprise Architecture group. Heh.

Update: Dale points out he made the joke about Rumsfeld joining EA, not Rick. My bad Dale.

The Two Types of Service Architects

Tomas Restrepo comments on my recent SSB and WCF posts:

Harry Pierson asks how well WCF supports long running tasks. He suggests that WCF does not support them very well, and says that's one reason he likes SQL Server Service Broker so much. I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database. Sure, this is an "expanded universe" with the CLR support in SQL Server 2005, but even so it makes me nervous at times .

You could also consider using a custom service with MSMQ or something like BizTalk Server for this if you had long running processes that were not completely tied to the DB (or a single DB for that matter).

Sam Gentile follows up:

In that same post, but I needed to call it out separate, Tomas rightfully says, "I'd say SSSB is a good match only as long as the long running tasks you're going to be executing are purely database driven and can be executed completely within the database," in response to Harry liking Service Broker so much. Talk about a narrow edge case. That's way I never really got excited or cared about Service Broker. Its a narrow solution to a special edge case when everything is database driven and can be executed totally inside the database. That's the old Microsoft Data-Driven Architecture for sure. Me, I'd rather have a rich Domain-Driven architecture most of the time. Then if you have Oracle databases in your architecture too, where does it leave you? Nowhere.

As you might expect, I have a few comments,  clarifications and corrections.

First, Tomas' statement that Service Broker only supports service logic "executed completely within the database" in flat out wrong. Service Broker can be used from any environment that can connect to SQL Server and execute DML statements. If you can call SELECT/INSERT/UPDATE/DELETE, then you can also call BEGIN DIALOG/SEND/RECEIVE/END CONVERSATION. This includes Windows apps and services, web apps and services, console apps and even Java apps. Of course, you can also access Service Broker from stored procedures if you wish, but you're not limited to them as Tomas suggested.

Tomas' misconception may come from a feature of Service Broker called Activation. Activation is a feature of Service Broker that dynamically scales message processing to match demand. For example, Service Broker can be configured to launch a new instance of a specified stored procedure if messaging processing isn't keeping up with incoming message traffic on a given queue. This is called internal activation and because it uses stored procedures it does execute within the database as Thomas said. Service Broker also supports external activation where it notifies an external application when activation is needed. You do have to build an application to host your service logic and handle these notifications, but that application doesn't execute within the database. So while you could argue that it's easier to execute your service logic within the database (no need to build a separate host app), it's not required.

Given that you don't have host your service logic in the database, then you're also not limited to "a single DB" as Tomas suggests. You don't, in fact, have to put your Service Broker queues in the same database with your business data. So if you have Oracle in your environment, like the scenario Sam mentioned, you would host your service logic in an external application that processed messages from a queue in a SQL 2005 database while accessing and modifying business data from tables in the Oracle database. Using multiple databases does require using distributed instead of local transactions, but if you're using MSMQ as Tomas recommended, you're already stuck with the DTC anyway.

Finally, I didn't get Tomas' "purely database driven" or Sam's "everything is database driven" comments at all. While there are exceptions, the vast majority of systems I've ever seen/built/designed have essentially been one or more stateless tiers sitting in front of a stateful database. If it's a traditional three tier web app, there's a stateless presentation tier, a stateless business logic tier and a stateless data access logic tier. For a web service, there's no presentation tier, but there's is the stateless SOAP processing tier typically provided by the web service stack. Does this mean the vast majority of web apps and services are  "purely database driven" too? If so, then I guess it's a good thing, right?

In the end, maybe there are two types of service architects - those that believe the majority of services will be atomic and those that believe the majority of services will be long running. For atomic services, Service Broker is overkill. But if it turns out that most services are long running, WCF's lack of support is going to be a pretty big roadblock.

I'm obviously in the long running camp. I'm not sure, but I get the feeling this is the less popular camp, at least for now. We'll have to wait to see, but I do know is that whenever someone brings me what they think is an atomic business scenario, it doesn't take much digging to reveal that the atomic scenario is actually a single step of a long running business scenario that also needs to be automated.

Here's a question for Tomas, Sam and the rest of you: Which group do you self select into? Are most services going to be atomic or long running in the (pardon the pun) long run?

Essential Windows Workflow Foundation

On Don's recommendation, I picked up Essential WF. In the forward, Don writes "[S]omething big is about to happen." I'm only part way thru chapter one, and this is already a must read. Go get it. Now.

In the preface, they define the term "Reactive Program", which I'm adding to my personal lexicon.

"Windows Workflow Foundation (WF) is a general-purpose programming framework for creating reactive programs that act in response to stimulus from external entities. The basic characteristic of reactive programs is that they pause during their execution, for unknown amounts of time, awaiting input."

That "unknown amounts of time" is the kicker. Here's a paragraph from early in chapter one that expands on that:

"Real-world processes take a long time - days, weeks, or even months. It is wishful thinking to assume that the operating system process (or CLR application domain) in which the program begins execution will survive for the required duration."

Gee, that sounds familiar doesn't it?

Is WCF "Straightforward" for Long Running Tasks?

My father sent me a link to this article on SOA scalability. He thought it was pretty good until he got to this paragraph:

Long-running tasks become more complex. You cannot assume that your client can maintain a consistent connection to your web service throughout the life of a task that takes 15 minutes, much less one hour or two days. In this case, you need to implement a solution that follows a full-duplex pattern (where your client is also a service and gets notified when the task is completed) or a polling scheme (where your client checks back later to get the results). Both of these solutions require stateful services. This full-duplex pattern becomes straightforward to implement using the Windows Communications Foundation (Indigo) included with .NET 3.0.

When I first saw duplex channels in WCF, I figured you can use them for long running tasks also. Turns out that of the nine standard WCF bindings, only four support duplex contracts. Of those four, one is designed for peer-to-peer scenarios and one uses named pipes so it doesn't work across the network, so they're obviously not usable in the article's scenario. NetTcp can only provide duplex contracts within the scope of a consistent connection, which the author has already ruled out as a solution. That leaves wsDualHttp, which is implemented much as the author describes, where both client and the service are listening on the network for messages. There's even a standard binding element - Composite Duplex - which ties two one-way messaging channels into a duplex channel.

Alas, the wsDualHttp solution has a few flaws that render it - in my opinion at least - unusable for exactly these sorts of long running scenarios. On the client side, while you can specify the ClientBaseAddress, you can't specify the entire ListenUri. Instead, wsDualHttp generates a random guid and tacks it on the end of your ClientBaseAddress, effectively creating a random url every time you run the client app. So if you shut down and restart your client app, you're now listening on a different url than the one the service is going to send messages to and the connection is broken. Oops.

The issues don't end there. On the service side of a duplex contract, you get an object you can use to call back to the client via OperationContext.Current.GetCallbackChannel. This works fine, as long as you don't have to shut down your service. There's no way to persist the callback channel information to disk and later recreate it. So if you shut down and restart your service, there's no way to reconnect with the client, even if they haven't changed the url they're listening on. Oops.

So in other words, WCF can do long running services using the wsDualHttp binding, as long as you don't restart the client or service during the conversation. Because that would never ever happen, right?

This is part of the reason why I'm sold on Service Broker. From where I sit, it looks like WCF can't handle long running operations at all - at least, not with any of the built in transports and bindings. You may be able to build something custom that would work for long running services, I'm not a deep enough expert on WCF to know. From reading what Nicholas Allen has to say about CompositeDuplex, I'm fairly sure you could work around the client url issue if you built a custom binding element to set the ListenUriBaseAddress. But I have no idea how to deal with the service callback channel issue. It doesn't appear that the necessary plumbing is there at all to persist and rehydrate the callback channel. If you can't do that, I don't see how you can reliably support long running services.

The Other Foundation Technology

I mentioned last week that WF "is one of two foundation technologies that my project absolutely depends on". Sam Gentile assumes the other foundation technology is WCF. It's not.

As a quick reminder, my day job these days is to architect and deliver shared service-oriented infrastructure for Microsoft's IT division. These services will be automating long running business operations. And when I say long running, I mean days, weeks or longer. While there will surely be some atomic or stateless services, I expect most of the services we build will be long running. Thus, the infrastructure I'm responsible for has to enable and support long running services.

The other foundation technology my project depends on is Service Broker. Service Broker was expressly designed for building these types of long running services. It supports several capabilities that I consider absolutely critical for long running services:

• Service Initiated Interaction. Polling for changes is inefficient. Long running operations need support for the Solicit-Response and/or Notification message exchange patterns.
• Durable Messaging. The first fallacy of distributed computing is that the network is reliable. If you need to be 100% sure the message gets delivered, you have to write it to disk on both sides.
• Service Instance Dehydration. It's both dangerous and inefficient to keep an instance of a long running service in memory when it's idle. In order to maximize integrity (i.e. service instances survive a system crash) as well as resource utilization (i.e. we're not wasting memory/CPU/etc on idle service instances), service instances must be dehydrated to disk.

In addition to these capabilities, Service Broker supports something called Conversation Group Locking, which turns out to be important when building highly scalable long running services. Furthermore, my understanding is that Conversation Group Locking is a feature unique to Service Broker, not only across Microsoft's products but across the industry. Basically, it means that inbound messages for a specific long running service instance are locked so they can't be processed on more than one thread at a time.

Here's an example: let's say I'm processing a Cancel Order message for a specific order when the Ready to Ship message arrives for that order arrives. With Conversation Group Locking, the Ready to Ship message stays locked in the queue until the Cancel Order message transaction is complete, regardless of the number of service threads there are. Without Conversation Group Locking, the Ready to Ship message might get processed by another service thread at the same time the Cancel Order message is being processed. The customer might get notified that the cancellation succeeded while the shipping service gets notified to ship the product. Oops.

There's also an almost-natural fit between Service Broker and Windows Workflow. For example, a Service Broker Conversation Group and a WorkflowInstance are roughly analogous. They even both use a Guid for identification, making mapping between Conversation Group and WF Instance simple and direct. I was able to get prototype Service Broker / WF integration up and running in about a day. I'll post more on that integration later this week.

Last but not least, Service Broker is wicked fast. Unfortunately, I don't have any public benchmarks to point to, but the Service Broker team told me about a private customer benchmark that handled almost 9,000 messages per second! One of the reasons Service Broker is so fast is because it's integrated into SQL Server 2005, which is is pretty fast in it's own right. Since Service Broker is baked right in, you can do all your messaging work and your data manipulation within the scope of a local transaction.

Service Broker has a few rough areas and it lacks a supported managed api (though there is a sample managed api available). Probably the biggest issue is that Service Broker has almost no interop story. If you need to interop with a Service Broker service, you can use SQL Server's native Web Service support. or the BizTalk adapter for Service Broker from AdapterWORX. However, I'm not sure how many of Service Broker's native capabilities are exposed if you use these interop mechanisms. You would probably have to write a bunch of application code to make these capabilities work in an interop scenario.

Still, I feel Service Broker's unique set of capabilities, its natural fit with WF and its high performance make it the best choice for building my project's long running services. Is it the best choice for your project? I have no idea. One of the benefits of working for MSIT is that I get to focus on solving a specific problem and not on solving general problems. I would say that if you're doing exclusively atomic or stateless services, Service Broker is probably overkill. If you're doing any long running services at all, I would at least give Service Broker a serious look.

Lip Service on Long Term Planning

Long term readers know my liberal political leanings. So it should come as no surprise to them that I read liberal blogs like Washington Monthly. But this isn't a post about politics, it's a post about planning:

This kind of long-term planning — in politics, in business, in nearly every walk of life — is something that nearly everyone says they support, but when push comes to shove very few people are willing to back it up. There's always something this week, or this month, or this year that seems uniquely crucial and demands our attention. Next year there will be something else, and the year after that something else again. The long-term stuff simply never gets done unless someone like Dean is willing to go to the mat for it.
[Building a Better Movement - Kevin Drum]

I don't have much to add to this, except that planning is a big part of architecture, especially architecture in the enterprise (which may or may not be "Enterprise Architecture"). Who "goes to the mat" for the long-term stuff at your company? Or does the long-term stuff simply never get done?

Thoughts on the SOA Workshop

Last week, I attended an SOA workshop presented by SOA Systems and delivered by "top-selling SOA author" Thomas Erl. It was two SOA-jammed days + the drive to Vancouver and back primarily discussing SOA with Dale. In other words, it was a lot of SOA. I went up expecting to take Erl to task for his "Services are Stateless" principle. However, that turned out to be a misunderstanding on my part about how Erl uses the term stateless. However, while Erl and I agreed on optimizing memory utilization (which is what he means by stateless), that wasn't much else when it came to common ground. As I wrote last week, Erl's vision of service-orientation is predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse.

Erl spends a lot of time talking about service reuse. I think it's safe to say, in Erl's mind, reuse is the primary value of service orientation. However, he didn't offer any reason to believe we can reuse services any more successfully than we were able to reuse objects. Furthermore, his predictions about the amount of reuse you can achieve are completely made up. At one point, he was giving actual reuse numbers (i.e. 35% new code, 65% existing code). When I asked him where those numbers came from, Erl admitted that they were "estimates" because "there hasn't been enough activity in serious SOA projects to provide accurate metrics" and that there is "no short term way of proving" the amount of service reuse. In other words, Erl made those numbers up out of thin air.

This whole "serious" or "real" SOA is a major theme with Erl. One the one hand, I agree that SOA is a horribly overused term. Many projects labeled SOA have little or nothing to do with SO. On the other hand, it seems pretty convenient to chalk up failed projects as not being "real" SOA so you can continue to spout attractive yet completely fictional reuse numbers. I asked about the Gartner's 20% service reuse prediction and Erl responded that low reuse number was because the WS-* specs are still in process. While I agree that the WS-* specs are critical to the advancement of SO, I fail to see how lack of security, reliable messaging and transactions are holding back reuse. If anything, I would expect those specs to impede reuse, as it adds further contextual requirements to the service.

While I think Erl is mistaken when it comes to the potential for service reuse, he's absolutely dreaming when it comes to the organizational structure and behavior that has to be in place for this potential service reuse to happen in the first place. I'm not sure what Erl was doing before he became a "top-selling SOA author," but I find it hard to believe it included any time in any significantly sized IT shop.

Erl expects services - "real" services, anyway - to take around 30% more time and money than he traditional siloed approach. The upside for spending this extra time and money is the potential service reuse. The obvious problem with this is that we don't know how much reuse we're going to see for this extra time and money. If you spend 30% more but can only reuse 20% of your services (as Gartner predicts), is it worth it? If you end up spending 50% more but are only able to reuse 10% of your services, is it worth it? Where's the line where it's no longer worth it to do SOA? Given that there's no real way to know how much reuse you're going to see, Erl's vision of SOA requires a huge leap of faith on the part of the implementer. "Huge leap of faith" doesn't go so well with "corporate IT department".

Furthermore, the next IT project I encounter that is willing to invest any additional time and money - much less 30% - in order to achieve some theoretical organizational benefit down the road will be the first. Most projects I've encountered (including inside MSIT) sacrifice long term time and money in return for short term gain. When asked how to make this 30% investment happen, Erl suggested that the CIO has to have a "dictatorial" relationship with the projects in the IT shop. I'm thinking that CIO's that adopt a dictatorial stance won't get much cooperation from the IT department and will soon be ex-CIO's.

In the end, I got a lot less out of this workshop than I was hoping to. As long as SO takes 30% more time and money and the primary benefit is the same retread promises of reuse that OO failed to deliver on, I have a hard time imagining SO making much headway.

PS - I have a barely used copy of "Service-Oriented Architecture: Concepts, Technology, and Design" if anyone wants to trade for it. It's not a red paperclip, but it's like new - only flipped through once. :)

Stateless != Stateless

A while back, I blogged that Services Aren't Stateless, in response to some stuff in Thomas Erl's latest book. At the time, I mentioned that I was looking forward to discussing my opinions with Erl when I attended his workshop. I've spent the last two days at said workshop. I'll have a full write-up on the workshop later this week, but I wanted to blog the resolution to this stateless issue right away.

At the time, I wrote "I assume Erl means that service should be stateless in the same way HTTP is stateless." Turns out, my assumption was way way wrong. When he addressed this topic in his workshop, he started by talking about dealing with concurrency and scalability, which got me confused at first. Apparently, when Erl says stateless, he's referring to minimizing memory usage. That is, don't keep service state in memory longer than you need to. So all the stuff about activity data, that's all fine as per Erl's principles, as long as you write it out to database instead of keeping it in memory. In his book, he talks about the service being "temporarily stateful" while processing a message. When I read that, I didn't get it - because I was thinking of the HTTP definition of stateless & stateful. But if we're just talking about raw memory usage, it suddenly makes sense.

On the one hand, I'm happy to agree 100% with Erl on another of his principles. Frankly, much of what he talked about in his workshop seems predicated on unrealistic organizational behavior and offer at best unproven promises of cost and time savings in the long run via black box reuse. So to be in complete agreement with him on something was a nice change of pace. Thomas is much more interested in long-running and async services than I originally expected when I first flipped thru his book.

On the other hand, calling this out as a "principle of service orientation" hardly seems warranted. I mean, large scale web sites have been doing this for a long time and SQL Session State support has been a part of ASP.NET since v1. Furthermore, using the term "stateless" in this way is fundamentally different from the way HTTP and the industry at large uses it, which was the source of my confusion. So while I agree with the concept, I really wish Erl hadn't chosen an overloaded term to refer to it.

Feasible Service Reuse

Yesterday, I posted about services and reuse. More to the point, I posted why I don't believe that business services will be reusable, any more than business objects were reusable. However, "can't reuse business services" isn't the whole story, because I believe in different kinds of reuse.

The kind of reuse I was writing about yesterday is typically referred to as "black box reuse". The idea being that I can reuse the item (object or service) with little or no understanding of how it works. Thomas Beck wrote about colored boxes on his blog yesterday. Context impacts reuse - the environments in which you plan to reuse an item must be compatible with what the item expects. However, those contextual requirements aren't written down anywhere - at least, they're not encoded anywhere in the item's interface. Those contextual requirements are buried in the code - the code you're not supposed to look at because we're striving for black box reuse. Opaque Requirements == No Possibility of Reuse.

As I wrote yesterday, David Chappell tears this type of reuse apart fairly adeptly. Money quote: "Creating services that can be reused requires predicting the future". But black box reuse this isn't the only kind of reuse. It's attractive, since it's simple. At least it would be, if it actually worked. So what kind of reuse doesn't require predicting the future?

Refactoring.

I realize most people probably don't consider refactoring to be reuse. But let's take a look at the official definition from refactoring.com:

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations. Each transformation (called a 'refactoring') does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it's less likely to go wrong. The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring

Two things about this definition imply reuse. First, refactoring is "restructuring an existing body of code". It's not rewriting that existing body of code. You may be making changes to the code - this certainly isn't black box reuse - but you're not scrapping the code completely and starting over. Second, refactoring is making changes to the code "without changing its external behavior". You care about the code's external behavior because somewhere, some other code is calling the code you're refactoring. Some other existing piece of code that you don't want to change - i.e. that you want to reuse.

When you refactor, you still reuse a significant amount of the code, but you’re not having to predict the future to do it. Refactoring is the kind of reuse I believe in.

In his article, David talks about types of reuse such as business agility, adaptability and easily changeable orchestration. These look a lot more like refactoring than black box reuse to me. Unfortunately, David waves these away, saying  "Still, isn’t this just another form of reuse?". Reconfiguration hardly qualifies as "predict the future" style reuse that he spends the rest of the article arguing against. It's just one paragraph in an otherwise splendid article, so I'll give him a pass this time. (I'm sure he's relieved.)

A Question of Context

A couple of weeks ago, David Chappell posted a great article on SOA and the Reality of Reuse. When someone mentions the idea of using SOA for reuse, I cringe. David does a great job blowing the "SOA for Reuse" argument out of the water. In the future, I will just send that link rather than spending the time arguing out the point.

But something nagged at the back of my brain about that post. David starts by talking about object reuse before making the parallel to services. The problem with that comparison is that object reuse hasn't been a failure. When was the last time you wrote a String class? A Linked List? A Button? There's been support for Buttons in Windows since at least the 3.x days (probably longer, but that's before my time). Whatever your OO language of choice, there's a big collection of reusable objects to go with it.

Given his position as "famous technology author", I'm assuming David is well aware of successes of object reuse. Furthermore, I doubt it was an accident that in his article he writes that "reuse of business objects failed" (emphasis added). While there has been success around object reuse, essentially none of those successes have been in a business scenario. In fact, there have been some high profile projects such as Microsoft's Business Framework and IBM's San Francisco Project that have crashed and burned been significantly less than successful.

So here's the question: given that general object reuse has seen some success, what's so different about business objects that causes reuse to fail utterly? Since we're really interested in service reuse, knowing why some object reuse succeeds and other reuse fails will help us understand which services are likely to be reusable and which wont. I would say that success of object reuse hinges on context.

Wikipedia gives this definition of context: "The context of an event, word, paradigm, change or other reality includes the circumstances and conditions which surround it." (emphasis in original) For example, the word "order" is ambiguous. If you're using a procurement system for the military, you could conceivably be given an order to place an order. (OK, that's silly. But you get the idea.) The word "order" has two different meanings. However, the words that surround the ambiguous term make the meaning clear. An order that you place is different that an order that you give. That's context.

A string or a linked list or a button has very little in the way of contextual needs. That is to say you can use it the exact same way in a wide variety of environments. A business object on the other hand has significant contextual requirements, which makes reuse difficult or impossible. For example, a Purchase Order object from the above-mentioned military procurement system sounds like it might be reusable. At least until you take into account the differences between branches of the military, between ordering tanks and ordering uniforms, between active units and reserve units, etc. Once the generic Purchase Order has been specialized to the point of practical usability for a given scenario, it's no longer reusable in the general case.

Taking this back to the service realm, likewise I figure the reusable services will be the ones with little or no contextual needs. A good example is the identity and directory services such as Active Directory and its competitors. Sure, you use LDAP not SOAP to access it, but AD is certainly both reusable and a service plus it's in wide usage. Other candidates for reusable services my team is looking at are service directory, management and operations, business activity monitoring and provisioning.

I actually think there will be less reuse in services than there was with objects. The value of reuse of services has to exceed not only the contextual issues but also the overhead of distributed access. Calling across the network is an expensive operation - whatever's on the other side better be worth the drive. I'm guessing for services, more often than not, reuse won't be worth the trip (if it's possible at all).

Update - David pointed out to me that the last paragraph of his article begins:

Object reuse has been successful in some areas. The large class libraries in J2EE and the .NET Framework are perhaps the most important examples of this.

Doh! I guess my "assumption" that David is aware of successful object reuse was correct.

SOA Sample Scenario

So now that I'm back, we're beginning work in earnest on my project. For those not following along at home, my project is to deliver shared service oriented infrastructure for Microsoft's internal IT department. We've spent the time since I moved over working on our business justification, and now we're moving into specifications and prototyping. That is, I get to starting getting my hands dirty.

As part of the prototyping efforts, we're looking to build some sample business services on top of our prototype infrastructure. The idea being to both illustrate what we're building as well as have a playground where we can experiment with new ideas. During the prototyping, I'll be pretty involved with the development. But once we start writing production code, the dev team will take that over but I will continue to own the service playground. I've been kicking a playground idea like this around for several years, so I'm pretty excited about it.

The question is, what kind of business scenario should we build? I want the sample business services to be something interesting and real-world-esque. But of course, it can't be too complex since I wasn't hired to build a playground as my primary job function.

So far, we have two primary ideas:

• Enterprise Management System: AdventureWorks is the primary sample database that ships with SQL Server. They have business scenarios around Sales & Marketing, Product Management, Purchasing and Manufacturing. This sounds suspiciously like an ERP/CRM/SFA type enterprise system. On the plus side, MS is an enterprise so things like ERP/CRM/SFA are the types of solutions we need/use/buy/build internally. On the negative side, it's complex to do real-world and teams that actually do ERP/CRM/SFA inside Microsoft might dismiss the infrastructure if the playground isn't real-world enough.
• Prediction Market: If you've ever seen Hollywood Stock Exchange (HSX) or Tradesports, those are prediction markets. The basic idea is that you trade on predictions, rather than companies like you would in a stock market. Using HSX as an example, you get 2 million "Hollywood Dollars" (i.e. play money) to invest in upcoming movies and / or movie stars. Those movies and stars pay out based on the money they make at the box office. They also have derivatives for opening weekend, blockbusters and the Oscars. HSX even sells forecasting and prediction services based on the HSX market. Of course, I can't build something quite so extensive, but we could get pretty far with this idea. The upside is that it's relatively simple (compared to enterprise management systems) and there's little conflict with existing systems inside Microsoft. The downside also is that it's relatively simple and not like anything we're building inside Microsoft.

I'm leaning towards the prediction market, as it sounds more fun to build and experiment with. What do you think?

Cut Out The Middle Man

Nick Malik is an architect in MSIT's Enterprise Architecture group. He's been blogging a while, though I only discovered his blog a couple of weeks ago. Yesterday he posted about OMG's SOA SIG's Draft RFI on EDA and it's relationship to SOA and BPM. That's a veritable alphabet soup of acronyms! To translate, the Object Management Group's Special Interest Group on Service Oriented Architecture has posted a draft Request for Information on Event Driven Architecture and it's relationship to Service Oriented Architecture and Business Process Management. Here's the summary from the actual document:

The EDA Sub-group of the OMG SOA SIG seeks information from members of the EDA, BPM and SOA community as well as anyone interested in promoting standards in this area. Requested information will be evaluated by the EDA Sub-group, resulting in the development of Requests for Proposal(s) (RFP) for standardization of Event definition, relationship between EDA, BPM and SOA that will ultimately allow development of standards for Complete Life Cycle of Events -Ontology of Events, Sense and Respond Services, Events Metrics and processing of complex events. Please note that it is our intent to develop modeling standards for the EDA/SOA and EDA-Business Process interaction and provide standards for the implementation of that interaction as well.

First off, I'm a bit wary about this part: "it is our intent to develop modeling standards". Of course, OMG is responsible for UML and long time readers should be well aware of my opinion of UML. I don't want to set the bozo bit on an entire organization, but I am skeptical that any new modeling "standards" from OMG will be any more effective than the Unwanted Modeling Language.

Secondly, EDA seems to be vaguely defined, if at all. Wikipedia has this to say about EDA:

An event-driven architecture (EDA) defines a methodology for designing and implementing applications and systems in which events transmit between loosely coupled software components and services. An event-driven system is typically comprised of event consumers and event producers. Event consumers subscribe to an intermediary event manager, and event producers publish to this manager. When the event manager receives an event from a producer, the manager forwards the event to the consumer. If the consumer is unavailable, the manager can store the event and try to forward it later. This method of event transmission is referred to in message-based systems as store and forward.
[emphasis in original]

Assuming events are encoded as messages, then you can rewrite "event consumers / event producers" as "message receivers / message senders" and you really blur the line with SOA? 

The big difference in EDA seems to be the use of an "intermediary event manager". The problem I have with this approach is that the "intermediary event manager" works fine if you have a small number of endpoints, but how will it scale to handle hundreds of systems? Thousands? Tens of thousands? I don't see how the centralized event manager approach can possibly scale to handle tens of millions of events delivered between tens of thousands of systems. The management of such a system would be a nightmare? If a business process went south, you would obviously look in the central event manager as the source of the problem, but I would think that would be like finding a needle in a haystack. You could federate the event managers, instead of attempting to scale out the center. But a federated event manager approach would seem to defeat much of the purpose of an EDA in the first place. If you're going to federate your event managers, why not cut out the middle man and make each event producer it's own event manager as well? What is the benefit of separating these capabilities?

I guess fleshing out EDA isn't a bad idea, but it seems more like a solution looking for a problem to me.

Business Processes Are Services Too

I've been having a conversation with Piyush Pant over on his blog that started as a comment he left on my Services Aren't Stateless post. He thinks that I'm "missing the crucial point here by implicitly conflating business process and service state". While Piyush hasn't really defined what he means by these terms, I think I understand what he's getting at. Yes, process and service state are different in many ways, but they are also similar in that they are both service private data.

Pat Helland (side note - I wish Pat would start blogging again) wrote an article some time ago titled Data on the Outside vs. Data on the Inside where he talked about the differences between service private data and data in the space between the services. For example, data on the outside is immutable, requires an open schema for interop, doesn't need encapsulation and is representable in XML. Service private data is not immutable, doesn't need an open schema for interop, requires encapsulation and is typically stored in a SQL RDBMS. So on this front, process and service state are both service private data so conflating them makes some sense.

However, what's not in the article is the idea of Resource and Activity data. Not sure why Pat didn't include this in the article, but he was talking about it as far back as PDC 2003. Stu Charlton described the difference between resource and activity data in his Autonomous Services article:

Activity Data - This is "work in progress" data for any long-running business operation, and is usually encapsulated by business logic. A classic example is a shopping cart in any e-commerce system. This data is mutable, but typically has low concurrency conflicts, as it is not widely shared. Typically activity data retires after a long running operation completes, and may be archived in a decision support system for later analysis.

Resource Data - This is "state of the business" data, which represents the resources of an organization, and is usually encapsulated by business logic. Examples are: room availability in a hotel, inventory levels in a warehouse, account statuses, employee and customer information. Some resources have a small life span, others may last a very long time (years). Resource data is usually volatile with potential for high concurrency conflicts.

So I'm fairly sure that when Piyush says "process state" I should hear "activity data". Similarly "service state" is "resource data". The differences between activity and resource data lead to some interesting implementation artifacts, which I assume he getting at when he says I'm conflating the two. For example, since activity data like shopping cart has low or no concurrency issues, using an optimistic concurrency scheme is entirely appropriate, which you would never use for highly volatile resource data like warehouse inventory levels. In fact, since activity data doesn't have concurrency issues, you could even store it inside an instance of workflow or orchestration, which gets serialized to a persistent store when it's in an idle state.

However, the fact that activity and resource data is handled differently doesn't mean that most services won't have activity data. When Thomas Erl says that that stateless services is a "common principle of service orientation", essentially what I think he's saying that services should only have resource data. And as I said before, this seems wrong to me. Sure, some services will be stateless. But all services? Services implement business capabilities. Most business capabilities are long running processes. Doesn't that imply that most services in the enterprise will need to be long running workflows or orchestrations?

So for the most part, Piyush and I just seem to have different names for the same concepts. The one issue I have with Piyush's descricription of process and service state is that he seems to implicitly assume that processes aren't services. Why not? Again, not all services will be processes, but if you're not exposing processes as services, how exactly are you exposing them?

Is SML Another Unwanted Modeling Language?

Piyush Pant wonders if SML solves a problem that nobody has? He also points out SysML project, which recently got folded into the OMG. Well, that explains why we called it "Service" instead of "System" modeling language.

Now that I work in IT, I can definitely say that SML will eventually solve a problem that I have. Most people agree that operations today is way to dependent on manual processes to scale effectively. Now SML doesn't solve that issue directly - as Piyush pointed out SML is a meta-modeling specification. However, SML is the foundation for the next generation of operational modeling tools like what we see in Visual Studio Team System for Architects. As I wrote several years ago, VSTS:A solves a very common problem - developers lack of understanding about the deployment environment. Piyush, haven't you ever had a long weekend going back to the drawing board because the solution you had built was undeployable and you didn't discover that fact until the operations team attempted to deploy it? If you haven't, I envy you.

On the surface, I agree with Piyush when he says that "history of software is littered with unsuccessful attempts to impose monolithic modeling constructs". However, the fact that it keeps happening indicates the problem hasn't been solved. Wanting to solve a problem and being able to solve a problem are two different things. Furthermore, the history of software is also littered with very successful attempts to raise the level of abstraction by the introduction of new programming languages: C, C++, VB and Ruby are all examples of this. Given that Code is Model, what we have is a history of software littered with some successful and some unsuccessful modeling constructs. I would argue that the successful modeling constructs have taken a bottom up approach - build a language a small abstraction step above something that actually runs and compile down. These unsuccessful modeling constructs (*cough* UML *cough*) take a top down approach - build a language way above anything that actually runs and hope a miracle happens to keep it in sync with the stuff you actually build.

The question is whether SML will be top-down (i.e. a failure) or bottoms-up (i.e. a success). So far, it's to early to tell, but I have high hopes.

Service Modeling Language

On the one hand, this seems like a somewhat arbitrary name change - from System Defintion Model to Service Modeling Language. And the use of the term "Service" instead of "System" seems sketchy to me.

On the other hand, you gotta love this line from the press release: "The group plans to submit the draft specification to an industry standards organization later this year." Given the companies involved in "the group" - MSFT, IBM, BEA, Sun, Dell, BMC, Cicso, Intel, HP and EMC - you gotta think SML has a bright future.

Here's hoping that publishing the SML spec is the first step in a public-workshop-based revision process, a la the Web Services Protocol Workshops.

deVadoss Down on SOA

My old boss's boss seems like he was in a downer mood yesterday. First, he blogged about the "Myth of Reuse in SOA", then the "Achilles Heel of SOA". Actually, truth be told, I agree with him on both counts.

I slam the door on the reuse argument every time it comes up in my new job. Actually, I slam the door on what I call "Naive Reuse". When John talks factoring for agility, he's talking about a form of reuse - similar to how use "reuse" code when you refactor. What does it mean to refactor service? How about refactoring your enterprise?

As for the Achilles Heel "data problem", I think that's an artifact of the prevailing stateless request/response mindset most people have about services that I touched on yesterday. I think Pat Helland described a very good approach for dealing with data in an SOA, but I haven't seen it implemented broadly. Rest assured, many of the concepts Pat described are at the forefront of my thinking as my new project takes shape.

Services Aren't Stateless

My teammate Dale and I are going to an SOA Workshop in Vancouver in mid September. The workshop is put on by SOA Systems, which was founded by "top-selling SOA author" Thomas Erl. I have a copy of his first book, but I've never really opened it. Dale let me borrow Erl's second book. I figured since I was going to see him speak, I should at least flip through his books.

I was looking thru the chapter 9 "Principles of Service-Orientation". Most of them are spot on, if not exactly news. Services are loosely coupled, autonomous and share a formal contract. Yep, with you so far. But then I got to this one:

Services are Stateless
Services should not be required to manage state information, as that can impede their ability to remain loosely coupled. Services should be designed to maximize statelessness even if that means deferring state management elsewhere.

This seems way wrong to me on several levels. Now I'm really looking forward to going to Erl's workshop, so I can discuss this with him face-to-face.

First off, his terminology is confusing. I have a hard time believing he really think services in general should have no state at all. I'm sure there are some examples of completely state-free services, but I believe they are both very rare and fundamentally uninteresting. A simple calculator service has no state, but why would you actually build or use one (except as a demo)? I assume Erl means that service should be stateless in the same way HTTP is stateless. IMO, stateless is poor description of HTTP. Connectionless or sessionless would be more accurate.

Regardless of my opinions on poor terminology, the problem with stateless services is that many - perhaps most - business operations aren't stateless. And while HTTP is stateless, as soon as you use cookies, it becomes a stateful protocol. If you don't believe business operations are stateful, try buying something on your favorite ecommerce site with your cookies disabled. Most sites will give you a "Your computer must have cookies enabled" error message. Sites that still work are embedding a session ID in the URL instead of in a cookie (ASP.NET has built in support for this type of Cookieless Session State). Either way, state is required for even the simple task of ordering something from a web site.

If most business operations aren't stateless, why should services that implement business operations be stateless? This seems like a violation of the "but no simpler" part of Einstein's famous paraphrased quote.

Slides from Gartner EA Conference

My old team decided to post the slides from my sessions at the Gartner EA Summit last month. Well, they've posted PDF versions of said slides and the fonts are somewhat off, but you'll get the idea.

• Beyond SOA: Understanding the Role of Users in Architecture
• The Dell Integrated Desktop: A Case Study of the Role of Users in Architecture

If you're interested in more info on the Dell Integrated Desktop, you can check out the full case study.

FYI, Gartner recorded audio for these sessions and supposedly it's on its way to me. I'll try and post the audio with synced slides as soon as I can.

New Teammates Blogging

I'm settling in to my new job. One way to tell, read Dale Churchward's blog. Dale's a teammate of mine. He only joined Microsoft a few months ago. Apparently, he used to blog at his old job, but either way we've now doubled the number of bloggers on my new team, with hopefully more to follow.

In addition to his opinions of political discource and the Seahawks chances next season, Dale's got some interesting posts on data integration and system diagrams. Check it out.

Gartner EA Summit Day Two

I didn't post a day two wrap up of the Gartner EA Summit because I only made it to my session and booth duty right afterwards. That wasn't the original plan - the pipes in the bathroom of my hotel room kept banging so I didn't get much sleep.

My session went well. I heard from several people afterwards that it was their favorite session or that it was the highlight of the conference. Nice anecdotal evidence, but I still want to see the scores. They recorded the session, hopefully I can get it so I can publish it here. I had lots of great conversations afterwards (as expected). Maybe Gartner will have me back next year with twelve months of my new project under my belt.

One suggestion for the Gartner folks. Next year, don't pick a logo with an arrow in it. I got a little confused when I first showed up because I followed the arrow and ended up on the wrong side of the hotel from the event. My friend Scott snapped this picture of a sign with two arrows pointing in different directions.

Architecture by Powerpoint

Adam Stanley left the following comment about the MS booth @ the Gartner EA Summit and I just had to share.

One of the true revalations of being an enterprise architect is in the books that Microsoft was giving away. One on enterprise patterns, one of powerpoint. I do a lot of powerpoint! :)

The book in question is Beyond Bullet Points, which I highly recommend to architects and non-architects alike.

Gartner EA Summit Day One

Moving On...

It's not the biggest job change news this week (or the day), but after three years on Architecture Strategy and six years total as an evangelist, I'm moving on to a new role. After six years, I decided it was time for me to put my money where my mouth is as well as get my hands dirty building something more substantial than buzz.

I'll be moving over into Microsoft's IT division as a member of the Integration Center of Excellence Architecture Team. Integration, as you might guess, is a euphemism here for service-orientation. My team is tasked with architecting and delivering the shared service-oriented infrastructure for four of the biggest projects Microsoft IT will be delivering in the next year. Last time I changed jobs, I lamented that "With each job I take at MSFT, coding seems to become less a part of the job description." Happily, this is NOT the case this time.

About a year ago, Microsoft hired Stuart Scott to run the business apps side of IT as one of our two CIOs (our other CIO Ron Markezich oversees the IT infrastructure). Stuart was kind enough to spend about an hour with me last week explaining his vision for how he sees MSIT evolving under his leadership. Here's what he said in a recent interview:

PressPass: How do you see Microsoft IT evolving?

Scott: There is a broader role for IT to play at the front end of the development of products and services. Our IT organization knows a lot about the challenges that other IT organizations face because we build and maintain the IT backbone of a massive worldwide enterprise. IT must become future-thought leaders in the development of the product roadmap for our enterprise products.

By using our internal applications and experiences to build better products for our enterprise customers, we have the potential to solve the challenges that other IT organizations face. We’re heavily involved in dogfooding our products once they’ve been developed, but we also see a role closer to the front end of the product development cycle. Business Intelligence is one area where we will be partnering with the product groups and Finance as we build out our internal capability. I want to ensure that any product we develop to meet the needs of Microsoft, also meets the needs of the marketplace.

He was also very frank about the current state of affairs in MSIT relative to the vision. He was quoted in China Information World (no link, sorry) as saying that "The systems Microsoft now uses are already 14 years old and based on previous versions of windows, so from a systems capability perspective, they cannot support the needs of the growing business."

All in all, I was pretty impressed with what he's setting out to do and the opportunity not only from a business perspective from from an industry perspective as well. Hence the whole "going to work in his division" thing. Of course, "Thought Leadership" is one of the things Architecture Strategy works on very diligently, so in some ways this isn't as big a change as it might be. On the other hand, giving advice to people solving hard problems is a lot different than solving those hard problems yourself.

I'll be starting this new role pretty much immediately, so expect the less-than-usual blogging to continue for the time being. But my external visibility on my blog and presenting at conferences and executive briefings is one of the things they hired me for. So after I get my bearings things should be back to normal. At that point, I'll hopefully be able to talk in more specific terms about what we're tackling on my new team. I hope to shake things up quite a bit over there and deliver the play by play here on my blog.

Massive thanks to John deVadoss and the rest of the Architecture Strategy Team. Back when I started, I think Simon Guest was the only other blogger on the team. Now there are only three non-bloggers on the whole team. It's been a great three years and I've been a part of so many great accomplishments:

• Making Microsoft the first (I think!) global sponsor for the International Association of Software Architects
• Delivering the CRM Integration for Outlook Sample Application
• Launching the Solution Architect and Infrastructure Architect MVP programs
• Delivering the first-ever Architecture Track at Tech-Ed 2004
• Taking Arvindra's amazing JOURNAL The Architecture Journal worldwide
• Launching the Architecture Resource Center and the MSDN Solution Architecture Center websites
• Presenting at PDC, worldwide Tech-Eds, p&p Summits and a host of other events
• Meeting tons of customers in user group meetings, at executive briefings and in architecture design sessions

I always joke that if I ever left Microsoft, I wouldn't want to go work for another technical company. Now, I get the chance to go affect the business of a Fortune 50 business while not having to leave Microsoft. Pretty sweet.

See you on the other side.

June DSL CTP

Congrats to the team for their latest version of the DSL Toolkit, integrated into the June CTP of the VS SDK. According to the published product plans of the VS SDK, they're suppoesd to ship their next release - including the final DSL toolkit - next month. Looking forward to it.

Thinking About Object Models

I'm doing some experiments with Amazon's S3 Service. Very cool service, I might add. Anyway, the sample C# REST code basically wraps the network requests with a single connection class that has individual methods for each type of service interaction (list all my buckets, list all objects in a bucket, create a bucket, create an object, you get the idea).

However, S3's service is a natural hierarchy. The Service contains many Buckets, which in turn contain many Objects. So another way to wrap the service interaction is with a series of objects that are related to one another and only implement the service interactions relevant to that class. (Service would implement List My Buckets and perhaps Create Bucket. Bucket would implement List Objects and Delete Bucket. Again, you get the idea.)

For an interface as relatively simple as S3 (the SOAP interface has a grand total of 13 operations) it probably doesn't matter one way or the other. Furthermore, it's probably a question of personal preference. My question: What's your personal preference? A single object with many methods or a hierarchy of objects each with fewer methods?

Enterprise 2.0 ARCast

Ron just posted his latest ARCast featuring yours truly talking about Enterprise 2.0. Some of the same stuff I blogged about last month, but in a conversational style. Check it out.

Architect Connections Conference

For any readers in LA, I'm keynoting the MS Architect Connections conference in LA next week. They have a few spots still open, so if you're interested you can sign up here.

TechEd Iron Architect Contest

For a change, I'm not involved with TechEd at ALL this year. It's all Simon and Marty. Looks like they've got some cool stuff cooking, particularly the Iron Architect contest. Almost makes me wish I was going this year.

SPARK Weblog

In preperation for SPARK later this month, we (i.e. the Architecture Strategy Team) has set up a SPARK Blog. So far it's mostly links to a few people talking about SPARK, but it's also appears to be an opportunity to use the work "SPARK" whenever possible, such as "SPARKs Fly".

I'm just waiting for someone to blog about SPARKitecture. :)

Tim's Wannabe Five

I don't know how I missed this before, but Tim Bray blogged about not inventing XML languages over a month ago. This comment is right on the money: "The value of a markup language is proportional approximately to the square of the number of different software implementations that can process it." Conceptually, I agree with him - my primary argument against XSPF is that it has the same basic semantics as RSS, but RSS is much more widely used. But his list of the "big five" markup lanugages seems more like the "wannabe five". How many different siftware implementations process any of the things on his list?

I'm not arguing the technical quality of these lanugages - frankly I'm not that familiar with any of them but Atom. But if you're arguing the network effect, none of these formats Tim lists qualify. I'm sure he wants them to be popular, but wishing doesn't make it so.

Redundant Specifications

Yet Another AJAX Toolkit

Hot on the heels of my post about the AJAX toolkit spectrum comes news that Yahoo! has released their own AJAX toolkit as well as a very cool web page design pattern catalog.

Some non-surprises:

• The Yahoo! toolkit has it's own animation library, it's own drag and drop library and it's own XmlHttpRequest wrapper, just like other libraries like Dojo and prototype.
• The Yahoo! UI Controls depend on Yahoo's Core Utilities. The libraries are modular, but not polymorphic with other these libraries.
• The code was released under an open source license (BSD).

All these different implementations of the same core set of capabilities at best means learning lots of ways of doing the same thing and at worst means incompatibility between components from different libraries.

BTW, last week I wrote that using a text-based high-level scripting language like Javascript in the browser "encourages business models where the in-browser code has little if any value." This release from Yahoo! supports that point. While their mapping component is subject a strict Terms of Use, their UI components were released with a liberal open source licence. I doubt that's a coincidence.

Thoughts on the AJAX Toolkit Spectrum

Last week, Dion wrote about the spectrum of AJAX tookits. He ended with a question, wondering which end of the spectrum will dominate? Will it be lightweight composable toolkits like prototype, script.aculo.us or Dojo? Or a more comprehensive toolkit like Atlas?

This came up in a chat I had w/ Jimmy Nilsson today. Well, not specifically about AJAX toolkits. Rather, we were talking about what he called technicalities vs. semantics:

I have noticed that there seems to be a focus first on the technicalities and then on the semantics. Take Indigo (or WCF) for example. There has been sooo much talk about its technical aspects, but very little talk about how to design for it. I'm pretty sure that when the technicalities have been sorted, it's time for the semantic side. I'm not thinking about technical semantics, but rather business semantics.

On more than one occasion, I've had a head-beating-wall conversations with WCF folks who are completely obsessed with the secure, reliable and transactional delivery of messages, but have given exactly zero thought to the actual contents of said message. So I know where Jimmy is coming from.

With respect to AJAX toolkits, the question becomes just how easy will these lightweight toolkits compose? Because while Dion describes Google Maps as "a simple JavaScript include", that's just the technicalities, it doesn't begin to deal with the semantics. For example, Dojo has Dictionary object, prototype has a Hash object. Dojo extends the Javascript Array, so does prototype. Both libraries wrap the XmlHttpRequest object. In each of these cases, it appears to me that the library authors have focused on the technicalities, but not thought about the semantics. These implementations are all semantically similar, but incompatible. So I don't buy that these lightweight toolkits will compose well. What do I do if I'm using prototype but want the rich text editor in Dojo?

The network effect that Dion doesn't consider is the component ecosystem phenomenon that Microsoft has a ton of experience with. Old school VB, COM/ActiveX and .NET have all had large ecosystems of components and controls evolve that extend the functionality of the baseline development platform. There's no reason to believe that won't happen with Atlas. I think it's wrong to describe Atlas as a monolith or self-contained or enclosing. It's an extensible baseline platform - i.e. the baseline functionality is set down once at the development platform and the ecosystem can extend it from there. Sure, overlapping extensions happen (how many rich text editor components are there for ASP.NET?) but at least they all have basic compatibility.

Update - Fixed link to Dojo Toolkit in the first paragraph.

Web 2.0 Evolution

For example, I think we can generalize about the bright future of peer-to-peer based technologies from looking at systems like Skype and FolderShare. Naturally, with the power shifting to the edge, I believe it's inevitable for more edge machines to communicate directly with each other rather than being mediated by a service in the center. In fact, in many cases I believe were going to want to shift a significant percentage of social computing to the peer-to-peer model. It scales better and doesn't have centralized privacy concerns. Furthermore, I think there may be be specific peer-to-peer capabilities that are difficult or impossible to replicate with a centralized model, though so far, I haven't them yet.

However, I'm not sure we can generalize about the future of mashups the same way. This isn't to say I think mashups are going away - far from it. I just think that mashups a year from now will look very different than they do today.

First off, I don't think we can  generalize the success of Google Maps. In the Programmable Web how to guide, they mention that "Plotting markers on maps is probably the easiest place to start". Apparently, many people are taking that advice because 297 of the 411 mashups listed use one of the three major (i.e. GYM) mapping services. However, maps are unique because of the massive amount of data, the extremely simple API and the ubiquity of location information. They are also one of the few mashup API's that runs in the browser - the vast majority of mashup API's are back end data type services like Amazon's E-Commerce Service. How many more in-browser mashup API's are out there waiting to be built? I'm not sure, but as I wrote in Browser as VM, the problem with these in-browser mashup API's is that you can't protect your IP.

As for back-end service mashup APIs, there needs to be a way for these service providers to make money. Even if the software they use to build the service is free, things like hardware and bandwidth are not. For an Amazon or Ebay, making money on thier services is relatively easy since they are facilitating sales transactions. In the end, they probably won't care much if a sales transaction originated on their site or on a site leveraging their APIs. However, if the service provider is ad-funded, the service API effectively routes around the site's revenue mechanism. Take, for example, a site for tracking events like Zvents, Eventful or Upcoming. They need to drive users to the actual site in order to drive revenue. So it remains to be seen exactly how the API based access is going to work out. Today, these API's are specifically provided for "non-commercial use only" so one way would be to charge for access via the API (either flat-rate subscription, a per-use charge or a combination of the two). Alternatively, they could be bought up by a larger company who could then afford to run the business at a loss. Yahoo already bought Upcoming and Google Base already has an event item type, but the other big companies in this space (I'd guess Microsoft, Amazon, Ebay and maybe Apple) might be interested. Again, I'm not sure how this evolves either, but it's got to evolve beyond "non-commercial access".

Latest Architecture Journal

It's been a while coming, but the print subscriptions of The Architecture Journal are underway. I got my copy of Journal 6 in the mail today. It's not online yet, but you can get the back issues and sign up for your own free print subscription on the website.

SPARK is Out of the Bag

As part of the new job, I'm involved in the planning a workshop called SPARK, which Dion Hinchcliffe blogged about this morning. (Dion also writes a blog here - bringing the total to three - so I created a combined feed just to keep track of all the places he writes). My new boss Mike also mentioned SPARK this morning. In the hopes of sparking futher interest (pun intended), here's the overview of SPARK:

SPARK is the first in a series of high-level forums hosted by Microsoft that use a workshop setting to examine “the issues that matter most” in the practice of strategic architecture and produce guidance for the industry as a whole.

Today, new social movements, advances in technology, and forces within business are overlapping to create a landscape glutted with challenges and opportunities. In many cases, these forces have driven the deployment of new technologies and the adoption of new behaviors, adding multiple layers to an already complex set of issues that must be navigated. Architects are searching for a solution that helps manage this complexity.

SOA, Software as a Service, Web 2.0, and Edge are all elements of the solution, but are they the complete picture? Are they a sufficient answer to the issues?  Can they be used together in a productive and efficient fashion? What matters most?

New DSL Toolkit Drop

I have been so focused on Web 2.0 stuff that I've been reading a bunch of new blogs and disregarding the old ones I used to read. So I didn't realize until today that the DSL Tools team released a new drop last week. According to Gareth, the highlights include:

• Integration into the Visual Studio SDK. According to the site, they are shooting for an April release for v2 of the VS2005 SDK, so does that mean the DSL Tools will be done in April?
• Single file format and complete visual designer for all aspects of a DSL. I'm guessing this mean we no longer have to edit the designer definition by hand. That's a good thing. But I liked the seperation of domain model and designer, so I'll be interested to see how I like what they've built.
• Domain-specifiic model serialization. This is huge - previously, the domain model dictated the XML serialization format. Now, if you can customize this, you can provide a clean model syntax and even possibly read in other syntaxes as well
• Port Shapes and a revised modeling API

Update: Apparently, I can't read. Only the VS SDK integration is done in this build. improvements to the file format and model serialization will be in the next drop.

Flash, the Other White Meat

When I wrote yesterday about the Browser as VM, I made the point that extensibility is difficult as we have four major browsers and multiple OSes to deal with. What Web 2.0 company is going to be willing to bet on a proprietary extension implemented in only one of those combinations? Not many if any I would guess. However, there is one option that works across all those browsers and OSes: Macromedia (now Adobe) Flash Player.

Unlike the browser, where AJAX is a relatively new idea, Flash has been positioning itself as a platform for nearly four years. Instead of AJAX, Macromedia coined the term Rich Internet Application or RIA. RIAs share a lot in common with AJAX in that they are downloaded on demand, execute arbitrary script code and can retrieve data across the network. But the most interesting commonality that Flash has with the browser is that runs across multiple browsers and OSes.

In platform portability, Flash has succeeded where Java failed. I haven't done enough research to know exactly why yet, but I suspect that it's because Java tried to be a complete portable environment on day one where Flash focused on specific functionality that weren't possible any other way - so called "skip intros" - and grew up from there. In other words, Java tried the top-down approach and Flash tried the bottom-up approach. I'm not surprised bottom-up worked and top-down crashed and burned.

While the modern browser has evolved to make it a capable platform, it still lacks some capabilities that Flash has. Most notably support for rich media. Thus, sites like Pandora, Google Video and MTV Overdrive need the capabilities provided by Flash.

While it's hard to imagine enhancements to the browser due to the difficulties across four browsers and multiple OSes, improvements to Flash are easy to imagine. According to Macromedia, Flash has 98% penetration. Even more impressive is that Flash reaches 80% penetration with new versions of the player within 12 months.

Check out this post from Kevin Lynch for more on Flash for Web 2.0 companies. So far, the only Web 2.0 company I know about (which is to say I'm sure there are more out there) is Goowy. Which ones am I missing?

Browser as Virtual Machine

Note: this is the first in a series of Web 2.0 entries. I know I’m on record as hating the term Web 2.0, but as I wrote in that post, I do belief there is a fundamental shift underway in computing. The industry is calling this Web 2.0, and I can either spit in the wind or go with the flow. Furthermore, for the more Web 2.0 savvy among my readership, much of what I write about in this series may be old news. But I want to blog what I learn as I learn it, so bear with me.

Just as the dumb terminal was eventually replaced with more sophisticated personal computers, the dumb browser has been replaced on the modern desktop by something significantly more versatile. When the ability to process arbitrary script code was added to the browser, it became a virtual machine in its own right. Perhaps unique and special-purpose when compared to environments such as the .NET CLR, but a VM all the same. And while its unique nature makes the browser unusable for entire genres of applications – you’d never use the browser to build a server application for example – it makes it well tailored for user-centric, software as a service style applications that have become commonplace. While the browser’s scripting capabilities have been around since the mid 90s, the industry has only recently started to leverage those capabilities to build applications that run on the client inside the browser. Jesse James Garrett coined the term “AJAX” – Asynchronous JavaScript and XML – to describe this style of application.

If the browser is a virtual machine, that makes JavaScript the “assembly language” of the browser. That is, JavaScript is the lowest level of abstraction you can program the browser with.  This has pretty dramatic implications on the applications you build for the browser VM. For one, JavaScript is at a sufficiently high level of abstraction that you can use it directly and be productive. Writing an entire application in IL or Java byte code is unthinkable, but isn’t really a big deal for JavaScript. Furthermore, Because JavaScript is a text-based scripting language, protecting your code as intellectual property is extremely difficult. While obfuscators exist, in the end they can only delay the reverse engineering of your code, not prevent it. This encourages business models where the in-browser code has little if any value.

For example, the big mashup functionality these days is mapping. There are three big mapping services out there: Google Maps, Microsoft Virtual Earth and Yahoo! Maps. 266 of the 368 mashups listed on ProgrammableWeb as I write this include mapping functionality from one of those services. That's nearly three out of four. Mapping is interesting because of the sheer amount of data involved. In fact, the code is pretty useless without the back-end data. So while I can get the code for Google Maps, it does me no good without access to the data for which I need the API key. Contrast this with the complete lack of market for browser-based rich text editors. Sure, there are various open-source script libraries like Dojo, Web Wiz RTE and Kevin Roth's RTE. But no companies offering a rich text editor service like they offer map services. Why is that? I would think the value of rich text editing would be even more widely applicable than mapping. The problem is that, unlike the map service, there's no back end associated with a rich text editor. There's no way to protect a client-side-only solution such as these rich text editors. The only people who do sell rich text editor components are ones who have integrated into some back-end programming environment such as Richer Components' RichTextBox for ASP.NET.

The browser as a VM also has broad implications with regard to extensibility. Similar capabilities are delivered by the four major browsers (IE, Firefox, Opera and Safari) across the major operating systems (Windows, MacOS, Linux, FreeBSD). So the question is, how will new capability evolve in the browser? Will the growing number of Web 2.0 companies looking to provide compelling features and differentiate themselves in the marketplace demand new functionality in the browser VM? Will one of the browser vendors be willing to take the heat of building proprietary extensions to their browser? I realize that many people have a dim view of proprietary extensions, but many features we take for granted today are de facto standards that arose from Microsoft's proprietary extensions to IE. Most notable of these of course is XMLHttpRequest, without which "AJAX" would just be "J". And JavaScript itself started life as a proprietary extension to Netscape before eventually being turned over to ECMA for standardization.

Developer 2.0 ARCast

FYI, I sat down for an ARCast with Ron Jacobs last month to talk about Developer 2.0. It's not he full presentation I did at the p&p summit, but it covers much of the same ground. Catch the show over on Channel 9.

Developer 2.0 at VSLive!

FYI, for those going to VSLive! in San Francisco, I'm a last minute addition to the schedule. I'm presenting a talk titled Developer 2.0: Finding Your Way in the Future of Software Development. I wrote and delivered this presentation originally for the patterns & practices Summit back in December. It was the second-highest rated talk of the summit (after Anders' LINQ keynote) so I'm excited to be delivering it again. I'm hoping to get a high-quality recording so I can publish it (I recorded the p&p summit version with my laptop. You can hear it but I wouldn't say it's "high quality"). The session is at 5:45pm on Tuesday. I've been told it's in room 2016/2018, but you should double check when you get there if you're interested in going.

There's a solid showing from the Architecture Strategy Team at the VSLive! Software Architecture Summit this year. In addition to my last minute addition:

• John deVadoss's day two keynote on Software Architecture on the Edge: Moving Beyond SOA.
• Ron Jacobs on Integration, Service Orientation and Pattern Driven Design .
• John Evdemon on Workflow Enabled Services.
• Mauro Regio on Collaboration Software Factory.

See you in San Fran!

Hating the Term Web 2.0

Now that I’m an Architect on the Edge (I’m thinking of putting that on my business card. Good idea or bad idea?) of course the first order of business is taking a closer look at “Web 2.0”. One thing leaps out at me right away – I hate the name “Web 2.0”.

First off, it’s a pure marketing buzzword. It was originally coined as a conference name. In a way, the fact that is has no underlying meaning is a good thing, because it gives people argue whether it really exists or not. In a way, it’s like the word “multimedia” back when we were first putting CD-ROMs into computers. There used to be lots of discussion if one thing or another truly was “multimedia”. Now, we don’t really worry about categorizing it as the marketing buzz around the term is long gone.

Secondly, I think it’s wildly arrogant to claim we’re only on version 2.0. The Internet has been around for 36 years. So everything before mid 2004 was Web 1.0 or earlier? And people are already talking about Web 3.0. Come on, let’s get real. The technologies that are driving the current revolution have been percolating for more than one major version of the underlying technology.

Finally, what’s with the version number anyway? One of the core principles Tim O’Reilly outlined was the “End of the Software Release Cycle”. Why are we using a holdover from the software release cycle days to indicate the end of the software release cycle?

Don’t get me wrong, I strongly believe that there is dramatic change happening in this industry. The way I explain my new job is to consider that one of the most basic axioms of distributed computing has been overturned.

From day one, all the computing power has been focused in the center. At first, the machines on the edge had no power at all – they were just dumb terminals. Slowly but surely, those machines at the edge started to become powerful in their own right. However, it’s only in the last seven to ten years that commodity hardware that was pervasive on the edge grew powerful enough to power the center. And it’s only in the last two or three years that the connection between the center and the edge grew fast and pervasive enough to make that edge power relevant.

The rules have changed. The power has shifted from the center to the edge. And we’re only just beginning to see the effects.

Maybe we should call it WebNT? :)

Architect on the Edge

So for the fourth time in seven months, I have a new manager. Way way way back and the end of June, I left marketing to be a solution architect. In doing so, I traded in Norman for John as my manager. Things were looking shiny but then Microsoft had a major reorg back in September. These big reorgs often cause small ripples, like the director of Architecture Strategy deciding to move back to a product group. John was promoted to head the team and Gianpaolo was hired to take over John's previous role as head of solution architecture. Finally, it seemed like things were calming down on the management front, so I was caught off-guard by what happened next.

Last Monday, John offered me a chance to switch roles. We have spun up an Edge Architecture team to focus on the architectural impact of the next generation of computing - what some call "Web 2.0" - and he thought I would be a perfect fit for the team. I agreed and jumped at the chance. So now I work for Michael Platt. Don't let the infrastructure focus of Michael's blog fool you - he incredibly deep on Edge Architecture and I am currently playing catch-up. The strangest thing so far is having to re-orient my thinking for consumer focused systems and away from the enterprise world where I have I've gotten the vast majority of my experience and where I have spent my entire Microsoft career to date.

In the immediate future, I'm been dumped into the MIX06 planning process. We have other stuff going on in the MIX timeframe that I'll get to soon enough. I'm also re-thinking my personal presence. As my career progresses, the moniker "DevHawk" seems more and more outdated. Is it?

CNET Picks Up Outlook Integration Sample News

I couldn't blog about the Outlook Integration Sample last week because we were waiting to see if the mainstream media would pick up the news. CNET ran an article about the sample today. There's also a link to the Customer Explorer case study that was the original inspiration for the Outlook Integration Sample and VSTO Outlook in the first place.

Evolving Language Fidelity

I haven't seen much in the way of response to my Hi-Fi Models post, but I did come across this great article by Ted Neward on the history of the tumultuous marriage of objects and relational databases, primarily in the context of LINQ. In the context of Code is Model, the following passage from the summary was the most interesting:

While Project LINQ doesn't purport to be the "final answer" to all of the world's object-relational mismatch problems, it does represent a significant shift in direction to solving the problem; instead of taking an approach that centers around code generation, or automated mapping based around metadata and type inference, both of which are exercises in slaving the relational model to an object-oriented one, Project LINQ instead chooses to elevate relations and queries as a first-class concept within language semantics and library-based extensions.
[Comparing LINQ and Its Contemporaries - Ted Neward]

When Ted says relations and queries are elevated to "first class concepts" within the language, it makes me think of Stuart's comment about language fidelity. I'm not sure I would say C# 3.0 is at a higher level of abstraction than 2.0, but I would say that the inclusion of these new abstractions does improve the language's fidelity. This fidelity improvement does come at the cost of complexity (TANSTAAFL) but compared to the current alternatives, I'm willing to pay that price.

The problem with increasing the language fidelity like this is dealing with the outdated code it leaves behind. You see this today with the addition of generics in the 2.0 CLR. How many hand-coded or generated strongly typed collections are floating around out there from the 1.x days? Lots. (As if 1.x was so long ago!) How much database access code is floating around out there today? An astronomical amount. Every app that touches a database or processes XML will be outdated with the arrival of C# 3.0 and VB 9.0. But the price of converting this outdated code to use the new abstractions probably won't be worth the time or risk. That means you're left with maintaining the outdated code while also writing any new functionality with the new language features.

I wonder how DSLs will be impacted by this evolving language fidelity issue? On the one hand, the nature of DSLs is that they have much narrower usage (i.e. one domain) than something like generics or LINQ. On the other hand, I expect DSLs to evolve faster than general mainstream languages like C# can. So I'm thinking the impact will be about the same.

Outlook Integration Sample

For the past few months, I've been heavily involved in a project but I wasn't allowed to blog about it. Last week, it went live on MSDN so finally the gag is off.

About a year ago, word started to surface about something called Project Elixir which aimed to integrate back end CRM systems with Microsoft Outlook. Part of that effort resulted in the addition of Outlook Managed Add-ins to Visual Studio 2005 Tools for Office. However, the VSTO team's primary deliverable was an add-in loader that enforced security, enabled shutdown unloading and provided a better startup/shutdown developer experience that IDTExtensibility2. (Check out the VSTO Outlook Architecture document for more details.) While those are important fundamentals that needed to be gotten right, VSTO Outlook doesn't provide much in the way of tools or guidance for building Outlook add-ins that leverage managed forms and controls or integrate with your back end systems. That's where the CRM Integration for Outlook sample comes in.

What we've built is a sample application that surfaces CRM style data inside of Outlook. Outlook is the natural home for your calendar and your personal contacts. Why not make it the natural home for your customer contacts, activities and opportunities as well? As part of the demo project we've implemented:

• Using Windows Forms for editing custom items. Check out this screenshot. The Activity form is a standard managed Windows Forms form, not an Outlook custom form.
• Using a Windows Forms user control as a folder home page. Here's a screenshot of the "CRM Today" page. Again, that's a standard managed Windows Forms user control.
• A framework for adding menu items and toolbars. In Outlook, the developer has to manage adding the custom toolbars and menu to each explorer and inspector window themselves. With our sample, we built a framework to handle that for you.
• Using SQL Express as a local cache of CRM data. It turns out that for many scenarios, storing a copy of all the back-end data directly in Outlook is a bad idea. First, it increases the size of the users mailbox, requiring more storage on the Exchange server. Furthermore, any custom data in Outlook has to be synced twice - once from the back end system to Outlook on the desktop, then from Outlook back to Exchange. By minimizing the amount of back-end data stored in Outlook proper, we reduce the mailbox size and sync bandwidth needs. In both the above screenshots, the displayed data is coming out of the local SQL Express instance, not Outlook.
• Having two separate storage locations (Outlook & SQL Express) means having to sync between them. We've built a local sync engine that can sync both individual items between Outlook and SQL Express as well as a collection of items between SQL Express and a given Outlook folder.
• Finally, there are some utility classes to make it easier to deal with Outlook folders and items. Of primary note is the ItemAdapter class which provides a pseudo base class for Outlook items (appointments, emails, tasks, etc). Those items all have a set of similar properties and methods, but don't have a common base class so they can't be treated polymorphicaly. ItemAdapter uses runtime reflection to implement those common operations without needing to cast to the concrete Outlook item type.

Check out the Architecture Design Guide, as well as the Outlook Customization Guide and the Local Sync Engine Guide up on the Solution Architecture Center. You can also pick up the source code. Also, I spun up a GDN Workspace so we can have a discussion forum and to track bugs and requests.

Going forward, I'm going to be focusing on the remote data sync story for this scenario. Among other responsibilities, I "own" the Data pillar of our Connected Systems model so this dovetails nicely. You'll note above that while we have a local sync engine in the sample, we don't have any way to move the data back and forth between the local copy in SQL Express and the remote copy in the CRM back-end. We are working on some guidance around this right now, but we didn't want to hold up publishing the rest of the sample.

Frankly, it's been nice to be involved with something so technical after spending time on the marketing team. I'm pretty proud of the project and I look forward to your feedback.

Update: Removed the link to the running demo as it's been taken off the download site for reasons I am not aware of. If you want the binary and you don't know how to compile it, drop me a mail.

My New Boss is Blogging

Gianpaolo took over the Solution Architecture team a few months ago and he rebooted his blog a couple of weeks ago. And he's active on the afore mentioned Architecture Forums. Subscribed.

Architecture Forums

Simon just emailed a bunch of internal architects about the new Architecture Forums on Microsoft.com. So far there's a general architecture forum as well as one for modeling and tools, with more to come I would guess.

What other forums do you think we need?

Hi-Fi Models

I'm slowly but surely working through my holiday backlog of email and weblogs. Slowly being the operative word here.

Anyway, Stuart has a great post on the process by which we build models. And he's not talking theoretically here, he's working on a model for the designer definition file in the DSL toolkit. (Which is good news in and of itself as hand-writing the XML dsldd file is a pain in the butt. Though until then there's the great Dm2Dd tool from Modelisoft). The iterative process he describes certainly looks a lot like development, in the same way that C# development looks like C development. Similar steps taken on different concepts. Additionally, he's working bottom up - the output of the model will eventually be a working program (a designer in this case) which is the point I made in Abstraction Gap Leapfrog. There are existing abstractions that work now (i.e. the code generated from the existing dsldd file) and he's trying to building something one level up from there.

I also like Stuart's use of "fidelity" instead of my use of "complete". Stuart uses it as an indication of how correct a given model is. That's what I was implying when I said "complete" but "fidelity" captures the idea much better. I could imagine both lo-fidelity and hi-fidelity models for a given domain, though I would imagine you would always want to use the highest fidelity model available. The difference would be the amount of custom code you have to write - the higher the model fidelity, the less code you write by hand. And I would imagine the model's fidelity would evolve over time, which introduces interesting questions regarding language evolution as well as the evolution of projects built with those languages.

I hope Stuart keeps blogging about this project.

Abstraction Gap Leapfrog

One of the cool things about having a blogging conversation with someone on the other side of the world is that while you sleep they are thinking of a good response to your post. The only downside? Having to deal with rampant misspelling like "artefacts". :)

Anyway, Gareth responds to my post:

Until we get models that are perfectly aligned with our business domains, we'll have people who want to create models but who get them slightly wrong from a precision point of view - usually in the places where the imperfect models interact with other aspects of the system across or down the abstraction stack.

With code, you'd likely not want to have people check in sources that don't even compile and then hand them off to other folks who do make them compile, but I think that's exactly the type of process we'll see emerging in modelling for a while. I feel this way because I don't foresee us getting modelling languages of pure business intent 100% right for some time yet - we're simply not close enough to formal enough descriptions of systems as intensely human as a business yet. However, I hope we won't want to try and keep modelling as locked away with the techies as traditional development has been. (Hope I'm not talking myself out of a job here…)

[Pseudomodels and intent]

I keep saying incomplete and Gareth keeps saying imprecise, but I think we can both agree on the term "imperfect". There's a massive difference between having an precise language that is imperfect versus a language that is inherently imprecise like UML.

However, I think the primary disconnect here has to do with Gareth and my views on how higher abstracted languages will evolve. Gareth's comments about modeling "pure business intent", having "models that are perfectly aligned with our business domains" and not "keep[ing] modelling as locked away with the techies" imply to me that Gareth wants to work down from the high level business abstractions into implementable technical abstractions. Frankly, I don't think that's very likely. Leapfrogging a few levels of abstraction hasn't worked well in the past (CASE and UML/MDA) and I don't think it will work well now.

I find it much more likely that we will build higher level abstractions directly on top of existing abstractions. Again, this is similar to the way C++ built on C which in turn built on ASM. Sure, that could keep modeling "locked away with the techies" for a while, but we're already beginning to see the light at the end of that tunnel. Windows Workflow Foundation is a significant leap in abstraction while also being something than non-techies can use. Reports about about Sharepoint "12" embedding the WF engine and FrontPage "12" providing a Workflow Designer for building SharePoint workflows. While I imagine (and I haven't used any of the new Office "12" suite so this is pure conjecture) these WF tools are targeting the "power user", they certainly aren't only for developers.

Believe me, I would love to be wrong about this. I would much rather work down from or business user intent than up from the technical foundation. I just don't think it's feasible. The process Gareth describes breaks the "Model Transformation must be Deterministic" tenet of Code is Model, though the word "must" may be to strong to allow for language evolution.

Imprecise vs. Incomplete

Gareth responds to the first tenet of Code is Model:

[A]lthough as an industry we desperately need to drag models kicking and screaming from the far left of pretty-picturedom a good long way to the right in the direction of precision, I don't want to throw the baby out with the bathwater.

I'm going to take it as a given that folks believe that precise models are valuable development artefacts. Why do I think imprecise models are also valuable? Here are three things that tools for imprecise models help you to do:

1. Communicate with people about design
2. Think out loud in a way that's more shareable that your whiteboard
3. Start with an imprecise model and progress gradually toward precise models

Hopefully the first of these is obvious - there is value in model as communication device - it's just not enormous.  I've talked before about the value of the second - I draw pictures on my whiteboard and when I'm on a conference call to Redmond, they're effectively useless.

The third is something I've only recently become a convert to.  I'm happy to have models which are not precise so long as I can still reason about them programmatically.  This allows me to have development processes that are about a quantitive process of iterative refinement.

Here's an example - in some infrastructure modelling tool, I have a node type which specifies a logical machine group.  One of its properties is the number of actual machines required to suit the proposed scale of the application to be deployed.  I'd like to be able to put "Depends on outcome of Fred's scalability investigation" into that numeric field, or perhaps "4->8".  I can still generate a pretty good report from this model, but I can't really provision a set of physical servers from it.

But here's the kicker - it's vital that I can write tools that programmatically assess this model and tell me what work needs to be done in order to make it precise.  I want to know exactly what must be done on this model before, for example, it is suitable for feeding into some kind of provisioning tool.  You might say it needs to be precisely imprecise; I prefer to think of it as quantifiable imprecision.

[Imprecise Models and Killing Hippies]

The only thing I disagree with Gareth about is terminology. Like the term architecture, model has become a catch-all for things that aren't code. Regular readers of this blog know I like to be more precise than that. As such, I think items #1 and #2 from Gareth's list aren't actually models at all. I think of them as pseudomodels, similar to the concept of pseudocode. Actually, I like the name pseudomodel - it also applies well to Grady's scaffolding. Like psuedocode, pseudomodels have tons of value in communication and reasoning about a problem but they can't be used as development artifacts.

As for the third, I think what Gareth is describing is an incomplete model, rather than an imprecise one. If the model is imprecise, there's no way to programmatically reason about it. But if we look to code as an example, obviously, there are many cases where code is incomplete. Every compiler error you've ever seen is an example of incomplete code. And because the language is precisely specified, the compiler can tell you what needs to be done in order to make it precise, exactly as Gareth requested. I don't think of writing code as "progressing gradually towards precision" and I doubt anyone else does either. And while I do see development as an "iterative process", I don't think of it as "iterative refinement". Modeling shouldn't be any different.

One area where I do see refinement being critical is in the development of the modeling language itself. Traditionally, the language stays stable while the program written with it changes. But with the introduction of DSLs, it becomes possible for both to vary independently. I would assume that a DSL would evolve over time to have better "coverage" of a given domain. For example, if I was building a CAB DSL, I would implement support for WorkItem right off the bat, but supporting WorkItemExtension would be much lower on the priority list. This represents language refinement, but I would argue it's a refinement of coverage not a refinement of precision.

Scaffolding Isn't a Model

Grady Booch @ the OOPSLA 05 Structured Design and Modern Software Practices Panel at (according to the transcript)

The most important part is the executable code. I typically throw my models away, but I always save my source code. I design because I need abstractions to help me reason out my projects.

Grady Booch ranting on his blog:

It's sad how one can be misquoted and then for that misquote to be picked up by someone else with both then making a spin of the events to support their position. How silly is that.

Juha-Pekka Tovanen quoted me from my OOPSLA panel as saying that "when the project gets closer to the delivery you normally throw away UML models."

<snip>

Let me be excruciatingly clear: Over the years I have been consistent in saying that in a) the most important artifact of any software development organization is executable code and yet b) modeling is essential in constructing such executables. This is because c) models help us reason about, specify, construct, and document software-intensive systems at levels of abstraction that transcend source code (and the UML is the accepted open standard for doing so). That being said, it is a pragmatic reality that d) some models are essential (and should be retained) while others are simply scaffolding (and should be discarded). I have never said and do not say now that one should throw away all models, as Juha-Pekka then Harry then Steve imply.

First off, Grady claiming to be misquoted his pretty disingenuous. Juha-Pekka's account of what Grady said on the panel is pretty spot on. Furthermore, Grady claming that I implied he said "throw away all models" is also disingenuous. I specifically wrote that I thought Grady was being taken out of context:

I've gotta believe that this comment was somehow taken out of context and that the Grand Poobah of the Common Semantic Model doesn't actually believe that tossing the model at the end of the project is a good thing.

But he-said/she-said nitpicking aside, where's the guidance on which models are essential and which are "simply scaffolding". Last time I checked the UML spec, none of the models are labeled as disposable. How about Rational Rose? Any dialog boxes that pop up reading: "Don't worry keeping this model up to date, it's just scaffolding"? I don't think so.

Obviously, working at a higher level of abstraction helps you reason about a project. But reasoning at high levels of abstraction doesn't mean you're modeling. When a building architect sits down with a prospective customer and a sketchpad, they may be working out great ideas but no one is going to call the result a blueprint. Grady's scaffolding "models" break nearly every tenant of Code is Model. Scaffolding isn't precise or deterministic. And if it ends up in the recycle bin, I guess it's not intrinsic to the development process.

Actually, it's good that scaffolding isn't a model. It means Grady is specifically not suggesting to throw away models. He just needs to get his terminology right.

As for Grady's request for "just one DSM out there in production", Don lists a few: Workflow languages (XLANG and WF), Business Rules Engine Languages and Build lanugages (MSBuild, Ant and NAnt). Juha-Pekka pointed to this list of DSM case studies. I'd also add UI Designers such as the Windows Forms and ASP.NET designers in Visual Studio. In the Window Forms case, the code is stored in a seperate file (yay partial classes) and are specifically marked: "do not modify the contents of this method with the code editor." In the ASP.NET case, the code for an ASPX file isn't even generated until runtime. And how about HTML itself? I'm thinking HTML qualifies as "in production"?

JavaScript Utilities

Can you tell it's a slow day in the office? :)

Speaking of raising the level of abstraction as well as browser based applications, check out Nikhil's JavaScript Utilities project:

The project introduces the notion of .jsx (extended JavaScript) and .jsa (JavaScript assembly) files. JSX files provide the ability to include conditional code via familiar preprocessor directives such as #if, #else, #endif and so on...The tool processes these directives in order to produce a standard .js file. JSA files build on top of .jsx files. While they can include normal JavaScript and preprocessor directives, they are primarily there for including individual .jsx and .js files via #include directives. This allows you to split your script into more manageable individual chunks.

Now, that's not raising the level of abstraction much, but here's another example of working in a higher abstracted environment (jsx and jsa) and then compiling down to something the underlying platform can execute (js). Nikhil provides three ways of doing this compliation:

1. A set of standalone tools that output standard .js files that you can then deploy to your web site. Command line parameters are used to control the behavior of the tools.
2. A couple of IHttpHandler implementations that allow you to dynamically convert your code into standard .js files. This is the mode where you can benefit from implementing per-browser code in a conditional manner. AppSetting values in configuration are used to control the conversion behavior.
3. As a script project in VS using an msbuild project along with an msbuild task that comes along with the project. MSBuild project properties are used to control the conversion behavior.

If you're going to raise the level of abstraction to do implement a preprocessor, you could also go all the way and implement an entirely new language that gets compiled down to JavaScript for execution in the browser. For example, I'm not as familiar or comfortable with JavaScript's prototype based approach. But I could imagine a more class based language that compiles to JavaScript. That's the same way early C++ compilers worked - they were a preprocessor pass that converted the C++ into C, which could then be compiled with traditional C compilers.

I wonder what JavaScript++ would look like

As Simple as Possible, But No Simpler

Chris Bilson left the following comment to my Thoughts on CAB post

I hesitate to agree that raising the abstraction level of tools is a good idea. That's just hiding the complexity that's already there inside of more complexity. If you ever need to look under the hood, it's even harder to grok. I think it would be better to go the other way. Try removing stuff to combat complexity.

Given that the software industry has been raising the level of abstraction of tools since the start, I found this comment surprising. Assuming Chris doesn't write in assembly code, he's leveraging something at a higher level of abstraction that's "just hiding the complexity that's already there". As I wrote in Code is Model:

The only code that the CPU can understand is machine code. But nobody wants to write and debug all their code using 0’s and 1’s. So we move up a level of abstraction and use a language that humans can read more easily and that can be automatically translated (i.e. compiled) into machine code the CPU can execute. The simplest step above machine code is assembly language. But ASM isn't particularly productive so work with, so the industry has continuously raised the level of abstraction in the languages they use. C is a higher level of abstraction than ASM, adding concepts like types and functions. C++ is a higher level of abstraction than C, adding concepts like classes and inheritance. Each of these levels of abstraction presents a new model of the execution environment with new features that make programming more productive (and sometimes more portable). [emphasis added]

I feel like Chris has mischaracterized what I wrote. Here it is again:

If you can't lower the complexity of your framework, it's time to raise the abstraction of your tools.

Note I'm not advocating raising the level of abstraction for abstractions sake. Believe me, I'm hate overly complex code. The project I've been on (and can blog about soon I hope) had a tone of overly complex code that wasn't particularly germane to solving the problem. I yanked all that code and have reduced the framework to a quarter it's original size while adding functionality. But the reality is, simplification can't always be achieved by removing code. To paraphrase Albert Einstein, solutions to problems should be as simple as possible, but no simpler.

CAB is the simplest solution to the problem it addresses, and no simpler. Since we can't make CAB simpler and still solve the problem, the only alternative we have is to have the tools hide that complexity. Given how well this has worked in the past, I see no reason why it can't work again in the future.

He's Bad at Taking Advice, but Man is He Smart

He in this case is Tim Mallalieu from the data programmability team. He and I used to work on the .NET Adoption Team in the field together before we both moved to campus. Tim, if I recall correctly, was the first outside hire we made on that team. We used to constantly argue about who was the second smartest on the team. He thought it was me and I thought it was him. We both agreed the Jeromy was the smartest.

Anyway, after literally years of haranguing, Tim finally started a blog. But he's still not really listening to me. I told him he should start with a series of smaller posts and he didn't. 

I'm back working with Tim in a fashion as both my team and his team try and get a handle on how data is going to evolve as we move to service orientation. Most of the existing data paradigms and semantics such as table joins and transactions assume everything is together in one database. Obviously, as we break down large systems into smaller services, having everything in one database just isn't practical.

Browser Based Applications in the Enterprise

I asked Scoble for his thoughts on my post about the term mashup, and he decided simply to link to it. While I appreciate the traffic, I'd appreciate opinions even more. Luckily, in the same post he linked to Sam Ramji writing about what he called "enterprise mashups". Again, love the idea but hate the name mashup.

According to Sam, Scoble said:

[W]hat's to prevent mash-ups from being the main way that departmental apps are built in most enterprises 3-4 years from now?

Again, it depends on the definition of the term "mashup". Are we're talking about browser component based apps that leverage what Scoble calls ICC's? If so, Scoble is only wrong about the timeframe. My team has a browser based application leveraging Virtual Earth running on our intranet server right now.

Quick terminology sidebar: I use the term browser based application to refer to these AJAX and mashup style apps, since they download code and run in the browser itself. By comparison, I use the term web based application for the more traditional browser apps that did all the processing on the server and used the browser essentially as a dumb terminal.

While I think these browser based applications will go mainstream in the enterprise soon, there are a couple of things holding up adoption: availability of tools and of the components themselves.

On the tools side, the AJAX programming model is just too difficult for most programmers. Rudimentary debugging support, late bound script languages, no compiler to help you find errors. Sounds like web development circa 1997, doesn't it? I'm hoping Atlas will be to AJAX what ASP.NET was to ASP.

As for existing components, most of the ones Scoble calls out have little relevance to the enterprise. There's little point to a Feedmap or a Flickr bar in my enterprise app. And I'm not sure I even want to go into the implications of serving up ads in an enterprise app. That pretty much leaves mapping components like VE and Google Maps as the only browser components of interest to the enterprise developer. So far anyway.

I wonder if Scoble knows of any other enterprise relevant browser components out there?

Thoughts on CAB

As I wrote earlier, I'm really impressed with p&p's latest release - the Composite UI Application Block. I had to fly to Atlanta for a customer meeting last week and I spent most of my spare time (flying, in hotel, etc) digging thru CAB. Well, digging through CAB and reading State of Fear by Michael Crichton. IMO, it's his best book in quite a while, perhaps second best he's ever written after Jurassic Park.

Back to CAB. First off, I want more information about ObjectBuilder. p&p's dependency injection framework is very impressive. However, with the exception of the code, the code reference in the help files, the unit tests, and a single article on how it was designed there just isn't much info on it. For example, I know I can use it to inject a concrete type where a class is expecting an abstract type. But I couldn't figure out how so I went and asked Peter and Brad. (Short answer - use TypeMappingPolicy). Do me a favor, go contact Peter and David and tell them you want more info on ObjectBuilder. Sample code especially. Go on, drop them a line right now while you're thinking about it. I'll wait.

Back already? Cool. One thing you can't help but notice about CAB (and OB for that matter) is the heavy use of attributes, which I feel is an extremely elegant solution. Remember the first time you looked at NUnit? How sensible it seemed to use attributes like [TestFixture], [Test] and [ExpectedException] compared to what other xUnit frameworks provide? Get ready to experience that all over again when you look at OB and CAB. Now you're looking at attributes like [CreateNew], [EventPublication] and [CommandHandler]. There's a reason why Sun cloned attributes for J2SE 5.0 - it's powerful (in the right hands). The attributes both drive human readability - when you're looking at a property adorned with [CreateNew] or [Dependency] it's obvious that these are injected - as well as the implementation. Win-win as far as I'm concerned.

CAB does a great job of codifying standard patterns in smart client design, such as events, commands and controllers, as well as implementing completely new ideas such as workspaces, work items and smart parts. And they've done it with an eye on the future. I love the isolation between the windows forms specific parts of CAB and the underlying infrastructure of CAB. Check out this picture of the layers of CAB. By isolating all the WinForms specific stuff in a separate assembly, they are well set up to support WPF while minimizing redundant effort. I can't wait to see a Microsoft.Practices.CompositeUI.Avalon project (because Microsoft.Practices.CompositeUI.WindowsPresentationFoundation is just to long a namespace).

However, I can't shake the feeling of how complex this stuff is. Yeah, you use CAB for solving complex problems, but there is a significant learning curve here. I imagine debugging a CAB app will be significantly non-trivial. Take a look at the final step of the walkthru app. Here's the sequence that gets "hello world" painted on the screen:

1. Main entrypoint creates and runs the ShellApplication.
2. ShellApplication creates a ShellForm window, which in turn contains the tabWorkspace1 workspace.
3. ShellApplication dynamically loads MyModule because it's listed in the ProfileCatalog.xml file.
4. CAB creates an instance of the MyModuleInit class from the MyModule assembly.
5. MyModuleInit creates and runs an instance of MyWorkItem.
6. MyWorkItem creates a MyView and MyPresenter.
7. MyWorkItem adds the MyView instance to the tabWorkspace1 workspace, hosted in the ShellForm
8. MyPresenter handles the MyView load event and sets the message property of MyView to "Hello, World"

Of course, the point of a demo app like the walkthru is comprehension. You would never use CAB to build this Hello World app. But I worry that the level of complexity will put CAB beyond the reach or inclination on many potential users. I imagine this single line of code will scare off many would-be CAB developers:

MyWorkItem myWorkItem = parentWorkItem.WorkItems.AddNew<MyWorkItem>();

Given that most people are used to writing "new MyWorkItem()", the line above represents a significantly rise in complexity. Of course, CAB is trying to solve a complex problem. I'm sure CAB's designers would rather the solution wasn't so complex, but that's the reality of the problem space their dealing with.

If you can't lower the complexity of your framework, it's time to raise the abstraction of your tools.

I wonder what CAB specific tooling would look like? At a minimum, it would like built in support for the primary concepts in CAB - WorkItem, SmartPart, Workspace and Service to name a few. Another opportunity would be to move from embedded strings, used to identify events, commands, UIExtensionSites and the like to true variable-style names that could be validated at compile time, sort of like how LINQ is extending C# to get rid of embedded database query commands. There's lots of possibilities and the more I work w/ CAB the better idea I'll have about them.

I Hate The Term Mashup

Wikipedia has two definitions of mashup:

• Mashup (music): a musical genre consisting of the combination of the parts of more than one song
• Mashup (web application hybrid): a website or web application that combines content from more than one source

I was originally introduced to the term mashup in the musical sense by Daily Source Code. In this usage, mashup implies mixing songs together. Given that songs are typically stand along entities designed to be enjoyed as is, mashup means (to me anyway) combining stuff that was never meant to be combined in the first place. Even Wikipedia includes musical mashup under the general heading of Bastard Pop.

So given that, by definition, mashups combine stuff that wasn't designed to be combined, I don't understand why an application like Zvents or Virtual Places is considered a mashup. Apps like these use browser based components (Scoble calls them Internet Connected Components) that are well defined, have public APIs and are designed to be used together. Not exactly "bastard pop" now is it?

In reality, a site like Zvents could have used a server side mapping component and provided a similar experience. Of course, a client side mapping solution is both sexier and more practical (no need for dedicated map data files or assemblies on your own machines) but semantically they provide the same information. Same thing goes for Virtual Places, except that Virtual Places is also pulling both functionality (i.e. the mapping component) as well as data (i.e. blogs and photos) from other sites across the Internet. Could that be done on the server side? You betcha. Would it be as cool or functional? No. Does that make it a completely different type of application that deserves a new name. IMO, no. These are component based apps - they just use the browser as the platform and the components are coming across the web (as you would expect when you use the browser as a platform).

The higher order bit for me is who controls the experience. For apps like Zvents and VirtualPlaces, it's the application developer. For something like Live.com, it's me. I decide what to put on my Live page. Not that one is more important than the other or that they aren't compatible experiences - I could easily imagine a Zvents gadget that lived in Live.com. Or consuming the RSS feed from Zvents in Live.com. Or a generic iCal gadget that could consume a Zvents iCal feed. But the point is that there are large differences between browser component based application like Zvents and a user managed composite browser application like Live.com.

To me, composite apps like Live.com fit better to the original definition of mashup than something like Zvents or Virtual Places. They both have their place and their value, but I like to call things that are different by different names in order to reduce confusion.

Of course, composite apps aren't limited to the browser. p&p just shipped their Composite UI Application Block last week. I dug into it a bit last week and it's awesome. More on that later.

What Is Up With The Three Amigos?

First, Amigo Grady Booch admits to throwing away models. Now, Amigo Ivar Jacobson has signed on as a VSIP partner:

Microsoft has tapped Ivar Jacobson, known as one of the fathers of the popular Rational Unified Process (RUP), to lead an effort to deliver a lightweight unified process to the Microsoft Solutions Framework.

<snip>

Jacobson said the first fruits of his company's relationship with Microsoft will be through the delivery of the Essential Unified Process (Essential UP), which is based on the Microsoft Solutions Framework and integrated with VSTS.

In short, Essential UP is a simplified or lighter-weight alternative to RUP, Jacobson said. Essential UP is an evolution of the unified process Jacobson helped create more than 10 years ago that forms the foundation of RUP, he said.

<snip>

"We need more lightweight processes," Jacobson said, noting that RUP has become too heavyweight and cumbersome. "We have competition from India, China and the former Soviet Union," he said. "It is not enough to be agile."

Indeed, Microsoft has done a good job with its Microsoft Solutions Framework, and "their process agility is a clear differentiator for them," Jacobson said.

"Essential UP stands on all the experience we have from RUP, but also offers us a chance to have a fresh start," Jacobson said. "We start in a new way because we've learned what works and what doesn't work."

<snip>

"RUP has a lot of good stuff, but it needs a correction," [Jacobson] said.

<snip>

However, among the primary problems Jacobson said he has with RUP is that it is "heavyweight."

Also, "the process architecture needs to be refactored," Jacobson said. "It is very difficult to add new practices because it will force a big change in the base. For instance, adding in a streamlined way practices such as EA [Enterprise Architect], SOA [service-oriented architecture], ABD [asset-based development], re-engineering legacy systems and commercial off-the-shelf software would be very difficult, if at all possible. So I believe in starting all over fresh but not throwing away anything that is good."

<snip>

"Having an industry luminary build the next version of his tool for our platform is huge for us," [VSTS General Manager Rick] LaPlante said.

[From Microsoft Taps Former Rational Heavyweight to Lend Credence to Enterprise Tools Play, eWeek]

I'd like to nominate Rick's comment for understatement of the year awards. It also gets me wondering what Amigo Jim Rumbaugh is up to these days.

Really Simple Code Templates in VS05

David Hayden shows off the Export Templates Wizard from VS05. Very cool. Frankly, I like GAT for most of this type of template work, but this really fits nicely from a simplest thing that could possibly work" perspective.

Thoughts on Architecture Journal 5

If you haven’t had a chance to check out the most recent issue of The Architecture Journal, I highly recommend it. In particular, I liked Metropolis and SOA Governance by Richard Veryard and Philip Boxer as well as Value Driven Architecture by Charlie Alfred.

Richard and Philip dive deep on governance, leveraging Chris Alexander’s Nature of Order and Pat Helland’s Metropolis. In particular, I liked the section on the structural implications of service orientation. As per Alexander, large complex systems can’t be designed in the traditional manner – they evolve over time. This leads Richard and Philip to discuss the highly fascinating concept of asymmetric demand. To quote the article: “Asymmetry means that the forms of demand are increasingly specific to the context in which they arise.” I can’t do the concept justice – just go read the article – but my takeaway is that what you sell isn’t necessarily what people are buying. Take SQL Server for example – you know, one of those products that launched a new version yesterday. Amazing product, but nobody buys SQL Server on its own. It’s always bought in the context of a larger solution. SQL is extremely flexible, so the number of contexts in which it’s appropriate is extremely high. But still, how many business people have ever said “We need to buy a database”. I’m guessing zero. Business people say things like “We need to keep better track of our shipments/inventory/customers/orders/etc” or “we need to better insight into business trends” or “we need to be able to demonstrate our compliance with <insert government regulation here>”. These business problems all need a database like SQL Server, but they need more than SQL Server alone. That’s the asymmetry. Microsoft sells SQL Server (among other things of course, but let’s focus for a second), but customers are buying a solution to their business problem.

On the topic of business problems, Charlie’s article on value modeling flat out states that traditional requirements based design is ineffective. Charlie’s agrees with Jack and Keith that requirements define a system, not the problem the system is intended to address. As such, often critical design decisions are made while defining requirements that have drastic impact on the development of the system down the road. As a somewhat silly example, if I define requirements for a software system, I am precluding any non-software system solution to the problem. What if the best solution for a problem isn’t software? Because of mistaken assumptions I’ve made in the requirements gathering process, I’ve eliminated the best solution to the problem. Woops.

And Charlie has this great quote about the requirements gathering process:

Traditional approaches, like use case scenarios or business/marketing requirements, start by focusing on the types of actors with which the system interacts. This approach has several major limitations:

• It focuses more on what things the actors do, and less on why they do them.

• It tends to stereotype actors into categories, where all individuals of a type are essentially the same (traders, portfolio managers, or system administrators, for example).

• It tends to ignore differences in limiting factors (for example: Is an equity trader in New York the same as one in London? Is trading at market open the same as trading during the day?).

• It is based on binary outcomes: the requirement is met or it isn't. The use case completes successfully or it doesn't.

There is a very logical, practical reason why this approach is popular. It uses sequential and classification-based reasoning, so it is easy to teach and explain, and it can produce a set of objectives that are easy to verify. Of course, if simplicity were the only goal that counted, we'd all still be walking or riding horses to get from one place to another.

I love the point about simplicity. Often, we do things in IT that are simple for IT but that are more complex (and thus bad) for the business. For example, a web app may be the easiest to deploy for IT, but if I have a mobile workforce that web application will cause more business headaches than it solves in IT.

In other Architecture Journal related news, you can sign up for a free subscription, so what are you waiting for?

Grady Booch sez "Throw Models Away"

I wasn't there, but apparently Grady Booch made a comment at OOPSLA last month that throwing away models is a good thing. Juha-Pekka's account of the Design and Modern Software Practices Panel has Grady saying "when the project gets closer to the delivery you normally throw away UML models. This is a natural choice since the efforts needed for keeping the design - user view, dynamics, behavior, interaction etc. - linked with the implementation are simply too high." Ivan Moore has Grady saying "I often throw models away but tend not to throw away the source code."

I've gotta believe that this comment was somehow taken out of context and that the Grand Poobah of the Common Semantic Model doesn't actually believe that tossing the model at the end o the project is a good thing. If we view these comments thru the lens of Code is Model, you realize this is a major violation of the "Models must be Intrinsic to the Development Process" tenet. Since code is model, it seems silly to throw out the UML model and keep the code model. The only reason I could think to do that, is because the UML model has no value.

And this leaves me wondering, exactly how much benefit does Grady Booch get from his UML model if he's willing to throw them away near the end of the project.

Tool Driven Development

I've gotten some great feedback on my Code is Model post. But before I get to those comments, I wanted to talk a little bit about the role of tools. Tools are important in every industry, but in no industry are they as important or central as they are to information technology.

One of the fascinating differences between IT architecture and "real world" architecture is the feasibly of automated transformation. If I have a blueprint of a house, there is no tool to automatically transform it into a house - someone has to do the actual work of putting up the walls, running the plumbing, etc. But in IT architecture, I can transform blueprints - i.e. the code - into the finished application automatically. That speaks volumes about the importance of tools in information technology.

Without the tools to automate the transformation, all IT models - code or otherwise - are nothing more than documentation.

So if we are to include models as part of our development process, we need tools to execute the transformations between levels of abstraction. But where do those tools come from? In the case of abstractions with widespread horizontal appeal, they are likely to come from language vendors such as Microsoft. For example, abstractions like generics and query expressions are useful in almost nearly every project, so it makes sense to include them in mainstream languages like C#. However, one of the key points of Software Factories is that in order to continue raising the level of abstraction, we are going to need narrow the domain to which those abstractions apply.

Developing language-based tools is currently quite expensive, and therefore makes economic sense only in broad horizontal domains, such as user interface construction and data access, where the necessary investments can be amortized over a large customer base. For this reason, commercial implementations of the Language Framework pattern are supplied almost exclusively by platform vendors.

This economic model is threatened by the trade off between scope and value for abstractions. As Jackson puts it, the value of an abstraction increases with its specificity to the problem domain. The higher the level of an abstraction, the narrower the domain to which it applies, but the more value it provides in solving problems in that domain.

We are now at a point in the evolution of the software industry where frameworks and tools must become more vertically focused, in order to realize further productivity gains. They must provide more value for smaller classes of problems. This breaks the current economic model because markets for vertically focused frameworks and tools are too small to support platform plays, and because the domain knowledge required to develop them is housed mainly in end user organizations that have not traditionally been software suppliers.
[Problems and Innovations by Jack Greenfield]

In other words, these domain focused languages, frameworks and tools are not going to come from traditional language vendors like Microsoft. They are going to come from organizations that have deep experience in the given domain, many of whom will not be traditional software vendors. I think SIs and ISVs will adopt this approach before many end user organizations do, but eventually any organization that invests in building reusable frameworks is going to want to invest in building languages and tools to automate the usage of those frameworks.

I think the best way to start developing these tools is to incorporate their construction directly into the software development lifecycle. I liken this to Test Driven Development. If you use a TDD approach, at the end of your project you end up with two outputs - the project and the unit tests. Assuming the code you're building now (with their associated unit tests) is designed to meet current requirements, you shouldn't need to refactor it significantly until those requirements change sometime in the future. The tests you build today become a tool to help you improve your code in the future when the requirements change. From a tool development perspective, Test Driven Development is a specific instance of the more generic concept of Tool Driven Development, as tests are a specific type of tool.

What if, instead of building code test first, you thought about building tools first?

The problem with tests is that hey are tightly bound to the current project. They may help you improve this project's code in the future, but they won't help you develop other applications. I think we should also focus on building tools that make it easier to build the existing project. The goal would be to be able to use these tools not only on future iterations of the current project, but on other similar projects as well. 

Of course, we don't really have the infrastructure to do this effectively today. We need to be able to make it easier to build tools, compose tools and evolve tools over time. Supporting evolution will be key since these tools will likely need to be refined as they are applied to different projects. Test Driven Development has xUnit style frameworks, Tool Driven Development needs the equivalent. The DSL Tools and Guidance Automation Toolkit are a good start, but aren't enough to enable Tool Driven Development on their own.

Do you believe this Tool Driven approach can work? What do you think Tool Driven Development tools would look like? Please let me know.

Which Abstractions Matter?

I've been following the ongoing discussion about typing systems in programming languages between Ted Neward and Stu Halloway with great interest. Given that I believe Code is Model, I'm eager to mine knowledge from successful tools to apply at higher levels of abstraction. And as an employee of a language vendor, I'm also very interested in what Stu describes as vendor-oriented vs. developer-oriented languages.

So why has the static/dynamic debate staggered on for so long? I think we could get closer to some answers with better choice of terms. “Static” vs. “dynamic” is highly misleading. I propose we use a new set of names: vendor-oriented vs. developer-oriented programming...So who do you trust most: vendors or developers?
[What’s new in C#, or who do you trust most?]

With a vendor-oriented language like C#, core abstractions are much more firmly controlled by the language vendor. Conversely, developer-oriented languages like Python leave more of these choices to the developer (although they tend to provide reasonable defaults)...Competency and trustworthiness are sprinkled all over our industry, both among language vendors and application developers. My concern is who controls the abstractions. Developer-oriented languages (like Scheme) give a lot of control (and responsibility) to developers. Vendor-oriented languages (like Java) leave that control more firmly in the hands of the vendor.
[Developer oriented languages]

Personally, I think calling them static and dynamic language is far less misleading than vendor and developer oriented languages. Further, I think Stu is making somewhat absurd statements to garner attention. However, I believe he's certainly onto something with regard to the language abstractions. The abstractions I and my team care about on our project are almost assuredly going to be different from the abstractions you and your team care about on your project. Having a programming environment that enables the abstractions you need on a given project is very very important.

The problem with Stu's argument is that he's focused on low level language abstractions. Abstractions like "inheritance, encapsulation, delegation, how symbols are interpreted, etc." Are you kidding me? Projects don't fail because developers can't change the language's concept of inheritance. They fail because the gap between the abstractions provided by the language and the abstractions needed by the solution are enormous. Modern software development is like building skyscrapers with Lego blocks. Furthermore, projects fail because business and IT don't speak the same language. Business people don't care about concepts like encapsulation and symbol interpretation. They care about concepts like ROI, business plans and regulatory compliance. Geeks may not feel comfortable talking about those concepts, but they are what keep a business in business

Imagine your CFO listening to Stu and Ted discuss these language abstractions. They would be thinking "What the hell are they talking about?" To Ted's credit, he bluntly states that he doesn't trust developers which would likely put him in well with the CFO:

I see the same concern every time a developer starts talking about doing bytecode manipulation at load-time--just because you can doesn't mean you should. In this respect, I trust the guys who've been down this road before much more so than developers who are just coming to this and are starting to flex their new-found freedom and will (undoubtedly) start building systems that exercise this power.
[Dynamic languages, type systems and self-modifying systems]

I wouldn't go so far as to say I don't trust developers, but Ted's point about can and should is spot on. I'm sure there are scenarios where bytecode manipulation is critical to the success of the project. Hell, in the project I'm currently heads down on (hence the lack o' posts in the past two weeks) I'm using much more reflection and late-binding than I ever have before. Not because I can - frankly, I like static typing - but because that's the best way to solve the problem at hand.

It's important to keep the big picture in mind when discussing minutia such as a given programming language's core abstractions. IT exists to serve the business, not the other way around.

The Irrelevant Semantics of Compiled XAML

From newly converted ARC MVP Sam Gentile, I found this interesting post from Drew Marsh about how XAML is compiled:

XAML is indeed a language, but it is never compiled into C# or IL... The truth is, it's not "compiled" at all. If anything you can say it is "compacted" and that only happens in scenarios where it is turned into a BAML stream. That, however, is an optimization and not a necessity for XAML to work. XAML can be interpreted purely in it's raw XML text form, using Parser::LoadXml. Even BAML is interpreted, it's just a far more compact and efficient representation of the object graph than raw XML text.
[Drew Marsh - The XAML Experience]

Given that I just wrote about compiling, I wanted to weigh with a couple of points:

• First, by definition compilation is a translation from one format to another. Therefore, converting XAML to BAML is a compilation step. The SDM folks have a command line tool for compiling deployment reports from the models in the Architect edition of VSTS. However, I assume what Drew meant here was that XAML isn't compiled into a directly executable format, so in reality I'm just being picky about the use of the word "compiled".
• Second, the fact that the XAML is compiled into an efficient binary representation and then embedded as a resource (as per Rob Relyea) is fascinating from an implementation perspective, but somewhat irrelevant semantically. Drew points out that the BAML is interpreted. With VM environments like CLR, the line between interpreted and compiled blurs considerably. Rob's post referenced above is based on the PDC 03 XAML bits, and at the time, XAML could be compiled into BAML or IL. However, at the time (20 months ago) Rob guessed that the IL compilation would be cut because the BAML perf was just as good or better, the file size was smaller and localization is easier. In the end, the XAML file is converted into a format the machine can execute - the specific choice of compilers and transformations isn't particularly interesting from a modeling perspective since it happens automatically.

XAML isn't the only place where the traditional compiling to executable model is being stretched. In Windows Workflow Foundation, though your workflow is defined as a type, you can actually modify running instances of the workflow. Given that WF supports declaring workflows as C# or XOML (soon to be XAML), I wonder if they are going to go the same route as WPF and eliminate the C#/IL way of declaring workflows. Another interesting example is LINQ and C# 3.0. This is interesting because you can use LINQ directly on in memory data, but when you apply it to database (via DLinq) the LINQ statements are parsed into expression trees then converted into SQL. (Check out this post from Ian Griffiths for deeper coverage of expression trees).

Anyway, it's late and I realize I've written quite a bit to basically say that the definition of "compiled" is pretty blurry at this point and getting blurrier going forward. In the end, it's much more interesting IMO to focus on the model environment you're working in (XAML in this case) rather than the details of how that model is translated into the execution environment, unless you're the one building those translation tools.

UPDATE - I had one other thought on all this. It's interesting that computing power (CPU + IO bandwidth) have improved to a point where the performance of interpreting BAML at runtime is as fast than executing XAML compiled to IL directly. I certainly wouldn't have assumed that.

Code is Model

In the foreword to Architecture Journal 3, I wrote the following:

Abstraction is the architect's key tool for dealing with complexity. We've seen the evolution of architectural abstractions techniques such as models and patterns; however, we have yet to realize much in the way of measurable value from them. Models and patterns are useful for communicating effectively with our peers, but so far they haven't helped drastically reduce the amount of resources it takes to build and operate a system. In order to continue to deal with increasing complexity, we need to get much more pragmatic about using abstraction to solve problems.

Because the lack of measurable value to date, the IT industry at large has come to view models at best as "pretty pictures" and at worst as a pointless waste of time and resources. But the reality is, we use models in the IT industry all the time. I don't know what you're favorite modeling tool it, but my current favorite is C#. Before that was VB and before that was C++. I realize some of my readers might be more partial to Java, Ruby, or even Assembly. But the reality is: all of these so-called "higher level" programming languages are simply models of the CPU execution environment.

The only code that the CPU can understand is machine code. But nobody wants to write and debug all their code using 0’s and 1’s. So we move up a level of abstraction and use a language that humans can read more easily and that can be automatically translated (i.e. compiled) into machine code the CPU can execute. The simplest step above machine code is assembly language. But ASM isn't particularly productive so work with, so the industry has continuously raised the level of abstraction in the languages they use. C is a higher level of abstraction than ASM, adding concepts like types and functions. C++ is a higher level of abstraction than C, adding concepts like classes and inheritance. Each of these levels of abstraction presents a new model of the execution environment with new features that make programming more productive (and sometimes more portable). Different languages offer different models of the execution environment. For example, the Ruby model of the execution environment allows for the manipulation of class instances while the C++ model allows for multiple inheritance. This isn't to say one is better than the other - they are just different.

In the past decade, we've seen the rise in popularity of VM based programming environments - primarily Java and CLR. In these environments, there are multiple models at work. CLR languages and Java are models above the underling VM execution environment. The VM execution environment is, in turn, a model of the physical execution environment. As an example, a C#/Java program is translated into IL/bytecode at compile time and then from IL/bytecode to machine code at runtime. So in these VMs, two model translations have to occur in order to go from programming language to machine code. It turns out that this multiple step translation approach is also useful in non-VM environments. For example, the original C++ compiler output vanilla C code which was, in turn, compiled with a vanilla C compiler. C# and Java use a similar approach, except that the second translation occurs at runtime, not compile time.

So if Code is Model, what can we learn from looking at the success of mainstream text-based programming languages to help us in the development of higher abstraction modeling languages that are actually useful. This isn't an exhaustive list, but here are a few things (tenets?) I've thought of:

• Models must be Precise
  There must be no ambiguity in the meaning of the elements in a model. In C#, every statement and keyword has an exact well-defined meaning. There is never a question as to what any given piece of C# code means. There may be context-sensitive meanings, such as how the keyword "using" has different meanings in C# depending on where it is used. If you don't have a similar level of precision in your model, there's no way to transform it to lower abstraction models in a deterministic fashion. Models that can't be transformed into lower abstraction models are nothing more than pretty pictures - perhaps useful for communication with other people on the project, but useless as development artifacts.
• Model Transformation must be Deterministic
  By definition (or at least by convention), models are at a higher level of abstraction than both your execution domain and mainstream programming languages - perhaps significantly higher. In order to derive value from a model, you must be able to transform it into the execution domain. Like the C# to IL to machine code example, the model transformation may comprise multiple steps. But each transformation between models must be as precise as the models themselves. When you compile a given piece of C# code, you get the same IL output every time. However, this transformation can vary across target models. For example, when you run a managed app on a x86 machine you get different machine code than if you ran it on an x64 machine.
• Models must be Intrinsic to the Development Process
  Even if you have precise models and deterministic transformations, you have to make them first class citizens of the development process or they will become outdated quickly. How often have you blueprinted your classes with UML at the start of the project, only to have that class diagram be horribly out of date by the end of the project? In order to keep models up to date, they must be used through-out the development process. If you need to make a change, make the change to the model and then retransform into the execution domain. Think of C# and IL - do we use C# as a blueprint, transform once to IL and then hand edit the IL? No! We change the C# directly and retransform into IL. We need to have the same process even as we move into higher levels of abstraction.
• Models Aren't Always Graphical
  Some things are best visualized as pictures, some things aren't. To date, we're much better at graphically modeling static structure than dynamic behavior. That's changing - for example, check out the BTS or WF tools. But generally, it's easier to model structure than behavior graphically. Don't try and put a square peg in a round hole. If a text based language is the best choice, that's fine. Think about the Windows Forms Designer in VS - you use a graphical "language" to lay out your user interface, but you implement event handlers using a text-based language.
• Explicitly Call Out Models vs. Views
  One of the areas that I get easily confused about is model views. If I'm looking at two different model visualizations (text or graphical), when are they different models and when are they views into the same model. People don't seem to care much one way or the other, but I think the difference is critical. For example, a UML class model and a C# class are two separate models - you need a transformation to go back and forth between them. However, the VS Class Designer is a graphical view into the model described by the C# class definitions. Changes in one view are immediately visible in the other - no transformation required. If you look at the Class Designer file format, you'll notice only diagram rendering specific information is stored (ShowAsAssociation, Position, Collapsed, etc.). I guess this could fall under "Models must be Precise" - i.e. you should precisely define if a given visualization is a view or a model - but I think this area is muddy enough to warrant it's own tenet.

I'm sure there are more thoughts for this list, but that's a good start. Please feel free to leave your opinion on these tenets and suggestions for new ones in my comments.

Ted on C# 3.0

I just discovered Ted Neward's blog has moved. In catching up, I found this great post on the new features of C# 3.0. Even though I had read thru the C# 3.0 spec, Ted's explanation was much easier to read.

FYI, speaking of Ted, I'll be speaking at his No Fluff Just Stuff .NET software symposium. Still working w/ Ted on the abstracts, but basically I'm basically talking about patterns, GAT and DSLs.

MVP Summit Wrap Up Thoughts

It's hard to believe it October already. The last three weeks have been jam packed, starting with PDC 05, then a variety of meetings culminating with the company meeting the following week, then the 2005 MVP Global Summit last week. This was the first MVP Summit to include Architect MVPs so it was pretty stressful. Of course, there were things we could have done better, but all-in-all I was happy with the event. A year ago, we had just awarded our first 14 Architect MVPs. Now we're 100 strong between our solution and infrastructure Architect MVPs and we had better than half of them in Redmond for the summit. I swear, it will take us the rest of the fiscal year to implement even half of their suggestions.

I'm sure each of the various groups that have MVPs think that their MVPs are the best, so I guess I'm no different in that regard. Our Architect MVPs are an amazing group and I am already looking forward to the next opportunity to get a bunch of them in a room together again.

DevHawk on C9

I'm sure it will get lost in the massive tide of C9 videos sure to come out of PDC, but Scoble posted an interview he did with me a few months ago. Check it out.

Portability without Productivity

I said this was going to be a slow week, so I dug out something I wrote while I was on paternity leave. I was saving it for a rainy day, which is pretty silly since if it was raining I wouldn't mind staying indoors and writing long posts about levels of abstraction, portability and productivity.

My father and I have a running argument debate about the value of platform independence in the system design and development process. As you might guess, as an employee of a platform company I stand firmly on the "platform neutrality is irrelevant" side of the debate. Having been around Bell Labs during the development of Unix and C as well as having done a stint for an ISV, my father is firmly on the "platform neutrality is important". Typically, these discussions turn into childish argument where my father continuously says "what if" and I continuously say "that never happens" and neither of us actually makes any ground on convincing the other of the error of their ways.

So herein is yet another salvo in the discussion, for your entertainment. It's long and drawn out, but since it's written it means he can't interrupt me to say "what if" :)

As mentioned above, my father was at Bell Labs in the early 70's when Unix and C were developed. I guess it's no surprise that he harps so much on portability - going back and reading some of the papers that came out of Bell Labs at the time, it's obvious that their culture heavily valued portability. On Dennis Ritchie's site (the 'R' in 'K&R') there is a wide variety of relevant material including The Evolution of the Unix Time-sharing System, The Development of the C Language, and Portability of C Programs and the UNIX System. Given the drastic evolution in computing at the time, it's not surprising that both Unix and C had portability as primary goals. While I jokingly refer to C's portability as "write once, compile everywhere", the reality is that the portability of C and Unix was a key to Unix's success. According to the portability article, 95% of the C portion of the Unix kernel required no changes when they ported Unix from PDP-11 to the Interdata 8/32. Only the core kernel, which was written in assembly, had to be completely rewritten. Even a significant portion of the device drivers ported over to the new machine.

However, C isn't just portable. It also provides significant abstraction above assembly code. Anyone who has done any assembly work knows how low level it is and how significant the abstraction jump up to C really is. For example, the simple C statement of "a = b + c" requires three lines of assembly code. Here's how Ritchie describes C's level of abstraction:

BCPL, B, and C all fit firmly in the traditional procedural family typified by Fortran and Algol 60. They are particularly oriented towards system programming, are small and compactly described, and are amenable to translation by simple compilers. They are `close to the machine' in that the abstractions they introduce are readily grounded in the concrete data types and operations supplied by conventional computers, and they rely on library routines for input-output and other interactions with an operating system. With less success, they also use library procedures to specify interesting control constructs such as coroutines and procedure closures. At the same time, their abstractions lie at a sufficiently high level that, with care, portability between machines can be achieved.  (emphasis added) [The Development of the C Language - Dennis M. Ritchie]

That last sentence is key. C's portability derives from raising the level of abstraction. But raising the level of abstraction also had an important productivity impact. Even if you're are only building for a single platform and you don't care about portability, most people would rather code in C rather than assembly because the raised level of abstraction makes the developer much more productive. However, raising the level of abstraction comes with a performance cost. C is pretty 'close to the machine' as Richie put it, but there is still a small penalty. If you're writing ultra-perf sensitive code, sometimes writing in assembly is necessary. There’s a reason why books on the topic keep getting written and the Visual C++ compiler supports inline assembly code.

So here's my point: Raising the level of abstraction is powerful because it can enable both portability and productivity. However, it also typically carries a performance penalty. As long as the portability and productivity benefits outweigh the performance penalty, everything’s cool. The problem is that productivity is typically WAY WAY WAY more important than portability. So abstractions that enable portability without significant positive productivity benefits will not offset the performance penalty associated with raising the level of abstraction.

The canonical example of "portability without productivity" abstraction that leaps to mind today is the Java platform. Certainly, Java has been pretty successful, though I would argue that its success has been extremely localized. Java on the client has never had mass adoption (the only non-toy Java client app I can think of off the top of my head is Eclipse) and the many of the parts of Java on the server bear a striking resemblance to Microsoft technology. (ODBC vs. JDBC, ASP vs. JSP, Session beans vs. MTS, etc.) Either way, Java adoption has fallen below .NET adoption even though Java had the promise of platform neutrality as well as several years head start. [1]

I would argue that one of the main reasons that Java has had only limited success is that while Java is portable, it doesn't provide much in the way of productivity improvements. Sure, garbage collection is much easier to program to than C++'s explicit memory management model. But Java's primary competitor (at least in hindsight) was Visual Basic, not C++. While Java focused on portability, VB focused on productivity and in the end it was productivity that drove VB's massive market share. If you were a Java developer, you had worse tools than VB and worse performance than C++ and the only advantage you had was portability which turned out to be more problematic and less important than advertised. Server apps are rarely re-platformed and Java's UI implementation caused as many problems as it solved.

From a geek aesthetic perspective, you would have guessed that Java with its clean language and programming model would crush VB and COM in the marketplace, but it just didn't happen. VB, with its software factory-esque approach, was easier to use and thus attracted more developers who got more apps written. I'd guess that ease of use / developer productivity is the key indicator of success for programming environments. If Java had focused on productivity and portability, I might be working for Sun today.

[1] Obviously, there are varying opinions on this point. SteveB said @ TechEd that .NET is the weapon of choice (my words not his) for 43% of all developers. Java was second with 35% and non .NET Windows development was third (no percentage given).  Even if you want to nit-pick on the numbers, you’d be hard pressed to argue that Java hasn’t been losing ground dramatically to .NET in the past three years since VS 2002 RTMed.

Blah Blah Architecture

If you heard my PDC05 Buzzcast but just can't get enough of my voice, check out Architect MVP Mario Cardinal's Blah Blah Architecture podcast. I spent some time with Mario at TechEd talking about the Architecture Resource Center, why we designed it the way we did, why we have a site on MSDN and MS.com, how are taxonomy works, etc. Mario then takes my basic explanation and helps fill in more of the details, such as the relevance of the Architecture Resource Center to folks not using Microsoft technology. We also talked about Architecture Journal as well as why Microsoft invests in architecture.

BTW, I make some of the same points about architecture - as it differs from engineering, that it's the space between business and technology, and the effect of title inflation - that I've made here on this blog this past week.

I guess I need some new material. :)

Architecture Innovation?

So if architecture is the connection between business and technology, where does innovation fit? Microsoft has been pushing an idea of "integrated innovation" for several years now, but that's primarily around technology innovation:

[T]he mission [of] Integrated Innovaton is about ensuring that the value of the Microsoft platform is greater than the sum of its components. It's the coordination of software products, the way entire systems can be made to work together better. It's a strategy to add customer-driven features and functionality to achieve specific business results while reducing cost and complexity
['Integrated Innovation' Provides Partners with Roadmap to Success]

But what about business innovation? How often do we talk about that? Not enough IMO. Especially since the business innovation is much more likely to make or break a company than technology innovation. For example, how did Dell became the dominant company in the PC business? To quote from the Amazon review of Michael Dell's book: "What makes Dell Computer unique is not what it sells, but rather how it sells it". Usually, people note Dell's direct-selling model as the catalyst for their success. Direct-selling might have been absent in the PC industry before Dell came along, but it's not like direct-selling is a particularly innovative business model. However, what made the direct-selling of highly-configurable computers a reality was the innovative approach Dell took to managing it's supply chain. Quoting from an Accenture case study on Dell's supply chain:

Explains Dick Hunter, vice president, supply-chain management: “We now schedule every line in every factory around the world every two hours, and we only bring into the factory two hours' worth of materials. We typically run a factory with about five or six hours' worth of inventory on hand, including work in progress. This has decreased the cycle time at our factories and reduced warehouse space—space that has been replaced by more manufacturing lines.”

Not surprisingly, the project has produced more than just enhanced supply chain efficiencies and accelerated, highly reliable order fulfillment. At any given time, there is less than four days of inventory in the entire Dell operation, while many competitors routinely carry 30 days or more. In addition, automation has helped Dell react more quickly to correct potentially out-of-balance situations, made it much easier to prevent components from becoming obsolete and improved response times across the supply chain by providing a global view of supply and demand at any specific Dell location at any time.

Certainly, there is technology innovation involved in the management of Dell's supply chain (and w/ RFID on the horizon, significantly more technology innovation in this space is still to come) but the primary innovation here was business oriented. I wonder where the next business innovations are going to be?

John asked  me over lunch if the people who read this blog would think I'm an architect or an engineer. Personally, using the definitions I've laid out this week, my heart's in engineering but I'm getting more interested in architecture. Maybe I'm wrong, but it feels to me that pure engineering problems are giving away to more architectural problems as Moore's law and it's corollaries in network speed and storage space keep pushing out the limits of computing power. Jim Gray & Charles Levine wrote an funny article pointing out that Jim's two year old TabletPC with a 1.6 GHz Centrino processor can handle over 8,000 transactions per second. To put that in perspective, in 1976, Bank of America's DebitCredit system reached 100 transactions per second. It took a decade to build a system that handle over 200 transactions per second. Now, most of us are walking around with machine that could easily handle 40 times that performance.

As Moore's law continues to solve technical challenges, I think it is creating new business challenges. And you know me...I like a challenge.

PDC05 Architecture Symposium Buzzcast

Michael cornered me on Tuesday to record a PDC05 Buzzcast with Mike Burner about the Architecture Symposium. At PDC03, the Architecture Symposium was one of the more popular and successful aspects of the overall confernece (though it was marred by a major room change issue that caused literally hundreds of attendees to be forced to watch from the hallway outside as the room was literally overflowing) and we're looking to do something engaging again this year.

Like last time, the Architecture Symposium will be held the last day of the conference, Friday from 8:30 until noon (with breaks of course). After lunch, we'll have a panel discussion featuring Gregor Hohpe, David Ing, Tony Redmond, Steve Swartz.

Here's the full symposium abstract:

You've had a tantalizing week of cool technology, but now you need to transition back to your real job: making all of the pieces work together. The PDC Architecture Symposium will zoom you through the solutions lifecycle - from requirements to modeling to requirements to iterative development to requirements to operational feedback (which you might look at as another set of requirements) - showing you how traditional best practices and recent innovations can be used together to build robust solutions that accelerate business value creation.

Topics include:
The Architecture of Connected Systems
In the beginning there are the models - from the thing you scrawl on a napkin at lunch to that enormously complex diagram that your network architect carries around in a cardboard tube. What models are worth creating, and how do they relate to each other? Who are the key stakeholders for each, and how can you help them talk to each other? This session explores how to decompose value chains into your key models - your process and work flows, the information at the heart of your processes, and the access, deployment, and other operational models that you need to stay trustworthy and compliant. We will then map these models into a collection of services, orchestrations, and policies that define a highly integrated solution.

Building Connected Systems
With so much complexity and so many stakeholders, how do you build the right thing on time? This session explores the techniques to iterate agilely through a connected system project, including the patterns and practices for building solutions that combine messaging, workflow, structured information, and human interaction across platforms and across organizational boundaries. How can we give the right access to everyone in the value chain, respecting the very real boundaries around information and process control? How do we keep our models current, and use them to communicate with all of the stakeholders throughout the development lifecycle?

Managing the Connected Systems Lifecycle
As each iteration of your connected system is deployed and used, new requirements and system refinements emerge. How do we design in the operational hooks that give us the insight to learn from our deployed solutions? How do we re-factor and version our services and orchestrations to improve service reuse, scalability and operational efficiency? The key theme is driving collaboration between development and operations groups from the earliest design phase through the ongoing maintenance of the system.

As Michael said on the buzzcast, you just can't miss the Architecture Symposium. See you there.

Believe Me, You're Not Architecting

So I said yesterday that I'm talking about architecture, not architects. However, today I am going to discuss the word architect and the dramatic misuse of the word I see pretty regularly. Or at least, I see now because Paul Preiss of IASA mentioned it when we were hanging out at TechEd.

"Architect" is a noun, not a verb.

Usually, when I hear the term "architect" used as a verb, it's being used as a synonym for design. In fact, the term "architecture" is also often used as a synonym for design. For example, Arnon wrote this:

Architecture is design (but not all design is architecture)
[Arnon Rotem-Gal-Oz, What's "Software Architecture"?]

Like Fowler's description I talked about on Monday, I don't like this one much either. Architecture isn't just "good design" which is what Arnon's description makes it sound like. This begins to get into the title inflation that Alan Cooper wrote about a few years ago. Speaking of Alan, here's his definition of architect.

The panoply of software construction includes three vital roles: the programmer, the engineer, and the architect. The architect is responsible for determining who the user is, what he or she is trying to accomplish, and what behavior the software must exhibit to satisfy these human goals. The engineer's responsibilities are comparable but focused on technology. A good engineer can and should ignore human issues, confident that the architect will cover the human side.

That definition of architecture (i.e. what Alan's idea of an architect does) dovetails pretty nicely with mine. The architect and architecture is the link between the users (i.e. the business) and the software (i.e. IT). The only thing I would change is that as we get deeper into service orientation, interop and connected systems we have lots of process that don't have direct user involvement. So the "human goals" Alan mentions may not be end user goals so much as business / organization goals. Either way, they certainly aren't IT goals.

As Dave Welsh said and my dad pointed out in my comments, Business is from Mars, IT is from Venus. But that doesn't mean they can't get together. In fact, they have to get together. You show me a system with no business drivers or impact and I'll show you a failed architecture.

Architecture at the Intersection

Arnon Rotem-Gal-Oz left a comment yesterday disagreeing with my characterization of engineering vs. architecture problems:

I can't say I totally agree that it is the software architect's role to come up with the list of functional requirements. That's what business analysts or requirement engineers do. The role of the architect is to identify which requirements are important and to surface the significant quality attributes (non-functional requirements) of the system (again, the architect is not the source of the quality attributes but his role is to determine which the most important ones are).

I'm glad Arnon responded. His blog looks pretty interesting and he's working with Architect MVP Udi Dahan. Subscribed.

However, I wasn't talking about architects, I was talking about architecture. That may seem like a subtle difference, but between the wide variance in the definition of architecture and the wide variance in types of companies, trying to figure out a general description of architect across all that variance seems like chasing your tail.

So with that in mind, here's an interesting distinction between building architects and engineers. I think Pat Helland told me this, but I forget. If you know the origin of this comment, please drop me a line.

Engineering is about walls. Architecture is about the space between the walls.

I love that distinction. That leads to my personal definition about architecture in the enterprise and/or software realm:

Architecture is the intersection between business and IT.

Most of what I see passed off as architecture lives purely in the realm of IT. For example, as much as I like Fowler's PoEAA, I would argue that those are engineering patterns as they have little to do with business. If you're building a system for an enterprise, there's a business driver funding that system construction. The business folks don't really care which domain logic or data access patterns you use. Domain Model vs. Table Module? Table Data Gateway vs. Row Data Gateway? Come on, you think business people care about that?

If a decision doesn't effect a business person, it's not an architecture decision. I'm not saying it's not important - I think the role of the software engineer is critical in large-scale enterprise system design and construction. And I will readily admit that often a single person is responsible for both architecture and engineering. But that doesn't make them the same activity. As long as we continue to confuse the two disciplines, we hold them both back.

Tomorrow, who's doing all this architecting?

What is Architecture?

So since one of my jobs is to change how we evangelize architecture, I thought it might be good to get some clarity around that term. I doubt there's a word more varied in definition in this industry than architecture. I often joke is that we call both the person who sets strategic technology direction for the enterprise and the person who decides whether to use a linked list or an array an architect. Maybe it's not that bad, but "architect" does appear to have a wide range of definitions.

I once moderated a discussion at the Strategic Architect Forum that featured Martin Fowler and this topic came up. Fowler's definition of architecture was something along the lines of "The activities on a project that you do first because you think they'll be hard to change". He suggested that asking someone to show you their architecture was a sure way to find out the parts of the project they thought most important or were most worried about. While that's interesting, I think it's harmful to not have a consistent definition of the term across the industry. Everyone knows what a developer does. Nobody knows what an architect does. Well, it seems that way anyway.

Maybe it's because my degree is in engineering, but much of what we call architecture in the computer industry feels more like engineering than architecture. One of the dictionary definitions of architecture is the "art and science of designing and erecting buildings." Engineering is defined as "The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems."

IMO, building a system that has a set of functional requirements (track customers, process orders, etc) and non-functional constraints (sub-second response time, support 10,000 concurrent users, use Microsoft Windows platform, etc) is an engineering problem. Coming up with the lists of functional requirements and non-functional constraints is the architecture problem.

More on this tomorrow...

I Said I'd Be Back

So my time off ended a month ago, but I haven't blogged significantly in over two months. New kid + new job + new house will do that to you. However, I spent the time this morning upgrading DevHawk to the new 1.8 version of dasBlog and I'm ready to jump back in.

Back when I was still on leave, my new boss John deVadoss this to say about my new job:

Our current thinking is that Harry will focus on two top-level areas

1. Being the storyteller (Metropolis, Connected Systems etc)
2. Changing the way we evangelize Architecture - its not all about n-dimensional frameworks, with m layers of abstraction, about perspectives and viewpoints, with n-layers of capability mappings, and enterprise frameworks up and down the wazooo, blah blah blah.

I loved #2. Architecture is such different things to different people (more on that later) but I thought John's description about what it's not all about was priceless.

Dad getting some link love

After a six month silence, Dad posted about what he called Business Oriented Architecture (BOA?):

If I had to guess what a service oriented architecture will eventually look like, I would guess that it would reflect the business architecture – Business Oriented Architecture (BOA). Business organizations have evolved over many centuries into a number of common “departments” – sales, accounting, personnel, etc. Perhaps that is a good starting place for services.

John forwarded me a post from Richard Veryard where he comments on Dad's post:

The issues addressed in my book are now becoming mainstream as the technological agenda of service-oriented architecture (SOA) starts to converge with the strategic agenda of the service-based business (SBB). This implies an approach to business strategy that involves dynamically managing the geometry of the business. (To achieve a fully adaptive enterprise we typically need to implement a variable geometry.) We can find elements of this thinking in some of the methodologies coming out of IBM and Microsoft, although from what I've seen so far I don't think any of these methodologies go far enough.

Hal calls this Business Oriented Architecture. If anything, I'd prefer to call it Architecture-Oriented Business. As Hal indicates, this calls for architectural thinking at the business level, which need to be aligned with architectural thinking at the information/software level.

This comes back around to the whole SOA top-down vs. bottom-up argument. Something I'll comment further on when I'm not up to my armits in moving boxes.

Real World GAT - NUnit Converter

Jim Newkirk has released an alpha version of a utility to convert NUnit tests to VSTS tests. What he doesn't mention in the post is that he's using GAT to integrate this conversion functionality into VS05. Basically this conversion is an "unbounded recipe" which means that any time you right click on an item in the solution explorer, the NUnit Converter uses Visual Studio's CodeModel functionality to analyze the contents of the file. If the file is a C# file and has any NUnit test fixtures in it, NUnit Converter adds a "Convert NUnit Test Code" item to the context menu.

From a cursory glance at the code (which Jim was kind enough to send me) it doesn't look like it took very much GAT code to integrate into VS. Of all the files in the solution, there are only three that relate to GAT - the Conversion Action (i.e. the code that initiates the NUnit -> VSTS conversion when you select the context menu item), the Conversion Recipe Reference (i.e. the code that determines if the conversion menu item should be added to the context menu) and the Selected Project Item Provider (i.e. the code that retrieves the selected file from the Solution Explorer). There's also the XML file that defines the recipes. Everything else as far as I can tell handles the conversion itself and has nothing to do with GAT.

It's cool to see a real-world usage of GAT and that using GAT is a pretty low-impact effort given the VS integration benefits it provides.

My Next Job

So after doing community and marketing for the Architecture Strategy Team in the last two years, I'm changing roles on the team again. I'm going to work for John deVadoss as a solution architect. In other words, I'm going to actually do architecture work for the Architecture Strategy Team. Go figure. :)

Seriously, I've actually enjoyed working for Norman and with Tanya. This has been in the works for a while and I chatted about it with some of my friends who I saw at TechEd. Someone quipped that I was "getting my brain back". Maybe I'm not getting the whole brain back, but I don't really see it that way. Marketing is very very very different from technical jobs like development and architecture but it's just as hard. However, it does use a different part of the brain. So while I'm excited to moving into a role that plays to my strength, I'm walking away from my marketing experience with a new respect and understanding for the job. John wrote about SteveB talking about "selling what we build, AND building what we can sell". I feel that spending time in marketing has made me a better equipped to build what we can sell. Frankly, I think more (all?) technologists should spend a stint in marketing so they understand what it's all about.

Anyway, I'm still going to be involved in community and Norman considers me a "stealth" marketing technologist (plus, he replaced me with an experienced marketing person) so I guess I'll also do some marketing stuff. So I'll continue to be involved with stuff like Architect MVP and TechEd ARC track, which I've really come to enjoy. But I'll also be involved in projects, doing customer briefings and I've got a bunch of other things brewing. So it should be fun.

However, before I make the switch I'm going on paternity leave for four weeks first. Ahh, four weeks of rest and relaxation...yeah right. We move into our new house on Monday so I'm looking at four weeks of heavy lifting, sorting stuff in boxes and changing diapers (it is paternity leave after all). I'll try and post occasionally, but I'm going incredibly busy and have spotty network access. I'll probably post pictures to my MSN Space on occasion since I can do that with my phone. Otherwise, I'll see you on the other side.

More TechEd 05 ARC Track Changes

As per yesterday's update, here are the latest changes to the ARC track for those in Orlando

• ARC304R - We're repeating the Smart Client Architecture talk Thursday at 5pm in room S 220 E. Note, this repeat session will only feature Billy Hollis as the speaker.
• GAT Lab - Due to technical difficulties, the Guidance Automation Toolkit lab didn't make it to the HoL floor. However, you can download the lab from the brand new GAT community site. In addition to blogs, forums and a wiki, GuidanceAutomation.net features a library where you can find and share GAT resources - including the TechEd Lab - with other members of the community.

Update (02-23-06): Updated link to the GAT Lab.

TechEd 05 ARC Track Additions

For those of you that are in Orlando, there are a couple of ARC track additions and other items of note:

• ARC305R - We're repeating Scott Hanselman's Code Generation talk again Wednesday at 5:30pm in room S220E
• ARCC17 - The TBA ARC Cabana Session will be on patterns & practices Futures with Rob Jacobs. That's at 9am on Friday.
• Speaking of p&p, they are having their weekly p&p Live webcast in the ARC cabana. This "cabanacast" is happening 2pm on Thursday.

TechEd ARC Announcement Roundup

Of course, TechEd is a big time for product announcements, new websites (like ARC) and other assorted new stuff. Here's a roundup of stuff I've been tracking:

• TechEd is the first time we're talking about the new Architect Certification in earnest. There has been quite a bit of cabana traffic on this topic. People hear "certification" and think it's a simple test, but when we explain that it's more like a PhD board, they get more impressed. When they talk some of the people who have been through the beta program and they hear how hard it is, they are more impressed. So far, I think we've only passed around two thirds of the candidates even though the beta program consists entirely of hand selected partners and MSFT employees. Mario made a great analogy to 18th century medicine - certification helped the public separate the real doctors from the charlatans. There's also a great roundtable interview about the program up on PressPass. Joe from that interview spent a lot of time talking certification in the cabana yesterday. He mentioned that while there is some product focus, a deep Java architect should be able to pass the certification. Good news for Ted who didn't have many takers to sign his book Effective Enterprise Java at the TechEd author signing on Sunday.
• patterns & practices launched their new dev center on MSDN. They've also released previews of their new Global Bank Integration Baseline Architecture (typically just referred to as GBI) and the WS-I Basic Security Profile Sample App. I got a preview of the GBI project a few weeks ago, and I was very impressed (though it will be even more impressive if they release it as a GAT package). GBI has lots of moving parts that are headless, so the team wrote an application called The Narrator to help users explore what is happening in the system. It provides multiple views into the running system, including a system view, a services view, a pattern view and a security view. When I saw this, my first reaction was "Let's get that on the web!" I mean, it's great for use with the actual GBI, but GBI consists of 40 projects running across as many as six machines. That's a lot of time investment to simple explore the system. So based on my suggestion, the p&p folks built a flash demo of the GBI Narrator. Of course, the flash demo doesn't connect to a running system, but it's otherwise a very similar experience to the "real" narrator. Let us know what you think of this approach.
• Lots of other people have linked to the new VSTO Outlook that SteveB announced on Monday, but I also wanted to link to David Hill's blog. David is an architect from the Architecture Strategy team, but the VSTO team "borrowed" him 3-4 months ago to help make this happen. Watch David's blog for info on what's happening under the covers with VSTO Outlook. Congrats David!

TechEd 05 Day One - This One's for Chris Sells

Back in November, we published a series of articles about Software Factories. As excited as I am by the concept, Chris Sells really brought me back to earth with this post:

I've been reading each of the Software Factories articles with great interest. Part 1 and part 2 did a particularly good job describing the elements of the problem space, I thought. However, when I get to part 3, I was ready to see a solution. Instead what I got was a long abstract piece defining the bits of what makes up a software factory. This is the kind of thing I'd be ready to read after I was shown a concrete example or two of working, running software factories. Do other people like reading these long, abstract articles? I find them tedious unless they're filling in and generalizing the details of something that I've already got a handle on.
[Chris Sells - Concrete Examples of Software Factories?!? - Nov 7th, 2004]

When we were planning the ARC track for TechEd this year, I sent this post to Keith and Jack and told them I wanted to show a concrete example of a working, running software factory. I wanted to impress Chris. Today was ARC302 - Building and Using a Software Factory with Jack and Wojtek. I don't think Chris was here today, but if he had been, I think he would have been impressed. It's one thing to see all the various parts working on their own, but it's very different to see everything working together in concert.

The factory scenario they demoed was for building smart client apps. p&p already has a bunch of existing assets like EntLib and UIP that is useful for building smart clients as well as a ton of guidance for building smart client apps. But long books and source code projects are not the easiest form of guidance for developers to consume. The factory ties these assets and guidance together to provide a powerful in-tool experience for building such applications. For example, they started by instancing a smart client solution template. This had two projects - the UIP related code and the main application code. The UIP project had all the boilerplate code that every UIP project needs to use, but it didn't have any code for individual UIP page flows. So they invoked a GAT recipe to create a UIP page flow. This is one way where factories differ from traditional wizards - wizards are typically only invoked when creating a new solution. Factory recopies are invoked after the initial wizard runs, meaning you can unfold the template incrementally as you go along. So in this case, you invoke the recipe to create a UIP page flow multiple times, once per flow you want to create. Running the recipe created a bunch of files, but of primary note was a DSL model file and two code generation templates. The model file was for a cool little DSL for laying out UIP page flows. The code gen templates generated all the code for the forms and the flow control as well as the config needed to implement the page flow as designed in the DSL. Then they wired up the generated forms to a web service, including usign a service agent to cache web service call results on the client.

All in all, it was a very full featured app to build in a very very short amount of time. What was interesting is that there was very little hand waving when it came to adding code. You know how demos go where they add literally pages of code? In this demo, they'd swap out the file with the empty method for one where the method had like ten lines of code. And on top of being full featured, it followed the best practices design put forth by out patterns & practices group. So it didn't fit the mold of a "quick & dirty" demo - how often does a demo app that you build on stage conform to best practices?

I gotta get through the end of this week, but then I want to get a video of the demo up on the web so you can see what I'm talking about, even if you didn't go to TechEd or attend the session. I'd love to get the code too - it's all running on VS05 Beta 2 - but you know product group guys...the next version of the demo is going to be even cooler...I think I can convince Jack to ship the current code and then ship the even cooler demo code when they get that finished.

TechEd 05 - Day Zero

The calm before the storm, as they say...

Things have been quiet around here between new baby & final TechEd prep. I think we're in pretty good shape, though we'll see how we look tomorrow. One good piece of news is that Dick Carlson, who manages the Hands-on Labs, sent out a list of labs either not received or that have blocking bugs. No ARC track labs on either list. That's a good sign...

Last night was the track owner dinner at Sea World. Funny, we had a track owner dinner at Sea World last year in San Diego. Is this a trend? It was fun to hang out with the other track owners and drink rather than have to sit around a table and plan. Didn't get to see Shamu up-close-and-personal like last year, but we did go ride the Kraken roller-coaster. Pretty cool, though reminiscent of Batman at Six Flags in SoCal. One slight bummer - some of us waited extra to ride in the front, but I was to big for those seats! And it wasn't a weight problem (though I could certainly use some work on that front), it was a barrel chest problem.

Today, my pal Chris is driving up from Hobe Sound and we're going to hang out this afternoon. I used to see Chris all the time, but then we both switched jobs and he doesn't have much chance to come out to the left coast. Last time I saw him was at an architect forum last spring in Orlando, when again he drove up to see me. However, I hear there may be a helicopter ride in the cards today, so I'll try and keep the "when are you coming to see me for a change?" grief to a minimum.

I've even gotten a chance to write up some thoughts on two new projects and to play around with the new DSL Toolkit CTP. Of course, having an extended battery for the plane ride was the reason I could do all that.

See you in the cabana on Monday...

TechEd Utility Player

So we're one week out on TechEd. This time next week, the final prep will be done and we'll be ready to let this thing fly. Some of the core team goes down this week, though as a track owner, I'm not really needed on site until Sunday.

What a difference a year makes. Last year, I was freaking out - it was my first TechEd. Now I feel like the old hand at this. Of course, I wouldn't have made it through last year with out the assistance of a few key individuals. Among others was Esther, who just started blogging for this year's TechEd. She's got a long post about the track cabanas. Last year, they were a big hit, but there were a few glitches. Esther writes about some of the changes they made this year to address those issues.

If you're going to TechEd, make sure to stop by the ARC cabana and say hi.

Finally, The New DSL Toolkit CTP

The May CTP of the DSL Toolkit is finally here, as per Jochen's blog. The big new feature is compatibility with VS05 Beta 2, but according to Jochen this CTP also includes:

• New features for the model explorer and property browser in the generated designers.   
• New shape type with collapsible compartments like in the Visual Studio Class Designer.
• New text templating (code generation) engine with richer features like an include directive.

They haven't updated the DSL Workbench pages yet, but you get the CTP directly from the Download Center. You can also leave feedback in the MSDN Product Feedback Center.

Introducing the Architecture Resource Center

Shortly after joining the Architecture Strategy Team, we worked with MSDN to re-launch the .NET Architecture Center. While we've had good success with that site, we realized after running it for several months that we needed a new approach in order to engage architects of all kinds. While Solution Architects are a key audience of ours that is well served by our MSDN site, there are also enterprise and infrastructure architects that we want to be able to engage with via the website. For these audiences, the MSDN site is not really the optimal channel. So today we've launched a new site - the Architecture Resource Center on microsoft.com. We've also re-launched the MSDN site, now called the MSDN Solution Architecture Center.

One of the key differences you'll notice about the sites is that we have a new way of categorizing our content. Previously, we used a topic based approach with categories such as service oriented architecture and application architecture. Now, we have a taxonomy with categories for think ahead, learn more, solve now and share ideas. This gives us a new way of differentiating content such as Metropolis, which really is about the future of architecture, from content such as the Smart Client Architecture and Design Guide, which really is about solving a specific design problems today. Over time, as we add both more content as well as more personalization features, we think this approach will make it much easier for our customers to find the content they are looking for.

Of course, as "community guy", I'm most excited about the Share Ideas section. In addition to a site wide RSS feed, we also have an aggregate architecture blog (and RSS feed)  featuring both Microsoft architects like myself and Simon Guest as well as and 3rd party architects such as Architect MVP's Jimmy Nilsson and Barry Gervin as well as Architecture Advisory Board members David Ing and Martin Fowler. There's info on the new architecture certification, upcoming events and webcasts as well as profiles of my coworkers on the Architecture Strategy Team.

Finally, I'm very excited to announce that JOURNAL is becoming The Architecture Journal and that you can sign up for a free print subscription to this great quarterly publication. We saw a huge traffic spike when JOURNAL was introduced on Architecture Center last spring, so I'm thrilled that that great content will now be available in print format delivered right to your home or office! Watch for print copies of the Best of the Journal issue at TechEd.

Of course, with any new venture, there will be tweaks, hiccups and improvements. Please leave a comment or email me with your thoughts, opinions and suggestions on how we can continue to improve this site to meet the needs of all kinds of architects.

Architecture Webcast Series

Coming in June, hot on the heels of TechEd is the Architecture Webcast Series. This is a series of four webcasts in June and July of architectural topics such as service orientation and reusable frameworks. In addition to the valuable content, you'll also get a free patterns & practices book and get entered into a drawing for a portable media center. Here are the webcasts:

Connecting Your Business with Service Orientation
This webcast invites you to delve into the Microsoft vision for service orientation and service-oriented architecture in enterprise computing. Discover the benefits that service orientation can bring to your business and how service orientation can play an integral role in developing connected systems. You’ll also become more familiar with Microsoft offerings and initiatives in the area of service orientation. And you’ll come away with practical guidance on how and where you should invest in the skills and technologies needed to capitalize on the short- and medium-term promise of service orientation.
Live Webcast: 10:00-11:00 A.M. PDT, Tuesday, June 14, 2005

Creating a Shared Technology Vision
Increasing business complexity and rapid industry changes are forcing IT and business executives to look for new ways of leveraging technology. Executives must become enablers for strategic innovation and sustainable competitive advantage (top line) rather than just operational efficiency (bottom line). The paths to business flexibility and innovation run through enterprise architecture that can allow companies to decompose their strategies, processes, and technologies into a set of services using proven theoretical frameworks and service-oriented architecture (SOA). Learn more about this effective and simple approach that can bridge the business-IT alignment gap and inspire the creation of the shared technology visions that can power real-time, event-driven network organizations of the twenty-first century.
Live Webcast: 10:00-11:00 A.M. PDT, Tuesday, June 21, 2005

The Value of Reusable Frameworks
Frameworks provide the building blocks of component-oriented application development through inheritance, abstract types, interfaces, and design patterns. Microsoft and Avanade have been working together to create a set of common application blocks and provide these blocks in an extensible and reusable manner. This joint effort has resulted in the release of Enterprise Library, a set of reusable software components designed to assist developers with common enterprise development challenges, such as Security, Data Access, Instrumentation, Exception Handling, and Caching. In addition, Avanade has extended Enterprise Library with a set of advanced application blocks for implementing Aspect-Oriented Programming and Service-Oriented Architecture called ACA.NET. This webcast will provide an overview of the architecture of Enterprise Library and ACA.NET, and discuss the design patterns that were used to implement the application blocks.
Live Webcast: 10:00–11:00 A.M. PT, Tuesday, June 28, 2005

Examining Enterprise Service Patterns
The demands you face to produce aggregated views of business information and automate business processes are more intense than ever before. You have to deliver on these demands to assist in the growth of the business while at the same time driving down the cost of IT by commoditizing integration, aggregation, and process management in a complex heterogeneous environment. In this session, we’ll examine a common set of enterprise service design patterns for commoditizing integration, aggregation, and business process management. We’ll consider J2EE, Microsoft .NET and host environments.
Live Webcast: 10:00–11:00 A.M. PT, Tuesday, July 12, 2005

Agile and DSLs Workship

I'm not sounding off in the pub, so I guess I'll blog the workshop that Alan and Steven Kelly from MetaCase are doing at XP2005. It's called "Agile Development with Domain Specific Languages" and here's the abstract:

This workshop will investigate the application of Domain Specific Languages within Agile development. A Domain Specific Language (DSL) is designed to express the requirements and solutions of a particular business or architectural domain. SQL, GUI designers, workflow languages and regular expressions are familiar examples. In recent years, Domain-Specific Modeling has yielded spectacular productivity improvements in domains such as telephony and embedded systems. By creating graphical or textual languages specific to the needs of an individual project or product line within one company, DSM offers maximum agility. With current tools, creating a language and related tool support is fast enough to make DSM a realistic possibility for projects of all sizes.

DSL Tools Web Chat

I'm still digging out my inbox after my two week vacation. I'm down to 134 items from nearly 400 this morning. I was at 16 before I left. If it gets this bad in two weeks, what's going to happen when I take my four week paternity leave?

While I'm getting up to date on email, I am way way way behind on blog reading. But I did see that Gareth is promoting a web chat about the DSL toolkit tomorrow at 9am Pacific time. Here's the abstract:

Using the Domain-Specific Languages Tools for Visual Studio 2005
Domain-specific language tools lay the foundation for software factories by providing a framework and a set of tools for delivering domain-specific visual designers that plug into Visual Studio Team System. These designers could be tools for industry verticals, such as the health care or telecommunications industries or, they could be tools for development across numerous disciplines, such as object-oriented modeling and architecture. The Microsoft Tools for Domain-Specific Languages is part of the Visual Studio 2005 SDK.

Now if we could just get a copy of the DSL toolkit that runs on Beta 2!

GAT is Available

Taking a momentary break from new dad duties to point out that the Guidance Automation Toolkit workbench is now live. In addition to the bits (both the toolkit and the extensions are available seperately) there's also an introductory article. Plus there's the on-demand webcast we did, now with a shorter URL.

Other People's Blogs

I try to spend more time writing original content for this site and less time simply pointing at other people's stuff. However, you'll notice the lack of updates around here lately. Things are busy at home - my wife and I are expecting our second child this week! So the dearth of content will continue for now...

However, while I remain too busy to blog, here are a couple of other new blogs worth reading:

• David Ing's blog From 9 till 2 is a must read. He asked for no gratuitous linking, but it's too good to keep secret. Not only does David have valuable insights, he's also funny. Check out his thoughts on SOA Versioning, Domain Specific Languages, Smart Clients and Wikis. But whatever you do, don't miss his post on the Cold War of OO->SOA, building on Ted's Vietnam War analogy of object/relational mapping.
• Wojtek Kozaczynski is an architect for patterns & practices and is heavily involved with the Guidance Automation Toolkit I blogged about two weeks ago. He writes in his introductory post that he's started blogging now to talk about the GAT, starting with a brief history of how it came to be.

Puget Sound IASA Meeting Tomorrow

FYI, I'm presenting at the monthly meeting of the International Association of Software Architects Puget Sound chapter Wed night (4/27). I'll be talking about DSLs and Software Factories, including a hands-on look at the tool. If you're in the Redmond area, the meeting is on the Microsoft campus in Building 43, Room 1560 - the Jefferson Room.

Introducing the Guidance Automation Toolkit

We had a good crowd for today's webcast - including my dad as it turns out. The session was on packaging design and architecture guidance for Visual Studio. In the session, Tom and Wojtek announced the newest p&p deliverable - the Guidance Automation Toolkit. Frankly, I'm very excited about this project so it was very cool that I got to guest host today's webcast.

The on-demand version of the webcast will be up soon, and there is no other info available on GAT as of yet, so let me summarize the purpose of GAT. Everyone is very familiar with the template solutions, projects and items that are in Visual Studio today. You select New Project from the menu and you get a choice selecting between things like Class Library, Windows App and Web App. Within one of these application, you get some contextual guidance - if you create a windows app, you get item template choices like Add Windows Form and Add Windows User Control but not Add Web Form or Add Web Service. GAT takes this templating capability to the next level while also making it much easier to develop the contextual guidance.

In the demo we saw today, Wojtek executed a solution template for a distributed application - similar to ones that come in the enterprise templates feature of VS today. However, the solution template Wojtek demoed has the ability to create the contextual features described above. For example, the solution template created an empty project for the data access layer. From that empty project, you could invoke a project template to create the data access layer. The reason it's important to separate out the step of creating the solution from creating the DAL is that you can ask further questions at this time. For example, you could collect info about what database the DAL connects to. You might even want multiple DALs - one for each DB you're speaking to OK, maybe that's not the best architectural model, but bear with me. The point is that it becomes very difficult to ask all those questions during the initial solution creation step. Breaking it out into separate steps makes it much easier for the developer to tailor the solution as he goes along.

In addition to having contextual templates that can be executed, GAT projects also have the concept of recipes that can be attached to any node the the solution explorer tree (folders, files, etc). These recipes can gather information from the user in a wizard and then carry out a set of arbitrary actions. In the demo, Wojtek created an entity object within his created DAL project. In the wizard, he pointed it to a table in a database. The recipe then retrieved the schema of the table, executed a template that read that schema, and placed the resulting file into the project. What's really cool is that you can create files that themselves have recipes attached to them. So you could have one recipe that created a baseline entity class from the schema - simply writing out fields and properties for the columns in the table. The create entity class could then have additional recipes attached to it to do things like create insert/update/delete methods or to create finder methods. Of course, often times you want to create multiple finder methods, so you'd want to be able to run that recipe multiple times. Note, recipes can do all sorts of actions - anything that can be automated in VS - so you're not limited to simply creating new files. The create finder method example would need to modify an existing file. You could also delete or move around files as you needed to. It's quite powerful.

Of course, all this power is useless if it isn't easy to leverage. GAT provides a strong guidance package authoring environment in addition to the guidance usage experience described above. It has several default recipe actions for common scenarios (execute template, add file to solution, etc) plus you can build your own. We didn't get into it in the demos today, but send us feedback and we'll see about scheduling further webcasts to get more info. Additionally, the GAT will be available in a few weeks so I encourage you to get it and play with it when it ships.

As I said, I'm very excited about this toolkit. It's another small step towards Software Factories. Still have a long way to go, but it's awesome to see tangible progress.

BTW, there will be both a GAT breakout and hands-on lab at TechEd. We didn't publish the titles yet since the project hadn't been announced. The breakout is being presented by Daniel Cazzulino who has been working on the GAT for some time now. He's also written the GAT hands-on lab. Watch Daniel and Tom's blogs for more breaking news on GAT.

Major congrats to Tom, Wojtek and Daniel and everyone else involved on GAT. I can't wait to get my hands on it and start playing around.

UPDATE - Looks like the on-demand version of the GAT webcast is live.

ANOTHER UPDATE - I fixed the on-demand link and removed the live link (since the event's over, what's the point of linking to the live webcast site?)

DMD 2 DD - Thanks Modelisoft!

Anyone who's played with the DSL toolkit knows that it's a 50/50 experience. Building the domain model means using a pretty straight forward designer (itself a DSL). Building a desinger means groveling around in a really ugly XML file. I hear that the designer design experience is going to get better in later builds, but for now it's all hand coding. If you work thru the DSL Toolkit tutorial, you spend a lot of time finding and replacing names of domain model elements in the designer definition file. But now, Modelisoft has a utility called DMD2DD (i.e. Domain Model Definition to Designer Defintion) that automates all of that nasty find and replace / hand coding. Once you build your DMD, you run their tool and it automatically adds reasonable defaults to the DD for new domain model elements and removes designer elements from the DD for domain model elements that have been removed. The DMD2DD tutorial is much more straightforward than the DSL walkthru, since you don't do all that hand editing.

Haven't tried it yet, but if works anywhere near as well as the tutorial makes it look I'm going to keep a sharp eye on Modelisoft. From their homepage, it looks like their primary business is generating and reverse engineering .NET code from/for Rational Rose. I wonder what they're doing w/ the DSL Toolkit?

System Definition Model SDK

Thanks to Pedro Silva, I now know about the DSL Tools forum on MSDN. There are also forums for VSTESA, MSF and VSTS Workshop. Plus, you can subscribe to RSS feeds of all these forums.

Speaking of the VSTS Workshops, the 3rd workshop (after DSL Tools and MSF Agile) just launched for the System Definition Model SDK. System Definition Model - or SDM - is the underlying model that powers the VSTESA modeling tools formerly known as WhiteHorse. VSTESA models 2 of SDM 4 levels - the Application Designer, the Logical Datacenter Designer as well as the Deployment Designer which essentially models the transformation between the two. These models map directly two the Application and Application Host layers of the SDM. Currently, we don't have modeling tools for the Network/OS and Hardware layers of SDM.

Barry Gervin recently complained that the Logical Datacenter Designer sucked because you can't model a web and database server deployed to the same physical machine. But in the context of SDM's layers, you model the deployment of multiple application hosts to a single OS instance one layer lower than we currently have modeling tools for. Alex Torone, PM for VSTESA (and presenter for ARC310 - the drilldown on the Logical Datacenter Designer) penned an extensive comment in response to Barry's complaints where he explains why SDM is factored the way it is and confirms that they are working on modelers for the other DSM layers "in future releases".

Anyway, back to the SDM SDK. What's cool about this is that while we ship some models "in the box" - primarily for our own stuff (SQL, IIS, WS03, etc) - there's no reason you couldn't model any system with SDM (Oracle, BEA, Linux, Z/OS, etc). The SDM SDK download is tools, samples and docs for building your own models and extending the modeling tools in VSTESA. (Note, the actual SDM SDK is a part of the VS05 SDK - i.e. what used to be called the VSIP SDK).

I wonder how long it is before the community models some of Microsoft's major competitors in SDM?

p and p Live! Webcast Guest Hosting

Last Thursday, I hosted the patterns & practices Live! webcast for Ron Jacobs. The session was on the Updater Application Block with Eugenio. I should have posted a link before hand - sorry about that. As soon as I find a link to the on-demand version, I'll post it.

Ron is still out this week, so I'm hosting another webcast this coming Thursday at 11am PDT. The session is on Packaging Design and Architecture Guidance for Visual Studio and I'm very excited about the stuff we'll be showing. Visit the MS Events site to sign up for the session.

Update - I located the on-demand version of the UAB webcast.

Various Modeling Info

I was in Barcelona all last week, so I'm behind on email and blogs. I missed a couple of great posts from various folks involved in modeling at Microsoft. Gareth blogged about the text template engine in the DSL toolkit. Ali Pasha blogged about the relationship between the Distributed System Designers and the System Definition Model. And R.Ramesh pointed to the second and third parts of the Channel 9 interview with John Stall from the Class Designer team.

BTW, if you want to keep up on all the goings on the VSTS world at large, you have to subscribe to Rob Caron's Blog.

Class Designer Demo

There's a great demo on Channel 9 of the new Class Designer for Visual Studio 2005. Note, as per the CD team blog, Class Designer is available in the standard, professional and team systems editions of VS2005. (i.e. everything but express). More info on the differences between the products in the VS05 product line are available on the VS05 web site.

KoolAid Drinker

I love this. Except Scott, didn't you mean to say "I'm an architect MVP"?

Crupi Blog

I met John Crupi - CTO of Sun's Enterprise Web Service Practice - at OOPSLA last year. Very cool guy. I hadn't realized he started blogging last month until Steve Vinoski linked to him. There seems to be some disagreement regarding using a top-down approach to SOA. Steve writes that SOA, like other "real-world development" is usually a mix of top-down and bottom-up. John writes that SOA is about a business driven architecture instead of an IT driven architecture. I liked this quote:

"[W]e cannot do SOA without a mutual effort between IT and the BU. Gone are the days of throwing the requirements over the fence and hoping it hits. Not only do these two groups have to work together, they have distinct roles and responsibilities. Basically, the BU runs the show and owns the business drivers, use-cases and processes. IT implements the BU requirements and owns the service definitions.  It's unfortunate that we really do have to refer to this as a "shift", because we should be doing this anyway. But, the reality is that IT and BU typical function as disparate groups and rarely work together to have the business use-cases drive the process and service definition."

John Evdemon made two points  relevant to this issue several months ago:

• SOA does not enable or ensure the alignment of IT and business. The IT industry has been promising this for decades – there is no silver bullet for aligning IT and business. Alignment of IT and business is an organizational issue that will not be resolved by an architectural design philosophy alone.

• Service Orientation will happen in your organization in one of two possible ways: chaotically (typical approach) or in a disciplined manner. The path your organization takes (and the cost of later fixing that path) is up to you.

I'm with John on this one. Traditional approaches to new problems are rarely successful.

DSL Toolkit Team Roundup

Talk about transparency: Pretty much everyone involved in the DSL toolkit is blogging. Keith, Jack, Steve, Stuart, Gareth, Alan and Jochen. Plus, I bug these guys on email alot and they always have answers. A couple of recent items from these folks:

• Steve was interviewed by DNJ Online about Software Factories. I didn't get to read this until yesterday afternoon, but I had to present the Microsoft Technology Roadmap at the Executive Briefing Center yesterday morning. We started talking about factories and one of the things that came up was the massive value of partial classes. Steve hit on this in his interview as well. As cool as yield, anonmous delegates and especially generics are, I now consider partial classes the most important new langugage feature in .NET 2.0. Funny, as it was probably the easiest to build.
• Gareth blogged about a very strange bug in the March CTP release involving the letter 'A' and a slightly hacky technique for dealing with blank files. He also talks about the model serialization format. Sounds like they'll be fixing both down the road a bit.
• Stuart confirmed that there will be a version of the DSL Toolkit for Beta 2 of VS05 "a small number of weeks" after it ships. He's also talks about some of the features in the next drop: better code generation and better support for containment hierarchies (i.e. classes contain fields and methods).
• Keith blogged a software factories elevator pitch and then refined the definition of a software factory into a single sentence.

QOTD: Paul Preiss

"Architecture is about how you decide not what you decide"

Thanks for the kind words Paul.

DSL Toolkit on CTP VS?

Doug wanted to know if the new DSL toolkit runs on the December CTP of VSTS. In a word, no. The March DSL Toolkit requires Beta 1 of VS05 Beta 1 or Beta 1 refresh. However, at some point, there will be a version of the DSL Toolkit that runs on VS05 Beta 2. ("At some point" meaning not on the day that VS05 Beta 2 ships but some short period of time after that.)

This is why I use Virtual PC. I can install the CTP build in one VPC and the Beta 1 build w/ DSL toolkit in another.

Update on DSL Toolkit and VC++

Yesterday, I mentioned that you need VC++ installed to use the DSL Toolkit. Apparently, it's needed for building a DLL related to VS integration. However, Gareth let me know that they should be able to do that with managed code in a future build which means they won't have a dependency on VC++. I didn't install VC++ because I was trying to save space in my VPC where I'm running the VS2005 beta 1 bits.

TechEd Bloggers

I guess we're still three months out, but TechEd Bloggers is just getting ramped up. Not to many registered bloggers yet. So far, I'm the only registered blogger in the staff category. Last year, I was staff and speaker (the only track owner also presenting I might point out) but this year I'm in marketing so they didn't let me speak! :) Actually, I picked all the ARC track sessions and speakers, so I have no one to blame for myself. Yep, no one to blame but myself for significantly easing my own workload to only worrying about organizing the track, hanging out in the ARC cabana and hearding speakers (note to self, get Ted's mobile number before the event starts) without the added worry of having to present.

Who else is going to Orlando?

New DSL Tool Drop Available

I've been doing a bunch of thinking about the use of heavily stereotyped UML class diagrams and how similar it is to the DSL approach. I'm working on a new post, but I wanted to parrot Gareth who announced earlier today that the March 05 CTP build of the DSL toolkit is now available. I'm installing it as I type. Well, not exactly true - I'm installing VC++. This drop apparently requires it.

Simulating Class Designer

It's Thursday, and the Class Designer team is sticking to their new entry every Thursday policy. I don't want to just post a link to their blog every week, but if they keep doing stuff this cool I'll have no choice.

Eugene Chigirinskiy could have just posted a screen shot showing tooltips at work in the class designer. But instead, he's built out an HTML image map on top of said screen shot and set up the areas with titles so that as you hover over areas the screen shot, tooltips pop up the same way they do in the tool. Awesome! I have no idea how long it took him to do this, but that's dedication!

Mr. CIO Guy Has Left The Building

As he wrote on his blog, Pat Helland's last day at Microsoft is tomorrow. He's busily cleaning out his office right now. :( He's starts on Monday @ Amazon to help them implement a service oriented architecture. I heard their CTO was hiring.  Pat's a big reason why I came to work for Architecture Strategy so I'm really sorry to see him go.

On the plus side, no one will quip "it's been nice hanging out with you" in the men's room anymore.

Seriously, Pat's been through some seriously hard times this past twelve months, and I think the change is a great opportunity for him. I imagine I'll see him often enough - he's back on campus next week to present at an architecture forum. Plus, I offered to setup dasBlog on pathelland.com so he'll keep blogging.

patterns And practices And podcasts

Ron from patterns & practices is not only blogging and webcasting but also podcasting. He's got two podcasts up so far. The first is a discussion with Billy Hollis about smart client architectures. It's pretty short - just enough to whet your appetite. (Quick plug - Billy is presenting with Rocky on Smart Client Architecture as part of the TechEd 2005 ARC track.) Ron's second podcast is with Scott Densmore on EntLib's ConfigurationContext. It's cool stuff, but it's alot harder to follow (for me anyway) topics related to code like this with just audio. Luckily, Scott blogged about this as well.

Modeling vs. Visualization

Javier got me thinking when he left a comment to my square peg model post. I'll get to intended semantics of class diagrams in another post, but I wanted respond to the latter part of the following comment:

I suppose you are assuming that the Class Diagram is intended to be used for design purposes. Then, a class diagram could be considered simply as a code visualizer.

That's a great point that ties into something I've been thinking about lately: the Class Designer in VS2005 is not a modeling tool, it's a visualization tool. For me, the difference comes down to abstraction and transformation. In VS2005, the Class Designer is built on top of the same metamodel as the underlying language. This means there is no transformation and that the diagram and the text of the code itself are at exactly the same level of abstraction. To me, you're not modeling unless you've raised the level of abstraction.

However, while I think VS2005's class diagram is a visualization, I also believe that UML's class diagram is a model. UML is not built on the same metamodel as the underlying language. It's at a slightly higher level of abstraction. That's why you can generate Java, C++, Ruby or C# from a given class model. That transformation step between diagram and code is what makes UML a model. Granted, the class model of UML is intentionally close in abstraction to the code, but it's still an abstraction.

The only reason it matters IMO if a given diagram is a model or a visualization is to be explicit about the need for transformation. And even the need or lack of transformation is only important for usage purposes. Each method has its pros and cons. UML can't model C# code as precisely as VS2005 can visualize it, but VS2005 can't be used to generate code for non .NET languages.

Javier's point cuts right to the idea that modeling classes "for design purposes" isn't particularly valuable as classes are such a low level abstraction. I think that's why so many people use UML's class model as a general purpose "thing" designer. The question is, what was the class diagrams intended use?

Speaking of the Class Designer

As I pointed out in my last post, VS2005 includes a Class Designer that those familiar with the UML class model should feel right at home in. By happy coincidence, I found out today the Class Designer team is blogging. Additionally, two of the Class Designer team members - Ramesh and Rakesh - are blogging on their own. Apparently, the team blog will feature a new post every Thursday. This week's post features an overview of the v1 Class Designer goals. Subscribed.

Putting a Square Peg Model in a Round Hole Tool

Robert Bauman left the following comment on my Separated at Birth post?

The nice thing about using a general purpose modeler is that you can house all of your requirements, use cases, etc. in the same model. Rational provides the 4+1 view, Sparx Systems Enterprise Architect provides several views out of the box that you can easily navigate around... It means that everyone is working off of the same set of rules.

As soon as you start putting those rules into Visual Studio, they change and deviate from the model. It's true that the AndroMDA does require you to remember to use certain stereotypes, but that's all part of the game anyhow.

That's like saying, "well, the GoF patterns are nice, but then you have to remember what it means to have an Observer pattern". Furthermore, UML tools let you customize the list of stereotypes that show in the dropdown, and even the picture that should be associated with those stereotypes. Why mess with some other modeling standard when you can do it all with a proper UML tool

The point I was making is that when you start using the class model to design something other than classes, you're using a domain specific language - even if you're using a general purpose modeling tool.  Take a look at this example from the AndroMDA website. Their example reads:

You tag a CustomerService class with a <<Service>> stereotype. AndroMDA sees this stereotype, looks into its internal dictionary of available code generation components (called "cartridges") and finds the EJB cartridge. In the EJB cartridge, two templates correspond to the <<Service>> stereotype: SessionBean.vsl and SessionBeanImpl.vsl. AndroMDA uses the internal representation of CustomerService loaded from the model, calls the processing engine twice, and two output files are generated: CustomerServiceBean.java and CustomerServiceBeanImpl.java.

In this example, classes with the <<Service>> stereotype actually generate two code classes - the Bean and the BeanImpl. But if we were using the class diagram as it was intended, wouldn't there be a one-to-one mapping between a class in the model and a class in the code? As soon as you break that one-to-one mapping, you're no longer modeling classes. A <<Service>> is something at a higher level of abstraction than a class - otherwise it wouldn't take two classes to implement it.

BTW, I'm not saying that there is anything wrong with this approach at all! I'm just pointing out the similarities between an approach that many people are using to achieve practical results with UML today and what you can do with the modeling tools that Microsoft is building.

The key difference comes down to tools. Yes, you can use the class diagram and stereotypes to model stuff at a higher level of abstraction like Services and Entities. But putting a square peg in a round hole like that has problems. Since you're not using the tools as they were designed, you have to manually enforce rules that the tool doesn't know about. Sure, you can add some semantics via stereotypes, but you can't take anything away. How easy is it to build a valid class model that isn't a valid service model? Pretty easy. For example, do services support inheritance? Classes do. My EJB is a little rusty, but I don't think beans do. It certainly doesn't make sense for a service to inherit from an entity or vis-versa. Yet, the class modeler will happily let you do this, even though it makes no sense in the domain you're actually trying to model.

The value of domain specific languages is that have a tool that is specifically designed to model the domain you're working in. If you're designing classes, of course you'd want to use a class model. We have a great one coming in VS2005. But if you're designing services or entities or page flows or whatever else, why wouldn't you want a tool that's specific to the problem at hand?

MDA And Software Factories - Separated At Birth?

Tonight I went to the monthly meeting of the local chapter of IASA. I should have also blogged this before the meeting, but I forgot. Sorry about that if you live near the Microsoft campus and wanted to go. Next meeting is on 3/30, so mark your calendars.

Anyway, tonight's topic was an MDA workshop featuring AndroMDA. AndroMDA is an open source tool for generating primarily J2EE code for *nix boxes using UML and MDA. (To be fair, the speaker - local chapter president Chris Sterling - demonstrated generating C# code as well. Of course, he ran it under Mono on a Linux box.) This provided a great launching point for a general modeling discussion that helped me get a few things straight in my head. Typically, the UML vs. DSL discussion turns religious pretty quickly. However, I believe that people - like those a the meeting tonight - who are achieving practical success with MDA in the real world are doing so by using a Software Factories style approach.

First off, if you look at how most people use UML for MDA, the class diagram appears to be the most dominant model used. When I say "UML for MDA", what I mean is people using UML as a blueprint or as a programming language. While UML has 12 different model types, class diagrams make up the bulk of the modeling effort. (The bulk of AndroMDA code generation works off the UML class diagram, though the BPM4Struts cartridge uses Use Case & State models as well) The other 11 diagrams are primarily used for sketching purposes. That means you're only blueprinting the structural aspect of your system - which in turn means that all the system's behavior has to be implemented by hand. Now, this is not to say that factories suggests you should only model the structural aspect of your system. However, I think this indicates that most pragmatic users have realized MDA doesn't live up to the hype.

Secondly, the class diagram that are used have to be heavily adorned with custom metadata - typically in the form of stereotypes - in order to be useful for code generation (i.e. blueprint) purposes. AndroMDA has a set of "cartridges" (essentially, target code generators) such as EJB, Hibernate and POJOs. Each of these cartridges has a supported set of stereotypes. While there is some overlap (for example, EJB and Hibernate cartridges both define the Entity stereotype). These stereotypes assign brand new semantics to the elements being modeled. In short, they turn the the generic class modeler into a domain specific modeler!

It appears to me that the pragmatic MDA crowd is using the class diagram as a generic "ball and stick" editor. Model elements that aren't needed are ignored and elements that are needed are added via stereotypes. For example, you can use a class diagram to model a database. Certain elements of the model are ignored (Can a column have protected visibility? Can one table inherit from another?) while other elements specific to the domain being modeled are added (primary and foreign keys, indexes, etc). The problem with this approach is that all of the knowledge of how to build a valid model is in the user's head, rather than the tool. Typically, that means a lot of training as well as a lot of in depth understanding of the framework underlying the model in order to capture the right amount of information. Since all that domain specific information is trapped in the users head, they have to do a ton of menial drudge work. It's different drudge work from things like writing tons of data access code, but it's drudgery nonetheless.

If you're going to need a tool specifically designed for your problem domain, why use a generic tool and a bunch of handwritten rules, when you can codify those rules into a domain specific language of your own? (I mean, other than the obvious "because the DSL Toolkit hasn't shipped yet")

Speaking of Steve

BTW, if you haven't seen Steve's entry on Isomorphism, go read it right now. He nails this whole services as contracts vs. services as code debate right on the head. The spot on nature of this post reminds me when he nailed the modeling problem on the head - another must read post if you haven't already.

Thoughts on Factories

Last week, I had a great discussion with the Product Unit Manager (or PUM) of VSTA. She wanted my perspective on a few things related to Software Factories and I figured I'd share some of them here.

First off, while I appreciate the vision of factories, I'm also focused on the short term gains of automating software construction. Today, most of that automation is in terms of code generation. For example, John asked the other day if I thought Yacc is a software factory. It certainly is a domain specific language! However, I'm not sure I'd go so far as to say it's a factory. But on the other hand, I'm not sure it matters that much if it's a factory or not. When the other John on my team blogged his thoughts on SOA, he included one I wrote: "Eventually we’ll stop talking about SOA and go back to talking about Architecture". I feel sort of the same way about factories. As long as we're talking about it as if it is something different from what we're already doing, we're not there yet. But if we keep taking steps in the right direction, eventually we'll get to the point where the process of building software doesn't look the way it does today. Sorta the same way that building software today doesn't look like it did pre-.NET, pre-VB, pre-Windows or pre-C++ (I could keep going, but I think you get the point). That's the thing about visions, you never really get there, it just provides a way to keep you going in the right direction.

Secondly, I think that one aspect of Software Factories that at least I haven't focused on is reusable frameworks. The book is called "Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools" but I think the focus has been mostly on models and tools. This is partially because of the whole DSL vs. UML flack (quick side note - how about we have both?) and partially because the DSL toolkit is the first factory-esque thing that we're shipping. However, DSLs big value, IMO, is to automate the construction of applications built on top of well designed reusable frameworks. For example, the OOPSLA keynote demo was a DSL that would sit on top of a UI process framework such as the p&p UIP block. But if there is not a good framework, there's little point in having a language. I've pointed out in the past that the big gap to cross for organizations to start using DSLs is the leap from building abstractions to building languages that automate that abstractions. However, that's not really true. The really big gap is the leap from building one-off abstractions to building reusable frameworks of abstractions. Once you have the reusable framework, building the DSL is an easier step IMO.

Even though I didn't figure out the framework / factory connection until last week, it must have been there in the back of my mind when I was working on the ARC track for TechEd. We're having a session on "Design Considerations for Enterprise Application Frameworks" with Steve Maine as the speaker.

TechEd Sessions Posted

I think TechEd 2005 is shaping up to be awesome again this year. They just posted the list of sessions (well, most of the sessions anyway). I can't link directly to the architecture sessions, but you can filter the list to show just one track at a time if you want to. We haven't got speakers or abstracts posted yet, but I can tell you that we've got some good speakers lined up like Scott, Ted, Roger, Rocky, and Steve.

Of the ARC track sessions, I think I'm most looking forward to "Building and Using a Software Factory". I jokingly call this session "Here's the Beef, Chris Sells". Chris blogged that he wanted a concrete example of a software factory. It turns out that we've been working on a concrete example since before OOPSLA. In the TechEd session, we'll see how far Michael, Jack and Keith have gotten since October.

PatternShare

About fourteen months ago, David Trowbridge of patterns & practices introduced me to a guy working in their testing group named Larry Brader. David is one of the primary authors of p&p's Enterprise Solution Patterns and Integration Patterns books. I wanted to talk to David about building a pattern repository and he handed me off to Larry. Little did I know that Larry is an information theorist and was one of the key authors of Testing Software Patterns. Frankly, when Larry gets rolling on info theory, I only understood a fraction of what he's saying. But the parts I do understand about how patterns relate to each other blows my mind.

Then p&p hired this guy...what's his name?...Oh yeah, Ward Cunningham. I hear he knows a bit about pattern repositories. :)

Anyway, around this time last year I was having regular meetings with Larry and Ward to talk about this repository stuff. Then stuff got crazy on my end - primarily being the acting marketing director for my team as well as the ARC track owner for last years TechEd. The regular meetings became more irregular and then stopped altogether. That is to say, my involvement stopped - Ward, Larry et.al. kept forging ahead. I heard about how things were going from time to time, but that was the extent of my involvement.

Last summer, Larry, Ward and David (plus others I've never met) published an article called Describing the Enterprise Architectural Space. (They also did a webcast on the topic.) In it, they laid out a way of thinking about how patterns relate to each other and they introduced the Enterprise Architectural Space Organizing Table (EASOT for short). That was just the first step. PatternShare is next one.

PatternShare is a community site that brings together the patterns from popular authors - Fowler, Evans, Hohpe & Wolfe, GoF, POSA and p&p - into a single repository. Furthermore, it provides a dynamically generated EASOT showing all the patterns in the repository and how they relate to each other. Finally, it provides a way to add new patterns to the repository so that they show up in the EASOT.

Major congrats to Ward, Larry, David and the rest of the p&p folks for pulling this off. I can't wait to see where the site goes from here.

Atlas Brand View, Wabi Sabi … and DevHawk?

Tanya, my cohort on the Architecture Strategy Marketing team, has finally - after much promising and subsequent delays - starter her blog. So far, just she's just written a hello world post where she explains why she named her blog "Wabi-Sabi".

Of course, long time readers are probably wondering: "Did he just say 'Marketing team'?"

Yes. Yes I did.

About two weeks ago, I was re-orged to be reporting to Norman. While you might expect me to be displeased about this, I'm really ok with it. First off, I was acting as the marketing director for about six months, so the fact that we now have three people doing marketing instead of just me is a huge bonus. Secondly, while I was acting as the marketing director (badly I might add) I was in total firefighting mode - no opportunity to do advance planning whatsoever. Now, I have the opportunity to focus on doing a great job on a few things well (like the TechEd Architecture track) rather than doing just enough to keep a ton of things from completely falling apart. It's taken a while to shift gears, but now I spend more time doing and less time running around like a chicken with my head cut off.

Finally, my move to the marketing team is inherently temporary. Yeah, it's fun while it lasts and I'm learning a ton, but don't think this is a long term career change. Before it even happened, Adam (head of Architecture Strategy) explained that he expects me to be over on the architecture side of the house "soon". Norman and I have already started planning that transition.

So get your "marketing slime" digs in while you can! (Apparently, we marketing slime prefer the term "marketing flacks".)

TechEd Session Triage

One of the reasons blogging has been a little light around here recently is because I was prepping for the TechEd Session Triage. Basically, all the track owners figure out what sessions they want on their own then come together for an entire afternoon to present their sessions to each other. Turns out there is often overlap with other sessions in other tracks that need to be worked out, suggesting changes to titles and abstracts and other wise cutting up. For example. I'm sitting next to Becky who owns the Connected Systems Infrastructure track. We had several sessions in both the CSI and ARC tracks that have caused changes and cuts. We actually got the ARC, CSI and DEV tracks got together earlier this week for a "pre-iage" which means today went pretty smooth for the ARC track. Not that it felt like things were going to go smooth running around this morning. Typically, the last 24 hours before triage are a flurry of emails with last minute suggestions. This year - no exception. But now, at least it's done.

DSL Toolkit Known Issues

Jochen just posted a list of known issues for the Dec 04 CTP release of the DSL Toolkit. I have not had enough time to play with this. Anyone know where I can get 3-4 more hours a day? :)

DSL Toolkit Walkthru

Jochen and Stuart both blogged that the walkthrus for the December release of the DSL toolkit are now available for download. Given the relative lack of documentation, this is a good thing.

Concurrency: Next New Major Language Feature?

Several people have pointed out Herb Sutter's great article on concurrency entitled The Free Lunch Is Over. When I blogged last week about new possible features of "full-grown" OO languages I mentioned dynamic typing but I didn't think about concurrency. I think Herb is right: "programming languages...will increasingly be forced to deal well with concurrency" as applications get more CPU bound. Maybe I need to take another look at Comega (or Cw). Cw extends C# in two areas - data typing/querying and concurrency. The concurrency extension used to be called Polyphonic C#, but the name got changed when it merged with Xen/X#. (BTW, there's a new Cw release (v1.0.2) but no specifics as to changes other than no longer needed VS.NET 2003 to be installed in order to use it.)

Cw adds the idea of asynchronous methods and something called chords - sets of methods with the same method body. The chord method body in only executed when all the associated methods have been called. In the simple buffer tutorial, the buffer class has a synchronous Get method and asynchronous Put method. If you call Get before Put, it blocks until Put is called, then the method body is executed. If you call Put before Get, then the Put call returns immediately (it is async after all) but the call is queued so that when Get is called, the method body is executed immediately. FYI, the Cw docs have a variety of other tutorials of async methods and chords.

BTW, speaking of my post on full grown OO languages...My father suggested that I not jump to conclusions regarding the X-develop's support for what they term "toy languages or little domain specific languages". In fact, Hans Kratz of Omnicore (which makes X-develop) had this to say:

This comment on our website was not intended to bash DSLs at all. Instead we wanted to make clear that the plugin API in X-develop is powerful enough to allow integrating support for "full-grown" languages without placing arbitrary restrictions on language complexity.

For a language developer/integrator this is a plus regardless if he wants to integrate support for a DSL or "full-grown" programming language.

Makes sense. Maybe I was just too sensitive to the use of the word "toy" so close in proximity to "DSL". Sorry about that Hans. 

New SSB Library and Community Site

A couple of weeks ago, Paul Murphy pointed me at a SSB Wrapper Class that's included in one of the SQL 2005 Beta 2 sample apps. That prompted an email from the SSB dev lead with a much better SSB library that they had written. You can get that library, along with two new samples that use it, up on the SQL Server Service Broker Developers Spot. The SSB Dev Spot is a community site dedicated to apps written on top of SSB. It's run by Dan Sullivan of Developmentor, who is also hosting his new blog on the site. I hear Dan and fellow DM-ers Bob Beauchemin and Niels Berglund are all very excited about SSB. Nice to know I'm not the only one.

New Features For "Full-Grown" OO Languages?

After seeing it on TSS.net, I checked out the X-develop product website. X-develop is a new multi-language cross-platform IDE that supports both .NET languages like VB.NET, C# and J# as well as Java. It includes support for all the .NET 2.0 features like generics as well as new Java 5.0 features like generics, varargs, enums and boxing.

When I checked out the website, I noticed a section covering "Custom Languages". Here's another example of bias against DSLs - this time in favor of "full grown OO languages":

"The [X-develop] language plugin API is not limited to toy languages or little domain specific languages. Instead it supports full grown OO languages. In fact the support for Java, C#, J# and Visual Basic.NET is implemented as language plugins." (Emphasis added)

This reminds me of the same kind of bias that was leveled against classic VB. Classic VB was valuable primarily because it was limited in scope and capability. But what it gave up in capability it gained in productivity. Along the same lines, DSLs are valuable because they are little, because they only focus on a single problem domain.

What exactly would be the value of another "full-grown OO language"? What features would be in a new full grown language that isn't already in Java/J#, C# or VB.NET? The only feature I can think of is dynamic language support such as type inference and duck typing a la Ruby and Boo. What non-domain specific language features are you still waiting for?

The Pharonic Architect is Blogging

Speaking of DSL Tools, Software Factories co-author Jack Greenfield just started blogging. He has jumped into the running debate between IBM's Grady Booch and a variety of MS folks including Steve, Alan and myself. He does a good job summarizing the argument including pointing out where he and Booch agree:

In particular, we share the conviction that packaging knowledge for reuse in patterns, languages, frameworks, tools and other form factors is “the right next stage in cutting the Gordian knot of software”.

Jack rightly points out that while we all agree on these mechanisms for packaging knowledge that the devil is in the details. I look forward to seeing more on these details from Jack in the future.

Early Xmas Present

Maybe it's not as exciting as what my 2 year old has on tap for Xmas, but some of my Xmas came early when I got the new release of the DSL toolkit. The website isn't updated yet, but the download is available.

This release is way beyond what the team shipped right after OOPSLA. For example, the code generation engine is in there now. So where you could only design the concepts of your modeling environment in the last release, now you can build a set of concepts in what is now called the Domain Model Designer, design a notation for that model using the Designer Definitions file (visual designer for this file to be a part of a future release) and build a set of code templates that can in turn generate code during the compilation process. Very cool.

I've only played with the updated walkthrus so far, but I'm pretty impressed. Major congrats to the team for getting this out. I saw Jochen in the cafe on Friday and discovered that the entire team has moved to building 25 which is right across the street from building 18 where my office is. Now I can go bother Jochen, Keith, Jack and the rest of the team without having to track all the way across campus! :)

Spelunking Service Broker - Dialogs

The simplest way to describe SQL Service Broker is "message queues in the database." If the queues are tables in the database then you can do your entire message processing using a local transaction. If you use a separate queuing technology like MSMQ, you can still pull items off the queue, update data in the database and send out messages in the scope of a transaction – it just has to be a distributed transaction. Being able to use a local tx gives us a big performance gain. But, while important, this perf gain isn’t the most compelling reason to use SSB. It turns out that SSB provides important semantic messaging benefits in addition to perf benefits.

If you study at the semantic messaging model defined by message queuing systems such as MSMQ, MQ Series and WS-RM you’ll notice that it is inherently one way. Section 2 of the WS-RM spec defines the reliable messaging model to be between a source sending the messages and a destination receiving them. The problem with this model is that as message patterns between services gets richer, we’re going to want bi-directional reliable messaging. SSB calls this a dialog.

Before you can send messages between services in SSB (assuming everything's been configured), you first have to BEGIN DIALOG and provide the initiating and target service, plus the message contract (more on that in a later post). Note that it's called initiator and target, not sender and receiver. Either side of the dialog can send messages to the other as part of the contract by using SEND ON CONVERSATION - the only requirement is that the initiator sends the first message (hence the name "initiator"). All messages are guaranteed to be delivered exactly once and in order, or both sides of the conversation are notified of the failure and the dialog is torn down. I'm not sure if it guarantees in-order delivery of messages in opposite directions. There are cases where this might be important - I send an order cancellation as I send you a shipping notification - but typically there's a business reason for who "wins" such a conflict. In this shipping/cancellation example, even if you sent me the cancellation before I sent shipment notification, it's not like I can recall the shipment.

My only issue with this bi-directional reliable messaging is that I'm so used to thinking in terms of one-way RM that it's taking me a while to wrap my head around it. Many of the sample interactions I come up with are simple patterns where the target simply acknowledges the incoming message. Order processing is a good example where I may get meaningful messages (i.e. not simple acks) coming from both ends. What are some others?

SSB Wrapper Class

Paul Murphy pointed out a SSB wrapper class that ships as part of the integrated storefront sample that ships with SQL 2005. It's a bit tricky to install, so I refer you to Paul's blog for instructions. (you have to install the sample installer) The wrapper class is in a file called ServiceBroker.cs and supports the following operations:

• Get Conversation Group
• Begin a Dialog
• Begin a Dialog (with related conversation)
• Send a Message
• Receive a Message
• Receive all messages from a conversation group
• End a Dialog

If you're not sure what dialogs and conversation groups are, I'll be blogging about them in the coming weeks.

Booch on DSLs (Round 3)

Steve Cook responded to Grady Booch's latest comments on software factories and DSLs, which was in turn a response to an entry by Alan Wills. It's obvious that Booch is never going to agree with the DSL approach, but there are a couple of fascinating elements of the exchange.

First off is Microsoft's "rejection" of UML, which is just plain FUD. I'm behind on news reading (as usual) but I was clued into this conversation by a post on TSS.com that linked to Booch's blog and read "Grady Booch explains why he disagrees with Microsoft's rejection of the UML in favor of proprietary domain-specific languages." Not exactly priming the pump for intelligent discourse on the subject. As Alan and Steve both point out, we're not rejecting UML. UML is a tool, and like all tools it has things it's good at and things it's not good at. We're talking about using DSLs for things UML is not good at.

The next is Common Semantics. Every time he posts on DSLs, Booch mentions something about "covering the same semantic ground as the UML". I promised Dan in my comments that I would summarize this argument "in simple English" and I never got around to it. The term semantics simply means "meaning". For example, in C#, the keyword "class" always means the same thing - it has what Steve pointed out is "objective semantics". Obviously, the word "class" has widely varying semantics in common use - when a high school senior cuts class he's not copying an object definition to the clipboard. But within the realm of C#, the word "class" has a well understood and precise meaning - it's so precise that the C# compiler can tell you if you use it incorrectly.

So when Booch writes about a "common semantic model", I take that to mean that he thinks there's a core set of well-defined concepts that all languages build on. And if that's what he means, I imagine he assumes all the interesting concepts are already defined by UML. I think that's where the primary disagreement lies - we don't think any one language can provide all the possible concepts needed for all programming domains. A language for developing a web app page flow will be built on very different semantic concepts that a language for developing telephone billing systems. Trying to build both of them on top of the same set of concepts is like putting a square peg in a round hole.

Furthermore, even if you wanted to build on a common set of concepts, it's not clear if UML provides a precisely defined set of concepts to build on. Obviously, Booch thinks it does, but there certainly isn't agreement in the industry. Steve refers to UML as having "cognitive semantics", which means there is no one objective definition for a specific element of UML. For example, in covering Aggregation and Composition, Fowler refers to UML's white diamond aggregation as a "modeling placebo" and having "no standard meanings". When there's no standard objective meaning, then each person brings their own experience and reason in order to formulate their understanding - hence the term "cognitive". Of course, the chance that any two people will reach the same understanding via cognitive reasoning is slim to none - there's just too much room for personal interpretation. Because of this lack of objective precision, Steve describes the resulting discussion of UML and its semantics as "political, rather than objective" which IMO is not a good foundation to build your own language on.

In the end, the proof is in the pudding. Personally, I think Booch looks at UML with rose-colored glasses and that his beliefs don't mesh with reality. (Of course, our DSL modeling tools and software factories approach isn't far enough along yet to test against reality.) How about your experience? What success or failure have you had with UML?

Be There or Be Square

PDC 2005. 'nuff said.

TechEd 2005 Call for Papers

BTW, speaking of TechEd 2005, we're currently inviting potential presenters to submit proposals to speak. Norman is taking over ownership of the Architecture track, but I'm still responsible for technical content and community for the track. If you're interested in presenting at TechEd on any topic, head over to the TechEd 2005 Call For Papers website. Proposals are due by the end of December.

Starting in on SQL Service Broker

Since Norman joined the team a few months ago, I’m no longer in firefighter mode. For about six months my team was short a marketing manager and I ended up picking up a bunch of extra duties. For example, I took over as TechEd architect track owner when out previous marketing manager left. I don’t know much about marketing, but I guess I did OK – I received a Marketing Impact Award for my work on TechEd. However, I’m very happy to have handed those marketing responsibilities back to their rightful owner. Even still, Norman is trying to educate me about marketing. He made me read the first two chapters of Building Strong Brands by David Aaker. I actually got thru the first three chapters before borrowing Birth of the Chaordic Age by Dee Hock from my father at Thanksgiving.

You would think that I would now have more time for blogging, but alas that has not been the case. When I was firefighting, I had no time for planning. Now, of course, I do. So between planning and thanksgiving vacation I’ve just been too busy to blog much.

One thing I’m getting into recently is SQL Service Broker. I’m working on an interesting community project that is building on top of SSB. Luckily, one of the primary architects of SSB sits down the hall from me. Of course, not everyone is so lucky, so watch this space as I dig deeper on SSB. A good place to start is the SSB First Look article. In order to start getting a handle on SSB, I ported the Hello World example from that article from T-SQL to C#. It’s a bit tricky, as there is no SSB-specific framework – you just use SqlCommand to execute SSB commands like BEGIN DIALOG and RECEIVE – but otherwise it’s pretty straightforward. My sample also demonstrates using the new SQL Management Objects to create databases and SSB related objects (message types, contracts, queues and services). Here’s the code – enjoy.

More Architect Bloggers

Two new architect bloggers to report. Javed Sikander is an architect on the Architecture Strategy Team focusing on RFID. He links to a pair of articles about Microsoft and RFID plus provides his thoughts on why RFID is a big deal. David Solivan is an architect advisor and an ex-teammate from my old .NET Adoption Team days. So far, David appears to be blogging at conferences. He started in July at MGB and then went dark for a few months. He picked back up at SAF. I hear he's at a conference this week, so maybe we'll see some more from him.

That brings the total number of bloggers from my team to 14. That's over half of the team and I know at least two more coming down the pipe.

Roger Sessions on WS-*

Roger Sessions, noted architectural guru, author and Microsoft Architect MVP, has posted his latest newsletter on the WS-* family of specs. In typical Roger fashion, he decides to give them his own name - WS-SCRAM with SCRAM standing for "Secure, Coordinated, Reliable Async Messages". (I forget where, but I once heard Roger refer to "three-phase transactions" instead of the relatively standard "two-phase commit transactions".) Roger claims MS et. al. are pushing WS-STAR where "STAR" is an acronym standing for "Secure, Transacted, Async, Reliable". Personally, I've never seen it written out like a an acronym and I always thought the * in WS-* was used as a wildcard, but it is true that we are focused more on transactions than coordination. On the Longhorn DevCenter, Indigo is described as providing "secure, reliable, and transacted messaging along with interoperability."

This is actually the second time Roger's taken on transactions in a web services architecture. His last newsletter on the topic had a much more detailed but harder to follow example. He points out that there are two related specs in this space - WS-AtomicTransaction and WS-BusinessActivity but that he thinks only WS-BA is going to work in the long run. WS-AT requires holding locks open in the DB, which is highly unlikely between services in different trust boundaries communicating with async messaging. Am I really going to lock the records in my database while I wait for a service that I don't trust to figure out if it is able to commit or abort? I don't think so. Pat's wrote a great scenario showing how unrealistic the concept of long-running transactions really are.

However, at the end of the newsletter, Roger takes Indigo to task for implementing WS-AT and not WS-BA and I don't agree with him. While I think WS-AT shouldn't be used in web services architectures, Indigo is also responsible for moving existing technologies like COM+ forward. Leaving out WS-AT isn't really an option in those scenarios. As for not implementing WS-BA, I've got to wonder just how valuable is WS-BA? While WS-BA avoids the DB locking issue of WS-AT, it still doesn't deal with the delegation of control. WS-BA still leaves me beholden to the decision of some outside coordinator. This seems to violate two of Don's tenets of service orientation: "Boundaries are Explicit" and "Services are Autonomous".

Another Model Blogger

Gareth Jones is another blogger from the VS modeling team. He's got some great posts on using OneNote as a shared whiteboard and the use of color-coding in modeling. Given the fact that a popular use of modelling today is for sketching, it's interesting to read about how the team is looking at bringing those idioms forward while also making the models precice enough to be used as development artifacts.

BTW, no problem on the image “borrowing” Gareth. :)

DSL Tools Now Available

We announced it last week, but the DSL Tools home page just went live today. The big thing is the Microsoft Tools for Domain Specific Languages Technology Preview. There's also a walkthru of the tool, a newsgroup and Jochen's blog. We also published the third article in the Software Factories overview series.

The DSL Tools is the first in a collection of "workshop" projects that relate to Team System. These projects will be run as transparently as is feasible, so watch that site to be pretty dynamic as we post things like specs and lots of interim builds. Scoble has talked in the past of transparent projects, this I think will be a much more transparent project than we've done in the past.

(BTW, if you want to try out the DSL Tools, you need VS2005 Beta 1 as well as the VSIP SDK 2005 Beta 1)

Other MSFT Bloggers at OOPSLA

Some other 'softies blogging OOPSLA:

• Keith Short talks about the DSL Tutorial and apologizes for our keynote demo.
• Steve Cook talks about the MDA panel and says the keynote demo "wasn't such a great idea"
• Stuart Kent describes the DSL Toolkit and calls the keynote demo a "commercial break"
• Michael Lehman briefly describes his work building a software factory for the DSL Tutorial and blogs Jack's GPCE keynote presentation on Software Factories

Finally, Brant Carter took me to task for pointing out that Alan Kay ended his Turing Lecture with a product demo. Brant makes good points and I agree with all him. I think all the 'softies blogging OOPSLA have acknowledged how poorly this went and I certainly didn't mean to imply that somehow we were vindicated by Alan's demoing stuff in his lecture. I plan on giving Squeak a deep look when I get some free time - I have an 18 month old son so Alan's lecture on teaching computer science resonated with me.

Building DSLs in VS

Last night, after the Turing Lecture, we hosted a FlashBoF on "DSL's in Visual Studio". Stuart answered a bunch of questions and gave a much more detailed demonstration of the new DSL Toolkit than we could show during the keynote. Here's what I learned from the session:

• Models are stored in XML files. The language designer outputs an object model and will eventually also output an XSD. For example, here's a screenshot of the language designer from the DSL Toolkit we're releasing. Inside the designer, I've got a sample UIP DSL (I hacked this up on my own, this is not exactly the same one we demoed yesterday). As you can see, there's a PageCollection concept which contains Page concepts that have Name and Kind values. Page concepts also has a collection of Transfer concepts, which in turn have Label values. Generating an object model makes it easier to write tools that manipulate models. Typically, I'm anti-XML-Serialization but in this case - where we have a relative simple XSD - it works fine. I could also manipulate the model by accessing the underlying XML if I want to.
• Code generation uses templates and looks a lot like CodeSmith or old-school ASP. You interleave the static elements of the generated code with blocks of code that access the model (via the object model described above) and generate the dynamic model-specific elements of the code. So I'm guessing that people using the codegen tools like CodeSmith will feel right at home with this toolkit.
• In the current builds (which is to say later than the build that we're releasing first - the first build doesn't include any of the code generation support) we're generating a single code file from a model. Eventually we'll be able to manipulate multiple files from a single model. This is similar to how the Class Diagram works - add a new class onto the diagram and a new file gets added to the project, delete the class from the diagram and the file gets removed from the solution.
• Not all models are used to generate code. For example, in VSTS the Logical Data Center and Virtual Deployment models don't generate code. They are useful  because I can use them to validate the Distributed System Model which does generate code.
• Someone asked about the implications of code coverage, profiling and test-driven development on a DSL-based process. Frankly, I don't know but it certainly got me thinking. The general consensus was that we're still in the bootstrap phase of making DSL-based development a reality and these are issues we'll have to deal with as we move forward.

OOPSLA Day 1

This morning was Rick Rashid's keynote with "the big announcement": a framework and tools for developing Domain Specific Languages in VSTS. Essentially, this is a toolkit for building your own modeling tools on top of the infrastructure that powers the VSTS Distributed System Designers (aka WhiteHorse). Personally, I think this is pretty exciting stuff and I can't wait to play with it more in the coming months.

As part of the announcement, they demoed the toolkit with a DSL for programming the flow of pages in a web application. Typically, you would use a framework like User Interface Process for this. However, the UIP comes with 130 pages of documentation and a variety of classes and configuration settings. By building a DSL on top of the framework, we can make the developers significantly more productive. I've posted a screen shot of the designer - it's essentially a box-and-stick modeling environment - boxes are pages and sticks are transitions. Dragging a new page onto the design surface from the toolbar generates a new web page in the solution. Dragging a transition onto the design surface changes the config file. That's much more productive than mucking around in source and config files to accomplish the same task.

Of course, the standard question when we announce stuff like this is "when can I get my hands on it?" We're going to release a tech preview version of the DSL Toolkit "real soon now". You'll be able to download it from the newly minted VSTS Workshop page. I hear that this project, now that it's public, is going to be very transparent to the community, so keep an eye out for more info (of course, I'll blog it when the toolkit is released). Honestly, the version we're shipping "real soon" is much rougher than what we previewed in the keynote this morning, but we'll be updating it periodically as it marches towards release.

I've had several good conversations in the booth with people wanting more info about VSTS and the DSL toolkit as well as Software Factories in general. As is typical, I learn as much as I teach in these sorts of discussions. For example, I met Gan Deng from Vanderbilt who showed me GME - the Generic Modeling Environment - which is DSL modeling environment similar conceptually to what we announced today for VSTS. Not sure when I'm going to get to play with it, but it's nice to see academia support for DSLs.

The only downer is that some of the OOPSLA crowd was put-off by a product demo during the keynote. Stuart described it as a "commercial break". Several people gave me "brutal feedback" that brazen marketing displays like product demos don't work in OOPSLA keynotes. But others thought it was cool. Good to keep in mind for next year.

UPDATE - Alan Kay, in his Turing Award Lecture, made a joke about about our product announcement this morning and everyone laughed, including me.

UPDATE 2 - After making a joke about our morning's product demo (and several other assorted jabs), Alan Kay ended his Turing Award Lecture with...wait for it...a product demo. Two product demos, actually. Squeak is educational programming environment, based on Smalltalk. It is freely downloadable and has an associated community and wiki. Alan did a simple little demo where he drew a car and a steering wheel and was able to steer the car as it moved by rotating the steering wheel. Cool. Alan's other demo, Croquet is a "massively multi-user virtual learning environment" which is built on top of Squeak. While I liked Squeak, I don't know many people who spend significant time in a virtual 3D space who aren't gaming. Frankly, Croquet doesn't seem particularly practical or useful - check out this screenshot of document collaboration in Croquet. In this example, the 3D space is more of a distraction than an enabler. The only place Croquet seems useful to me is for dynamic 3D simulations, such as this screenshot demonstrating the Coriolis effect.

OOPSLA Welcome Reception

I called it an early night yesterday (which explains why I'm blogging this today instead of last night) but I did go to the OOPSLA welcome reception. Ran into Martin again who pointed me in Gregor's direction. Gregor presented at TechEd last year and was one of the top five speakers in the architecture track so we chatted about doing it again at next year's TechEd. (I also got a chance to meet EIP's co-author Bobby Woolf). Then Martin introduced several other attendees:

• Joshua Kerievsky, author of Refactoring to Patterns
• Michael Feathers, author of Working Effectively with Legacy Code
• Gerard Meszaros, who runs XUnitPatterns.com
• Mark Striebeck, Director of SourceForge Engineering

We all stood around chatting for a good while. I learned quite a bit and now have to add a few books to my very long reading list. I hadn't realized that SourceForge is a product in addition to being an OSS code repository. Actually, their aren't exactly the same - for example SF.net is built on PHP/Perl/Python while SF Enterprise Edition is built on J2EE. As we all started to drift off, Mark mentioned that he appreciated talking with me since didn't always talk in "glowing terms" about MSFT. Hey, we do a lot right, but that means there's always areas we can improve on and I always err on the side of honesty and transparency.

OOPSLA Day 0

Day 0 is almost done. Like yesterday, it's been quiet, but the foot traffic by the booth is improving. I missed the afternoon of the DSL Tutorial tracking down a shipment of JOURNALs that we are handing out at the conference. Martin Fowler stopped by to say hi and we chatted a while about the Architecture Advisory Board. He missed the meeting a couple weeks ago because of a communication screwup (i.e. I didn't call him) but he's looking forward to working with us going forward. Some one ribbed him for sitting in the Microsoft booth, so he pulled out his PowerBook (or maybe it was an iBook). I told him that since he like UNIX so much, we should install Services for UNIX on his Windows laptop. :)

This is a very different crowd from TechEd - not better or worse, just different. Very heavy academic presence. I got a very interesting code visualization demo from an MIT grad student. Flipping thru the conference program, the vast majority of the speakers are from universities. The majority of the remaining speakers are from IBM and to a lesser extent Sun. There's a small Microsoft presence among the speakers - primarily related to Software Factories. It's interesting, however, that three of the six of the invited speakers this year are from Microsoft.

By Request: My Response To Booch's Doubts

Mac asked me to respond to Booch's doubts about DSLs in his blog. This is at least the second time Booch expressed his skepticism about DSLs, having also done so back in May. The upshot of Booch's argument is that the semantics across all stakeholder viewpoints must be common.

"There is no doubt that different domains and different stakeholders are best served by visualizations that best speak their language...but there is tremendous value in having a common underlying semantic model for all such stakeholders. Additionally, the UML standard permits different visualizations, so if one follows the path of pure DSLs, you essentially end up having to recreate the UML itself again, which seems a bit silly given the tens of thousand of person-hours already invested in creating the UML as an open, public standard." (Grady Booch, Expansion of the UML, May 21st 2004)

"[W]hile I agree that development is a team sport and that multiple stakeholders must collaborate in weaving together their diverse, interdependent views, one still needs to have a common semantic basis for all those languages. If you accept that not unreasonable position, you will end up covering the identical semantic ground as has the UML - albeit in an open manner, quite unlike Microsoft's historical record." (Grady Booch, Domain-specific Languages, Oct 25th, 2004)

Stuart responded to this argument back in May:

"[Booch] seems to imply that a domain specific-language is just about having a different graphical notation - that the semantics that underpin different DSLs is in fact the same - only the notation changes...[W]hat if the semantic model excludes the concepts that the stakeholders actually want to express?...Surprise, surprise, there are differences from one domain to another, from one organization to another. Far from there being a common semantics, there are significant and actual differences between the interpretations of models being taken." (Stuart Kent, Domain Specific Modelling. Is UML really the best tool for the job?, May 26th, 2004)

In other words, Booch's argument against DSL's falls apart if we don't have common semantics across all phases of the system's lifecycle. And frankly, we don't. Furthermore, Booch offers no explanation as to why you need a common semantic model across all the stakeholders, instead choosing to call it a "not unreasonable position". Sorry, but I'm not ready to accept the need for a common semantic model as an axiom. I will accept that you need traceability between viewpoints, but I think that is more readily achievable by precise transformations between viewpoints than by a common semantic model of all stakeholder viewpoints.

Common semantic model starts to sound too much like "monolith" as far as I'm concerned. UML 2 addresses a subset of the stakeholder's viewpoints and many people consider it unusable. After last year's OOPSLA, Fowler commented "Even on the MDA panel at OOPSLA, the pro-MDA speakers based their assumptions on the fact that they would be using a simplified subset of UML". If we can't get a single usable semantic model across UML 2, how are we going to have a single usable semantic model across all the stakeholder viewpoints? Booch paints a pretty rosy picture of UML , but I think the reality of UML is much less attractive, certainly according to the customers I've spoken to.

OOPSLA DSL Tutorial

I'm sitting in the back of the room listening to Jack present the Factories overview for the DSL Tutorial. I've seen most of the content for this session before, but not brought together like this. Currently, he's talking about mapping business capabilities to system requirements. I've started to see quite a bit of discussion of mapping business capabilities recently. Gurpreet talked about it in his EA session n Malaysia. Jack Calhoun also talked about it at SAF. Memo to self - more transparency around business capability mapping.

BTW, check out Michael Lehman's blog. Michael has been building (it's not really done) the factories demo for the tutorial. He's got some great points about DSL's, abstraction and product lines. As I type this, he's demoing expanding an e-commerce factory template inside of VS.NET. Cool stuff. I expect him to be blogging a lot more about building factories after this week.

OOPSLA Day -1

When I blogged TechEd, I started on Day Zero, the day before TechEd officially opened. At TechEd, that day is used for preconfernce sessions as well as other meetings. At OOPSLA, there are two days of preconfernces - I arrived today but the conference doesn't officially open until Tuesday with Rick Rashid's keynote. I guess that makes today "-1". So far, it's very quiet around here. I've got booth duty until 5pm, but I doubt I'll spend much time with attendees. There's only about 15 attendees in the exhibit hall right now. All the booth staff are chit-chatting with each other or are working on their computers. I'm sitting with Ajay, a product manager from VSTS emailing, chatting and deailing with some last minute shipping issues. (My group has just had really bad luck sending stuff recently). Of course, I'm guessing most people are in their precon sessions. I stuck my head in Jack's session on Generative Software Development and it was full. Tomorrow is the DSL Tutorial that Keith blogged about. By then, I'm thinking things will be in full swing.

Two Down, One To Go

It's been quiet around here as the last three weeks of October are insane for me this year. Two weeks ago was Strategic Architect Forum as well as the second face-to-face meeting of the Microsoft Architecture Advisory Board this year (the first was reported in the February Architecture Center Update newsletter). I'll have much more to say on the MAAB later, but having SAF and MAAB the same week is tough - I worked seven days in a row including four 12+ hour days.

This past week, I was in leadership training with a bunch of teammates and p&p folks. Spent quite a bit of time “in the circle“ with Ed, Ward and Jim from p&p plus David, Chris and Javed from my team (note to self - lean on Javed to get a blog). The sessions were intense and I learned quite a bit. I agree with Norman (who was also in the training but not in my group) that we're very fortunate to work for a company willing to invest so much in it's employees. However, it was two more 12+ hour days and a total of six over nine days. That's now I've started re-reading Software for Your Head, which seems to share many of the same learnings from the leadership training course. Maybe we can start using SFYH in practice in my team - last time I read it I was part of a distributed team working more as a collections of lone wolves.

Next week, of course, is OOPSLA in Vancouver. I'm driving up Sunday morning and working the booth in the afternoon. I'll be there all week and in the MSFT booth off and on during the conference, so stop by and say hi. I hear you can get flu shots in Canada, so I'm worried traffic going across the border will stink. It'll be a long week, but at least I get next Friday off. 

Keith with an OOPSLA Preview

Keith is getting ready for OOPSLA:

In the day-long tutorial that runs on Monday 25^th (T40 : Using Domain Specific Languages, Patterns, Frameworks and Tools to Assemble Applications) we plan to use a large proportion of the time to walk through a detailed example of using Software Factories ideas to build four different variations of the same application. This involves us using some existing technology, some technology that will ship in Visual Studio 2005 Team System, and some technology that isn’t yet planned, but which could readily be built by Microsoft or one our partners. As we plan to show the whole process – from soup (business case, business capabilities and business processes) to nuts (running code), this involves a lot of work with help from several teams here at Microsoft. It is coming together well though, thankfully!

You thinking what I'm thinking? Software Factories hands-on lab...

More Architect Bloggers

Two new architect bloggers to note. Jim Clark is a business architect on the Architecture Strategy Team. Jim spends a lot of time with what he calls "Red River" - identification and definition of business architectures, ontologies and environments that promote trusted business solutions. His first post is about Familiarity and Trust. Steve Cook is a contributor to Software Factories and works for Keith. Steve is looking forward to OOPSLA. So am I.

SAF Day 2

I sat through four session yesterday - two keynotes and two informational presentations. My favorite of the four was the session on Business Architecture by Jack Calhoun, CEO of Accelare. I can't summarize it effectively, so I won't try. However, one of the key points I took from the presentation is that the architecture of the business is evolving away from an organizational-based model towards a capability-based model. That's fascinating because business capabilities are a natural match to technology services. Jack made a point that SOA isn't going to be successful if it's just as new technology mechanism - it also needs to enable to business models. He also introduced a term I had never heard before - "chaordic". The term chaordic is a combination of chaotic and ordered and is used to describe the dynamic organization of complex systems (like enterprises). It was coined (I believe) by Dee Hock, former CEO of Visa who wrote a book on the subject called Birth of the Chaordic Age. Yet another book to add to the tall stack that I'm still working through.

This morning, I ran a breakout on Refactoring Your Best Practices. I didn't bring much presentation material to this session since I'm not sure we do a great job of this. Actually, we (i.e. Microsoft) do a pretty good job capturing best practices, but we're less effective at reusing those practices. One of the reasons we're not great at using these practices is because it's very hard to capture the context of the best practice. The breakout session was awesome. We had a variety of experiences represented, from a company where best practices have become rigid rules that must be followed blindly, even when they stop making sense, to a company that has a "Chief Methodologist" who runs a group of architects who build best practices the way that other groups build products. I thought that the project-esque approach was fascinating. One of the implications is that we need to capture baseline information about the practices, so we can tell if we've improved as we refactor. All in all, it was a great discussion. I think everyone got value from it as we decided to "keep the conversation going" on email after the event.

The second morning session was a Q&A with Bill Gates and Eric Rudder. This is one of the high points of this event for the attendees. He was asked about a variety of topics, but the one that caught my attention was Bill's opinion of blogs and if he is planning to start his own soon. Not surprisingly, he was very positive about blogging but made no commitment to starting a blog anytime soon. He's said he didn't want to start a blog unless he had the time to make a commitment to blog regularly. Of course, that's exactly what has happened with Eric's blog. However, in his defense, Eric made a great point - one of the reasons he started his blog to encourage the employees in his division (Servers & Tools) to blog. I'd love to see Bill and Eric blog, but I think it's the rank-and-file Microsoft bloggers that are really making a difference in the community.

Architect MVPs

In addition to event activities, we also take advantage of SAF to have side meetings with customers. Yesterday, I spent an hour discussing software factories and high-performance computing with Simon Cox from the University of Southampton. Simon is one of our inaugural group of Architect MVPs. We have fourteen architect MVPs across two categories: Visual Developer - Solutions Architect and Windows Server System - Infrastructure Architect. We'll be growing this group significantly over the rest of the fiscal year, but there are already some influential names in this group, including Roger Sessions, Paul Preiss (founder of IASA), Ingo Rammer and Clemens Vasters. Congrats to all the new Architect MVPs!

Strategic Architect Forum IV

This week is Strategic Architect Forum IV where we are bring over 200 top strategic architects from around the world to Redmond for a series of presentations and breakout discussions. Norman Judah, CTO of MSFT Services, is doing the opening keynote. Other keynote speakers include Pat Helland, Software Factories' authors Keith Short and Jack Greenfield, plus a Q&A session with BillG. Last year, we recorded all the sessions and published them as The Architecture Strategy Series. We're still working out the specifics for this year's content, but we will be publishing this year's sessions as soon as we can.

One of the things that's interesting about SAF is that we do a small number of traditional presentation sessions and a large number of discussion breakouts. For these discussion breakouts, we get around fifteen to twenty architects in a room to discuss a problem. For example, I'm doing a talk discussion breakout tomorrow on Refactoring Your Best Practices. Each breakout also has a moderator and a note taker - the goal being to record the knowledge and to track a list of action items.

In addition to SAF, we have two ancillary events going on. All of our field architect evangelists are in town with their customers, so the past two days we had training for them. I had to speak Sunday morning about our influential and community programs. We also have most of our Architecture Advisory Board in town for SAF and their twice-yearly face-to-face meeting. This is the first face-to-face meeting of MAAB since I took over the program, so I'm looking forward to meeting these folks in person. 

Good IASA Meeting

Last night's IASA meeting went very well. I always take it as a good sign when community get-togethers end because building security kicks you out 45 minutes after you were supposed to be done. About 25 people showed up and were treated to food, free materials and factories. Steven did a good job presenting Software Factories, and he was even nice enough to let me chime in on a regular basis with my $0.02. BTW, the next meeting is on Oct 27th (last Wed of the month) and I'm going to miss it because of OOPSLA. Watch the IASA website for more details.

One of the key questions asked was if MS was going to "break tradition" with factories and embrace open systems. I guess we still have a long way to go to change that perception, even though we standardized CLI & C# and continue to be a major driver of the ongoing web services standardization effort. Regardless of perception, I agreed - I think it's important to enable Software Factories to be embraced by the industry at large. The Architecture Strategy team has several members who are heavily involved in standards bodies, and they had similar feedback. Of course, we have nothing to embrace beyond the book so far, but we have hinted at tooling to come. As promised, I sent last nights feedback to Keith, Jack and the other folks involved and I hope we'll see that incorporated into their long term plans.

I asked the hard-core folks who stayed late last night what they thought was the best approach for MS to take for getting Software Factories embraced by the industry. Obviously, any tooling around Software Factories would be built on a set of specifications. We could standardize these specifications as we have for CLI, C# and web services. Alternatively, we could simply publish the specifications with an open and royalty-free license (similar to the Office 2003 XML Reference Schemas). Given recent comments by Grady Booch and Simon Johnston (both from IBM) disparaging domain-specific languages and software factories, I'm not sure how well a standardization process around DSLs would proceed. On the other hand, I'm not sure how much industry buy-in there would be for specifications that weren't collaboratively developed. Leave your opinion, and I'll make sure it gets to Jack, Keith, et. al. just like the feedback from last night did.

Architect User Group Meeting Tonight

It's late notice, but tonight is the inaugural meeting of the Puget Sound chapter of the International Association of Software Architects. Steven Houglum, MS architect evangelist for the Pacific NW, is presenting on SOA and Software Factories. The meeting is on the MS campus from 6:30-8:30 and you can get all the details from the IASA site. As I wrote yesterday, I'm interested to see the general response to software factories, so I'm going to be there.

IASA is a relatively new organization with seven chapters in the US as well as one in Australia. It started as an architect user group in Austin - their first meeting was almost a year ago. The overarching IASA organization started about six months ago, according to the IASA weblog. I've spoken to the founder and director Paul Preiss on several occasions and hung out with the Twin Cities chapter president Alex Rupp at TechEd. I think they are doing a great job build a technology agnostic user group community and it's great to see our local field architects supporting it.

As I said, the first meeting is tonight and that's pretty late notice. If you're interested and can't make it tonight, there are two more meetings scheduled for the Puget Sound group in the next three months. Of course, the other chapters are on different schedules, so maybe you can find a meeting in a city near you. If there isn't a chapter near you, I'm sure IASA would be interested if you wanted to start one.

Sounds of Silence

I came back from the far east and got buried by email and meetings. Plus my stupid HP laptop is still stupid - now the both the built in wireless and the PCMCIA card slot are not working correctly so I can't get on wireless. Amazing how quickly the ability to get online from anywhere in my house has become routine to the point where if I can't get on wireless, I barely get online at all.

A couple of quick takes:

• Keith blogged about giving BillG the third copy of Software Factories that came off the presses. When he came to present Software Factories to my team, he gave me copy number four. He and Jack even signed it. Cool!
• Keith and Jack's Software Factories presentation was a huge hit with my team. Our architects want their own copies of the book now. Pat even stole borrowed my signed copy to read over the weekend. I can't wait to see the general response to this book.
• I got the scores back from TechEd Malaysia and Beijing. Metropolis was the top architecture session of TechEd Malaysia and Data in SOA was the top architecture session and in the top ten of all sessions at TechEd Beijing. Gurpreet's sessions also did very well. There are still a few worldwide TechEd's left, but I think it's safe to say that the new architecture track has been a big success. We're already in planning for next year, so look out for us to build on that success.
• I'm still pretty much at the "what good is this?" stage on WS-Transfer and WS-Enumeration. I can possibly see using WS-Enum and/or WS-Xfer's Get action for retrieving reference data, though recent issues with RSS and bandwidth demonstrate the inefficiency of polling for changes. David Ing commented that WS-Xfer provides "semantic understanding" of actions. True, but at the cost of semantic understanding of the data. It's much easier to change the name of the action that to change the schema of the message, yet WS-Xfer and WS-Enum's operations are completely untyped. So I remain unconvinced of the value of these specifications.

Architecture Events

While I was in the far east last week, the Architecture and Design Events page of Architecture Center got updated. Thanks go to Lawrence, Richard and Mark - Architecture Center's own three amigos if you will. Looks like the Software Factories guys are going to very busy, most of the upcoming events feature some type of presentation on factories, models or domain specific languages.

What's cool about this list is that it features MS events like TechEd and PDC as well as non-MS events like OOPSLA, UML Conference and OOP. The only bad thing about this list is that I'd like it to be longer. Should there be more architecture-centric events or more architecture content at developer-centric events on this list?

Final Day of My Far East Trip

I said yesterday that the Metropolis session had gone very well. Got the score back - 7.71/9 speaker satisfaction and 7.83/9 content satisfaction. Not bad, esp. given that it was a translated session. Today, I presented Data in SOA, and I thought it well also. This time, I did it without the translator at the audience's request. We'll see how the scores come back. I thought it was encouraging that attendance at the architecture sessions steadily improved. Gurp's session yesterday afternoon had more people than my Metropolis session and today's session had more people still.

After the session, I spent some time answering questions. After Metropolis, there was one guy from China Mobile. This time, he was joined by two dozen other attendees. And again, Mr. China Mobile had some great questions that really made me think. He pointed out that often architecture is focused on the non-functional requirements of a system (perf, scale, reliability, etc) that are very technology platform dependent. As the platform evolves, the decisions made in implementing a system become obsolete, which makes it very hard to evolve a solution to take advantage of the improved platform. He's right. The reason he's right is that we don't a good mechanism to capture the design decisions made during the development of a system, just the outcome. Software Factories can really help out here.

After the presentation and Q&A with the attendees, several of the speakers and I were treated to a trip to the Great Wall of China. One of the other speakers was Stan Lippman, whom I had never met. As for the Wall, what can I say but wow. We climbed for about two hours - glad I put on my sneakers! We reached at least a local summit, where there is a quote from a Mao poem stating that you can't be a real man until you've climbed the Great Wall. Well, I guess that makes me a real man now. The wall is over 45 degrees in several places that we walked. I sure got a good workout today.

I'm exhausted, but I figure I can sleep on the 12 hours worth of plane ride I have tomorrow so I'm heading out (after a shower) to do a little shopping. I've had a great trip and met with some great people - both MSFT employees and customers alike - but I sure am looking forward to being home.

Metropolis in Beijing

So the Metropolis session went very well, even though it was being translated. We did what is called interactive translation - where I say a few sentences in English and then Quanzhan translated it into Chinese. It meant that we couldn't cover as much material but we still made really good time - I did 52 slides (which is a significant reduction from the full 75 slides) in exactly an hour. The audience understood at least some of what I was saying - they laughed when I made a joke - but it really helped me to have the time to think about my next sentence while Quanzhan translated my last one. I was able to choose my words carefully in order to make sure the topics were covered as concisely as possible. I'm hoping the Data in SOA talk goes as well tomorrow. After I finish this entry, I have to working on cutting that presentation down to fit in an hour with translation.

After my presentation, I spent some time with an architect from China Mobile. I love conversations where a customer challenges me to codify my thinking - I learn from the customer as well as myself. He made a great point about architecting for an unknown future vs. engineering for the present. Sounds like a strong believer in YAGNI. While we were discussing SOA I was able to articulate something I knew but had never really expressed before. SOA is a means to an end, it's not an end in and of itself. It's a great way, but not the only way, to enable the flow of information across disparate systems. It's also a great way, but not the only way, to enable more flexibility in the way we build systems. A key success metric for companies today is time-to-market. However, time-to-market usually refers to the time it takes to develop something the first time. In the future, I think the time-to-adapt will become even more important than time-to-market. Flexibility - you are gonna need it.

Stage 1 on New WS Specs

I'll be honest, with a quick glance at WS-Enumeration and WS-Transfer I'm at what Don refers to as Stage 1. Especially WS-Transfer which appears at first glance to be CRUD for web services. Maarten talks about using CRUD only when you can afford it. My biggest issue with CRUD is that it assumes a trust relationship - that some other service is responsible for deciding when and how to CUD entities that I manage. I can't imagine exposing an interface like that on any service I build.

But it is nice to see we've started publishing specs in easy to download PDF format.

Preparing to Present in Beijing

I made it to Beijing yesterday with no problem. There was a car waiting for me at the airport, and one of the speaker managers met me at my hotel shortly after I arrived. They even provide a loner dopod SmartPhone for use while I'm here though I'm not sure who I would call. I went to the attendee dinner last night and on to a party for local partners. I spoke to a few people, including a gentleman from Digital China who reads my blog. It's nice to be recognized at all, much less as an International Man of Architecture.

After dinner, I worked for an hour or so with the translator for my Metropolis session, Quanzhan. Quanzhan is from China, but he's now a program manager in the Windows Server division at corp - his office is just on the other side of campus. He's here doing presentations of his own as well as helping out as a translator. I've met at least one other Chinese person here who works in Redmond - I'm guessing presenting TechEd China provides a great way for these folks to visit home and also give back to the community they are from in some way.

Delivering these presentations via a translator will be an interesting challenge. I've never worked thru a translator before. Each session has it's own unique challenge - Data in SOA is very technical while Metropolis is pretty abstract. I ended up cutting 20 slides from today's Metropolis talk in order to ensure I have enough time to hit the important points. Presenting this way requires a truly conservative approach to word choice - even though my session is 75 minutes long, in reality I have closer to 30 as everything I say has to be spoken twice. I'm grateful that Quanzhan and I spent the time last night going thru the presentation even though we were both exhausted. I guess we'll see how well the session goes today, but I think we'll do well.

Leaving KL

I'm sitting in the Kuala Lumpur airport - my flight to Beijing leaves in about half an hour. I didn't get a chance to blog all day yesterday, so here's a quick recap:

• I've run into a bunch of people who I first met at TechEd Malaysia 2002. In particular, Adrian and Rathi who both wanted to see pictures of Patrick. The big bummer of only spending two days here is that I didn't get to do much more that the conference.
• After the sessions on Wednesday, I hung out with a bunch of the other speakers at dinner. Several of the RD's who spoke at TechEd US also came to our Architect Road Rally and had a great time. Always nice to be told you threw a great party.
• Thursday started  with Gurpreet's session on Architecture Vision & Direction. I didn't think this session did as well as the EA talk the day before. Part of that comes from the fact the EA talk set a high bar, but some of it comes from the fact that the V&D talk wasn't as polished. In contrast, I did much better with my Metropolis Thursday than the SOA Data talk on Wednesday. Maybe I just need one talk to get back in the groove - I usually think I do better in my second session.
• After our sessions, lunch, and hanging around in the cabana with some other attendees, Aaron (the local Architect Evangelist) drove Gurp and I around Putrajaya. Pat often points out when he presents Metropolis that you can't bulldoze Boston just because the roads aren't straight. However, you can build a new city about a half an hour away and move into it when it's finished. That's Putrajaya - the new administrative center of Malaysia.  It was stunning - huge and ultra modern. But it was also strange as it is completely underutilized, so far anyway. We also did a little shopping and I was able to find a few things for the family back home.
• Last night we had a regional architect dinner, where we got to hang out with a bunch of the local architects in the region. Lots of good discussion.

Next stop - China!

Gurpreet on Enterprise Architecture

I just finished my Data in SOA talk for TechEd Malaysia. The room was very hot and the session was right after lunch - not exactly optimal conditions. I did OK - could have been better. Between time in hotels and on airplanes with nothing to do but code, I've written more in the last four days than the previous four months combined. I didn't spend as much time as I could have reviewing the deck, so I guess I can't say that I was utterly prepared. But I have delivered this talk many times and spend a significant amount of time thinking about the concepts and discussing them with teammates. BTW, this talk is now available as a whitepaper on Architecture Center.

This morning, Gurpreet delivered his Enterprise Architecture talk. I thought it was pretty good, esp. given that he wrote quite a bit of the deck on the plane to Malaysia. I had only seen him present once before when he was exhausted (hotel screwed up his room) and on pain killers (he fell of the stage and messed up his back). He was much better this time - he's a great storyteller. He also had a few choice quotes I thought I'd share:

"We could spend the next month in this room talking about enterprise architecture and only just be getting started."

"Don't tie your ego to your design."

"If you don't do EA, you can't do SOA."

He spent most of his time talking general EA topics, with the remainder spent on MSFT incarnations of those topics, such as EASOT. What was funny was that when he showed me his deck, I thought the MSFT stuff play better with the audience, but it turned out the general EA stuff was great. For example, Gurpreet started by talking about the Winchester Mystery House, built by Sarah Winchester (of Winchester Rifles fame). This place is an architectural nightmare with stairs that lead into the ceiling and nearly half of the doors that open onto walls. It's also over provisioned with 40 bedrooms, 5 kitchens and 17 chimneys. An over-provisioned architectural nightmare? Sounds like a typical enterprise.

He talked about the typical conceptual/logical/physical viewpoint of the enterprise with an interesting twist - the contextual level. This viewpoint is above conceptual in the model. To take his example, if a bank builds an online banking system, we're all very familiar with the conceptual, logical and physical views of that system. The contextual view might be something like "We're losing customers due to the fact we don't have an online banking system." I need to spend more time thinking about this, and how to map between these views, but I thought the contextual viewpoint was very interesting.

I really wanted to blog what Gurpreet claimed was EA's biggest fallacy: that it doesn't change. Because we've based the concepts of software architecture on building architecture, there's this belief that you first design your architecture and then you build things to that architecture - i.e. like a building. For example, when I saw John Zachman present his framework, he was asked how one goes about implementing the framework. His response was something along the lines of: "Build all the layer one models, then build all the layer two models, etc. etc. etc. and then hit compile." Obviously, that kind of revolutionary waterfall approach to enterprise architecture just won't work. While buildings evolve and "learn", they aren't in a constant state of flux the way enterprises are. This is where Gurpreet made the comment about not tying your ego to a particular architecture - you have to realize it's going to change.

I'm hoping that Gurpreet's Architecture Vision & Direction talk will be equally good. Now, I just need to pester him into publishing this material in a whitepaper or blog or something. He's presenting at Strategic Architect Forum next month, so I expect these talks will continue to improve.

Japanese Architects and Malaysian TechEd

After spending Monday morning exploring Shunjuku, I had meetings and dinner with the local architect evangelists followed Tuesday morning by a meeting with the local evangelism lead. (which was in turn followed by a flight to Kuala Lumpur which is why I'm not posting this until now). In my first year on the job, I focused on the "basics" which included helping improve Architecture Center and handling the TechEd Architecture Track. As part of that focus, I spent little time worrying about globalization issues. Not that they aren't important, I just didn't have the bandwidth to handle it. Now that we have worked through the basics, plus it's now "we" and not just "me", we can start to have some focus on better globalization support. For example, there's a Japanese Architecture Center that needs to have the same content as the MSDN site. Machine translation may have come a long way, but there's still longer to go. Given the massive number of characters in eastern languages, machine translation to English works pretty well - here's the Japanese Architecture Center in English (translated by Babelfish). However, translating to Japanese (or other eastern languages) is much much more difficult.

Like several other MSFT subsidiaries, Japan has someone responsible for broad reach architecture - i.e. architect with a lower case "a" - so we spent most of our time discussing plans and figuring out how best to leverage each other. I was really impressed how deep these guys were on Software Factories and modeling already - they had a session on Software Factories at the TechEd Japan last week!

Now I'm in Kuala Lumpur for TechEd Malaysia. I'm presenting the Data in SOA talk today and Metropolis tomorrow as part of the Architecture track. I had breakfast with Software Legend Tim Huckaby who I met the first time two years ago at TechEd Malaysia. My teammate Gurpreet just showed me the deck for his Enterprise Architecture talk he's presenting in about 30 minutes. He's also doing a presentation tomorrow on Microsoft Architecture Vision & Direction. The EA deck is pretty good and the Vision & Direction talk is something he's been working on for several months, so I'm looking forward to seeing both of these. We're working on the next release of The Architecture Strategy Series and I'm thinking these talks should be included.

Quick Architecture Takes

It's been quiet around here as I'm prepping for my next trip abroad. TechEd Malaysia and Bejing are next week and I'm presenting at both, plus I'm stopping over in Tokyo to meet with the local field architects. Should be pretty cool. I did TechEd Malaysia two years ago, but I've never been to China before so I'm pretty excited.

A few short takes before I go:

• Three more Architecture Strategy team members are blogging: Josh Less, Norman Guadagno and Drew Gude.
  • Josh is the "FinServGuy", focusing on the architecture, patterns and standards in the financial services industry. He just started on our team and just started his blog. Go check out his post on the Insurance Value Chain.
  • Norman actually not a team member yet and he's the blogging vet of these three. When he starts in October, Norman will be our marketing manager, a skill set that we've been sorely missing around here. His blog - Atlas Brand View - has no coverage of architecture per se, but interesting marketing tidbits like Eight Reasons Marketing Makes Sense and Whatever Happened To Burger King. Can't wait to see what he writes about architecture marketing.
  • Drew is a vertical architect like Josh, but focused on automotive and manufacturing. He's doing some cool work with RFID and is a member of the RosettaNet Architecture Advisory Committee. However, as I write this, Drew has yet to blog anything. I keep hearing how he's working on something but since I'm going out of town I figured I could put a bit of a squeeze to get him to post. :)
• I'm stoked about the .NET Languages site. Not much up there yet, but I subscribed. Given my interest in software factories and domain-specific languages, I guess this isn't a surprise. But my interest in language design goes way back to the early days of my blog. BTW, Compiling for the .NET CLR is a great book.
• I hung out with new-red-pill-taker Peter Provost last weekend. Our families went to Bumbershoot together. With my 18 month old son and Peter's 30 month old daughter, there was little, if any, debauchery - unless riding the kiddy rollercoaster when you're not really tall enough counts.  There was, however, lots of geek talk, at least when the wifes were off doing other things.

I've got some cool stuff "in the works" which I hope to talk about more when I get back. I have 35 hours on airplanes in the next ten days, so I get a chance to bang out some code.

Software Factories at OOPSLA

Keith blogged about his hectic days since he returned from vacation. Getting a book published, preparing for a BillG review, you know all the usual stuff. :)

I keep waiting for my hardcopy of Software Factories, and Keith wrote that it should hit the shelves on Sept. 15th. This means that it will be available in time for OOPSLA 04. I've blogged this before, but it's worth repeating that there is going to be quite a MS presence at OOPSLA this year:

• Dr. Rick Rashid (director & founder of MS Research) on the Future of Programming
• Jack Greenfield on Software Factories (part of the co-loed GPCE conference)
• Ward Cunningham on Objects, Patterns, Wiki and eXtreme Programming: All Systems of Names
• Herb Sutter on Concrete Languages on Virtual Platforms
• Steve McConnell on Code Complete 2 (Steve isn't a 'softie, but we do publish his book)
• Full day tutorial on Using Domain Specific Languages, Patterns, Frameworks and Tools to Assemble Applications (i.e. Software Factories) by the authors of Software Factories
• Half day tutorial on Generative Software Development by Jack Greenfield and Krzysztof Czarnecki
• Panel discussion on MDA: The Realities with Steve Cook and Granville Miller (among others)
• Panel discussion on The eXtreme Programming Customer: Fad or Fashion? featuring Granville Miller (among others)
• The Great J2EE vs. .NET Shootout panel discussion featuring Don Box (MSFT Web Services Architect), Anders Hejlsberg (MSFT C# Architect), Jon Crupi (Sun Java Archtiect), Rob High (IBM WebSphere Architect), Alan Knight (Cincom VisualWorks architect) and chaired by Martin Fowler (ThoughtWorks Chief Scientist)

You know I'm not going to miss all that. We'll have coverage on Architecture Center during the event and I will endeavor to get as much of this content Architecture Center after the event. I think the tutorials in particular will be very interesting to the audience at large.

Anyone else going to OOPSLA?

Update: Added the three panel discussions. I'm guessing the J2EE/.NET shootout will be standing room only.

Presenting Software Factories

I presented Software Factories for the first time today and I think I did a pretty good job. We had some architects were in town from a new managed SI partner and they wanted to discuss modeling. They are (were?) an IBM partner, so they're a big WebSphere shop. They're also XDE users, so I laid out the Software Factories concept as well as the modeling tools that are coming in VS2005. They seemed pretty impressed. Of course, they're having what I expect is a typical experience with UML tools - they use XDE for documentation and communication only (i.e. UmlAsSketch). They don't even try to generate code from the models anymore.

To help explain the Factories concept, I used Steve Maine's Efficiency/Precision/Generality modeling approach (and gave him credit for it, of course. Steve, I will find a way to properly thank you for that brain.save) as well as my own ideas about VB as a Software Factory. Both worked out very well to help communicate the goals of the Factories approach, though I could refine the delivery quite a bit. I also talked about the Evolving Frameworks Pattern Language, which Jack outlines in the JOURNAL factories article this way:

• After developing a number of systems in a given problem domain, we identify a set of reusable abstractions for that domain, and then we document a set of patterns for using those abstractions.
• We then develop a runtime, such as a framework or server, to codify the abstractions and patterns. This lets us build systems in the domain by instantiating, adapting, configuring, and assembling components defined by the runtime.
• We then define a language and build tools that support the language, such as editors, compilers, and debuggers, to automate the assembly process. This helps us respond faster to changing requirements, since part of the implementation is generated, and can be easily changed.

The problem is that each of these steps is much much harder than the preceding ones. Identifying problem domain abstractions & patterns is something that most organizations are already doing, even if they don't do it explicitly today. Codifying those abstractions in a reusable framework is much harder, primarily because it's hard to think through all the usage variations a framework may experience. Still, many companies have the skills to develop reusable frameworks. However few companies have the language and tool development experience to make investing in custom tools cost effective. For example, even though Gregor and Bobby have defined a language for integration patterns, the only tooling available is a Visio stencil.

For Software Factories to work, we need to make it much easier to build domain specific languages and modeling environments. This is one of the big hurdles to adopting the factories approach today.

Zooming In From High Levels of Abstraction

Denny Figuerres left the following comment on my last post:

[O]ne of my "wish" items would be a kind of editor that would be a merge of flow-chart and text editor so that I could view my function as text or as a kind of zooming diagram with more details as I drilldown. like when you use a map program, first you see an area and major routes, rivers, lakes etc... and as you zoom-in you see a smaller area with more detail. at some point you are down to a single line of code that is an expression of some kind.

I think this is a great idea, and is completely in line with Software Factories. Denny, what you're talking about is working at higher levels of abstractions. If you read the Software Factories article on TheServerSide.NET, Jack writes:

How...do we work at higher levels of abstraction? We use more abstract models, and move the platform closer to the models with either frameworks or transformations, as illustrated in Figure 4.

• We can use a framework to implement higher level abstractions that appear in the models, and use the models to generate snippets of code at framework extension points. Conversely, the models help users complete the framework by visualizing framework concepts, and exposing its extension points in intuitive ways. A pattern language can be used instead of a framework, as described by Hohpe and Woolf. This requires the tool to generate the pattern implementations, in addition to the completion code. This approach is illustrated in Figure 4 (a).
• Instead of a framework or pattern language, we can generate to a lower level DSL. We can also use more than two DSLs to span a wide gap, leading to progressive transformation, where models written using the highest level DSLs are transformed into executables through a series of refinements, as shown in Figure 4 (b).

I think we'll see a combination of these two approaches. Obviously, as an industry, we have lots of experience building frameworks and I don't see those going away anytime soon. The second approach, however, is much more fascinating as you get that "zooming" effect that Denny describes.

One thing to note about this zooming approach of using higher level abstractions - different systems use different abstractions. The abstractions you use to build an ERP system are not the same as the ones you would use to build a telephone billing system. So the view of the top-level abstractions will be very different, even if they both end up implemented on the same platform using the same programming language.

The Most Popular Modeling Environment Ever (So Far)

Steve's post on "the modeling problem" hits the nail on the head. We're all familiar with the concept of "fast, good, cheap - pick two". Steve breaks down modeling into "general, precise, efficient - pick two (and favor one)". Furthermore, you can't have a language that is both general and precise. UML takes what Steve calls the "Favor efficiency, accept generality and compromise precision" approach:

The UML metamodel is flexible enough to allow it to describe virtually any system out there. However, from a formal semantic perspective, the resultant model is gooey and formless which makes it very difficult to compile into anything useful. At best, we can get some approximation of the underlying system via codegen, but even the best UML tools only generate a fraction of the code required to fully realize the model. The lack of precision within the model itself requires operating in both the model domain and the system domain, and implies that some facility exist to synchronize the two. Thus, the imprecision of UML forces us to solve the round-tripping/decompilation problem with 100% fidelity, which is generally difficult to do.

Software Factories, on the other hand, takes what he calls the "Favor efficiency, accept precision, and compromise generality" approach:

This, I think, it the sweet spot for Microsoft’s vision of Software Factories. Here’s why: the classic problem faced by modeling languages is Turing equivalency. How do you model a language that is Turing-complete in one that’s not without sacrificing something? The answer is: you don’t. You can either make the modeling language itself Turing-complete (which sacrifices efficiency) or you can limit the scope of the problem by confining yourself to modeling only a specific subset of the things that be expressed in the underlying system domain. Within that subset, it might be possible to model things extremely precisely, but that precision can only be gained by first throwing out the idea that you’re going to be able to efficiently and precisely model everything.

When describing Software Factories, I have two analogies that I use to explain the idea. The first is the "houses in my neighborhood" example I blogged before. That does a good job describing economies of scope, but doesn't really cover the modeling aspect of software factories. Talking about how you model cars or skyscrapers doesn't really capture the essence of software modeling - you don't generate the construction plans from a scale model of a skyscraper. However, it turns out that all developers have at least a passing familiarity with my second analogy: Visual Basic, the most popular DSL and modeling tool of all time (so far).

The original Visual Basic was a rudimentary software factory for building "form-based windows apps". (Today, VB.net has been generalized to support more problem domains) Like the factory approach that Steve describes, VB was very efficient, sufficiently precise, yet not particularly general (especially in the early years). There were entire domains of problems that you couldn't build VB apps to solve. Yet, within those targets problem domains, VB was massively productive, because it provided both a domain specific language (DSL) as well as a modeling environment for that domain.

A DSL incorporates higher-order abstractions from a specific problem domain. In the case of VB, abstractions such as Form, Control and Event were incorporated directly into the language. This allowed developer to directly manipulate the relevant abstractions of the problem domain. Abstractions extraneous to the problem domain, such as pointers and objects in this case, got excluded, simplifying the language immensely. Both of these lead directly to productivity improvements while limiting the scope of the DSL to a particular problem domain.

In his post, Steve makes the point that it's pointless to distinguish between modeling and programming languages. VB certainly blurred that line to the point of indistinguishably. Regardless, graphical languages are typically more compelling and productive than textual ones. It's hard to argue with the productivity that VB form designer brought to the industry. Dragging and dropping controls to position them, double clicking on them to associate event handlers, changing properties in drop down boxes - these idioms have been widely implemented to the point that essentially all  UI platforms provide a drag-and-drop based modeler. It's such a great design that 10 years later, UI modelers are essentially unchanged.

Once you realize that VB's DSL and modeling environment was a rudimentary software factory, you realize that Software Factories methodology is about generalizing what VB accomplished - building tools that achieve large gains in efficiency by limiting generality. Since each of these tools focuses on a limited problem domain, you need different tools for different problem domains. The problem is that while building apps with VB may be easy, but building VB itself was not. Most enterprises have the expertise to develop abstractions in their domain of expertise and to codify those abstractions in frameworks, but very few can develop tools and DSLs for manipulating those frameworks. One of the goals of Software Factories (and VSTS Architect for that matter) is to make it easier to build tools that are really good at building a narrow range of applications.

Note, it's important to note that the term "narrow range" is relative. Darrell seems to think narrow range only means vertical market applications that don't "solve new and interesting problems". It's true that the narrower the range, the more productive the tool can be. But VB shows us that you can achieve large productivity gains while solving new and interesting problems even in broad scope problem domains.

Another Team Blogger

Actually, I don't think he's “officially” part of the team 'til next week, but Josh Lee has already started a blog about his new job on the Architecture Strategy team. Josh is “The FinServ Guy”, and is a member of the IFX Forum Board of Directors. Nothing really meaty on his blog yet, just a Hello World post, but I hear great things about him.

Speaking of the Architecture Strategy team, I finally took 5 minutes to term serv into the machine that hosts DevHawk to update the theme. I keep mentioning the Architecture Strategy extended team OPML file, but I wanted to add a blogroll to the site theme. Now, I have a Team Blogroll on the left hand side of my website featuring all of my blogging teammates as well as all the blogging architect evangelists. Enjoy. 

More MSFT Architect Bloggers + a Standard Rant

We keep getting more and more field architects and architecture strategy team members blogging. Remember, I keep a list (I am becoming the Scoble of Microsoft Architecture). Anna Liu is a field architect evangelist who presented at TechEd Australia (but we didn't get a chance to hang out). Anna's also been thinking about software development as an engineering discipline.

In addition to Anna, two of my teammates are blogging: Chris Keyser and Dave Welsh. Chris is a solution architect who's doing some awesome next gen SOA work. He's been blogging about using WSE2 to manage Security Context Tokens. Chris, like John deVadoss (who has relapsed into silence), is very pragmatic so it's great to run radical ideas past him.

Earlier this year, our team "inherited" a group of awesome vertical architects - I've blogged about John Evdemon before who's from that group. Dave is also from that group. Like many of our vertical architects, Dave is heavily involved in standards bodies - in Dave's case it's UN/CEFACT. He's got an great article on how Standards Development Organizations traditionally work and another on how MSFT (and our specification partners) is improving on that process. He's shining a light on the dark corners of the standard process, which is a good thing since so many people act like standards are a silver bullet solution. I love Dave's description of the traditional standards process:

[L]aunch a committee, politically pick a chair, generate lots of hype and expectation on how this spec will solve world hunger, stack the new committee with people who may be able to contribute, host conference calls and arm wrestle the original idea down to some compromise that seems to make sense, then hope someone’s got a number of free weekends over to write up a draft of the new spec.

You want an example of the results of a traditional standards process? How about XSD? I think XSD is the ugliest widely-used spec around.  Don agrees, according to his comments from last years SellsCon:

Nothing illustrates [the cost of standardization] more than XML schema. XML schema is the quintessential example of what happens with a design by standards body specification. Rather than taking something that worked and something that was done and that there was experience with and effectively dotting the i’s and crossing the t’s you had two from every company off doing wanton innovation and invention without implementation experience. It was a train wreck in the making, especially when you consider the fact that you had people who vehemently disagreed about what they were building. Some people thought they were bringing object orientation to XML. Some people thought they were bringing database schema concepts to XML. Some people thought they were just, you know, reliving the SGML dream. So what do we get? We get a Frankenstein’s monster that is dumber than the dumbest person in the committee. No one person on that committee could have produced something this bad. It took an army of people to work hard day and night to build something that is not good. It’s not terrible – can we make it work? Yes. But it’s going to take a lot of work from a lot of plumbers and a lot of tool vendors to make XML schema palatable to the average developer.

A great example of the opposite approach is RELAX NG. It is widely believed at this point in time that RELAX NG is a better schema language for XML than XML schema. Why? Because two guys who were really smart said why don’t we go do this and let’s get it working and let’s build it while we do it and let’s iterate it and see what works and what doesn’t work. And then when we’re done we will take it to the rubber stamp – I’m sorry, Oasis – where they will carefully vet every decision and bless it and give it UN status.

I'm with Don and Tim: I want RelaxNG. More importantly, I want standards that are built like WS-* and RelaxNG.

Reading Is Fundamental (To Forming An Educated Opinion)

So while I was on vacation, my trusty teammates plus our cohorts @ MSDN published JOURNAL 3 on Architecture Center. The first article from JOURNAL 3 is called The Case for Software Factories by Jack Greenfield, co-author of the Software Factories book. Jack noticed that the article, an introduction to the concept of Software Factories, was being rated inordinately low. Currently, it's been rated 4 out of 9. To compare, Pat's introduction to the concept of Metropolis from JOURNAL 2 rated a 7 out of 9. So a 4 is a pretty low score. Granted, I'm not going to lose any sleep over it, but it was curious. Then Jack discovered this thread on Slashdot, where most of the posts were from people who don't appear to have read the article at all. Here's an example comment:

The "software factory" analogy has been around before. It's nonsense, of course. The software analogy of a "factory" is the plant that presses CD-ROMs. Pressing the 10,000th CD-ROM of a software product is the software equivalent of building the 10,000th Nissan Maxima on a production line. But writing the software which will go on that CD-ROM is the software analogy of designing the 2005 model of the Nissan Maxima. Now, some software development is not very creative. Just as tweaking the design of a car model that's been around for 10 years, to get something a little bit new for a new model year, is not very creative mech engineering. But it's still design, not assembly-line production. A competent software engineer will be able to do it better and faster than a bad one. And a factory worker will not be able to do it at all.
[Comment by njdj from Slashdot thread Hackers As Factory Workers?]

What's hilarious about this post is that point of the article is that the software industrialization will not look like a factory stamping out of exact duplicates that njdj describes! The key to understanding software industrialization is understanding the difference between economies of scale and economies of scope. What njdj describes is economies of scale - where you stamp out a boatload of multiple identical instances. Economies of scope is when you create similar, yet distinct, instances. For example, in my housing development, the houses are all similar, but not exactly the same. There are three or four basic designs used for around a hundred different houses, but the different instances of each design vary slightly. Since you can't stamp out identical copies of the house, economy of scale doesn't apply. However, economy of scope does apply - these houses use the same basic building materials, similar patterns of construction, the windows and doors are the same size, the plumbing and electrical wiring are the same, etc. Even though the houses aren't identical, the construction company can build a bunch of similar, yet distinct, houses much cheaper than if they built each one "from scratch".

I know njdj (and most of the other posters on the Slashdot thread) didn't read the article, because the article comes right out and states:

We can now see where apples have been compared with oranges. Production in physical industries has been naively compared with development in software. It makes no sense to look for economies of scale in development of any kind, whether of software or of physical goods. We can, however, expect the industrialization of software development to exploit economies of scope.
[The Case for Software Factories by Jack Greenfield from JOURNAL 3]

I don't know how much clearer Jack could have made his point, so I'm surprised that so many people missed it (or bothered to post their opinions of an article they clearly didn't read). However, I'm not going to lose any sleep over it. I blogged a while ago about the the Oakland A's exploiting the inefficiencies of baseball. Jack's talking about exploiting the inefficiencies of software development. At some point, someone will be successful adopting a factory approach and everyone else will have to adapt. Software development has to change - you can't expect to stay in the major leagues with a .160 batting average.

Back in All Blacks

So I'm back from vacation and ready to head into work tomorrow. I'm almost over jet lag, but Patrick still thinks his bedtime is his afternoon nap. I slept thru the Tri Nations match yesterday morning, where the New Zealand All Blacks, my new-favorite rugby team, got beat bad by the South Africa Springboks. After losing to the Australia Wallabies while we were in Sydney, the All Blacks are out of the running for the Tri-Nations Cup. Winner of next weeks' match between South Africa and Australia takes it. (note - apparently, several readers pointed out that I got my cups mixed up. Bledisloe Cup is between New Zealand and Australia only, and since NZ and Aus drew in their home and home series, NZ keeps that cup.)

I didn't blog any of my vacation as it was happening, but I want to revisit two items that I think will be interesting to my blog readers - the Buildings of Sydney and the Sydney Opera House.

I went to Australia (and New Zealand) to present Metropolis. Delivering this talk (as well as the follow-up talk on Buildings and Applications) has changed the way I look at buildings and cities. I've lived in three different cities - Washington DC, Los Angeles and Seattle. DC is filled with old buildings, but there are no skyscrapers. No building in DC is allowed to be taller than the Washington Monument. Seattle is relatively young, and there was a big fire in 1889 that destroyed a great deal of the city. Los Angeles is...well...I've blogged my opinion of Los Angeles before. LA is like a movie set - it only looks good on TV. Drive around LA and you'll find miles and miles of mini malls, but no history.

Sydney is very different. Many of the older buildings are under "heritage protection" meaning that their facade's are protected and can't be changed. This leads to a fascinating mix of older buildings side by side with modern skyscrapers. Paddy's Markets, which has been there since at least 1834, is housed in a building originally built in 1909 (according to the building's facade). However, if you check out the picture, you'll notice that the building's second floor is notably more modern than the first floor. That's Sydney to a "T", new sitting right next to, or on top of, the old. I imagine that some of the older eastern seaboard cities of America have the same combination of historical and current, but none that I've spent a significant amount of time in.

Of course, you can't go to Sydney and not visit the world famous Sydney Opera House. You can read the history online, so I