Is your Enterprise too big to fail?

We have met the enemy, and it is Complexity.

About a year ago, the media was flooded with debate about financial institutions that were deemed "too big to fail" by industry analysts, financial experts, and government representatives.  A failure by one of these institutions, these experts told us, would have catastrophic effects on our economy, and it was therefore incumbent upon us as a nation to keep these firms solvent.

And if you believe this, I've got some great beachfront property in Kansas I'd like to talk to you about.

[callout title=What dependencies?]How, exactly, are all these financial firms connected?  Obviously, a full exploration of this is out-of-scope here, but the single biggest problem child in the financial meltdown was the web of financial derivatives, which financial institutions applied as a sort of insurance policy, but which, in the end, turned out to be anything but that.  The idea behind these derivatives was to aggregate a bunch of debt obligations together and resell them, and financial firms believed this would act, among other things, as a sort of reinsurance guarding against risk in any of the individual debt obligations.  As these things were packaged and repackaged, though, the cumulative effect was to create a virtual ponzi scheme of obligations linking all of these firms together in a frighteningly incestuous fashion.[/callout]

"Too big to fail," you see, resulted directly from "too complicated to understand."  The real nightmare scenario for these experts wasn't the failure of a single institution, no matter how big it was -- it was the domino effect where the failure of one institution would cause a chain reaction of failures across all the top financial firms.  Here's the killer, though: these experts couldn't tell you which dominoes, exactly, would start a chain reaction, which mean that (1) these experts understood that there were horribly complicated inter-dependencies among these financial firms, and (2) they didn't understand these dependencies well enough to understand what would happen when one failed.

The problems with this scenario go on and on.  Very few people have any real grasp on the actual implementation of complex financial instruments within a single organization, let alone the contractual obligations that link firms to firms.  This includes the executives running these companies, who, in true empty-suit fashion, blindly accepted whatever horse-hockey they were fed by their underlings because there wasn't a chance in hell that they understood all the details of the operations under their control.  This isn't to say that these executives aren't bright guys -- nothing could be further from the truth.  In fact, the complexity of these operations is so staggering that no one person could ever hope to grasp the whole picture at once.

At this point, you might be wondering what this has to do with software development.  Fair enough.  The connection is the common problem: complexity.

Software developers are no strangers to complexity.  Despite an onslaught of tools, techniques, frameworks, and design patterns which all claim to make Enterprise software development "easy", our applications invariably end up complicated as hell by the time they're installed.  I opened a Visual Studio solution the other day that had 43 projects in it.  What's worse, this solution clearly followed patterns that have been demonstrated by Microsoft, so there's a clear argument to be made here that this behemoth is crafted according to industry best practices.

A developer who's immersed in such a solution can have a ghost of a chance of understanding what's happening among those 43 projects, but just barely, and only if he's really good and really focused on this solution.  This is functionally equivalent to a Financial Analyst who understands in great detail how an individual financial derivative is constructed.  Like the financial example, though, there's no way that business leaders could ever hope to fathom the complexity of software like this, and they're the ones who are making decisions about it.

This has to end.  The status quo is one where our leader don't understand the things they're making decisions about.  How is it any wonder that they all behave like pointy-haired bosses?

It turns out that I'm not the only one that believes this.  Roger Sessions is a well-known architect and author who's been promoting simplicity since the dawn of the Doppio Macchiato.  Recently, he's been supporting his book, Simple Architectures for Complex Enterprises, with some excellent blogging and a white paper, which can be found on the book's companion site.

Honestly, some of Roger's material isn't easy to swallow.  Software designers by nature are a pretty self-assured bunch, and usually, that's well-justified, as they're also a pretty bright crowd.  We'd all like to believe that the designs we've crafted are perfectly extensible and scalable.  Furthermore, we're reluctant to admit that anything is too complicated for us to grok.

It's not easy for business leaders to grasp this stuff, either.  Since they're already oblivious to the complexity of these systems, it's a pretty big leap of faith for them to accept that complexity is the problem, and furthermore, that there can actually be a solution.

Adding to this whole mess is the predisposition of nearly all the individual role players to keep right on doing what they've always done.  Our understanding of capitalism and efficient markets suggests that groups of people will collectively find efficiency and balance, but when the individuals involved don't really understand how their actions affect outcomes, you start to see some pretty odd collective behavior.  Dan Ariely has done some remarkable work exploring this paradox in his book, Predictably Irrational.

All this suggests that it's going to take a pretty phenomenal level of commitment to change the status quo.  Let's start by admitting we've got a problem.  I believe that Complexity is the #1 problem facing Enterprise software development today, and it needs to be confronted and mastered.

Are you with me?

Related articles by Zemanta

• The Science of Irrational Decisions (science.slashdot.org)
• Are We In Control Of Our Own Decisions? & The Surprising Science Of Motivation (beatschindler.com)
• The Problem With Planning (agile-software-development.com)
• Banks made $5.2B trading derivatives in 2Q (seattletimes.nwsource.com)
• The Enterprise 2.0 Crock (soacenter.com)