Process and culture drive reliability for Shuttle software

Fast Company has a great article about the team that builds the real-time mission control software that runs Space Shuttle flights. The team, the mission, and the statistics are all over the top; as the article points out, very few software projects carry the dreams of a nation when they run. There are lesson here for the rest of us, and this software team would have us understand that improvement is more about the process than the technology. To be fair, the software we write every day isn’t (ahem) rocket science in the same way that Shuttle software needs to be, and we don’t have some of the advantages that this team has:

  • A large staff. This team is comprised of 260 men and women.
  • Mature code base. This software has been in development for over 20 years. As long as people have been paying attention, there are a lot of lessons that should have been learned by now.
  • Stable work force. Many of these engineers have been in place for a good portion of the Shuttle’s life cycle.
  • Best of the best. Let’s face it – few of us can compete with the prestige of working on the Shuttle program

Nevertheless, there are some ideas in here we should consider carefully in terms of how and where they apply to our work. The two themes that keep popping up in this article are process and culture.

The process in this group is tightly-controllled, double-checked, documented, reviewed, and revised. These guys understand implicitly what it means to go off-script, and they don’t let it happen often. I’ve always championed the use of the right amount of process for the job, and in this case, the job calls for planting a handful of astronauts on the top of a large, pointy bomb, lighting the bomb in such a fashion that it squirts into the upper reaches of the atmosphere at a speed approaching 20,000 mph, and then landing the astronauts back to earth aboard the spent bomb. This takes a sophisticated process: there’s no service pack in space.

The culture is also striking. Unlike the go-go silicon valley “cowboy” coders, these developers are methodical, measured, and deliberate. They work regular hours and have lives outside work. In a word, they’re boring. This, as it turns out, is comforting to the astronauts, who would much rather get their thrills from the great views in space than from “excitement” in their mission control software. This team understands that the mission is the important thing, and the software exists only to support the mission.

Think about your development shop a bit, and then read They Write the Right Stuff.

Fix an Arctic 7 heatsink

Abour two years ago, I built a new home PC for myself. I got a nice Antec Sonata case, and a MB with variable-speed fans to cut down on noise. When I put everything together, I was very happy with the sound level except for the stock Intel CPU fan, which idled quietly, but sounded like a vacuum cleaner when it kicked into high speed. I decided to upgrade the heat sink and fan, and decided on an Arctic 7 Cooler, which ran very quietly until the last few weeks.
I started to notice my PC getting noisier and noiser, and when I popped the side off, I could tell that the noise was coming from the fan on the Arctic Cooler. I just got done fixing it, though, so if you’ve got a noisy Arctic 7 Cooler, don’t throw it out — try some machine oil.

(BTW, in browsing for a picture of the Arctic 7, it looks like they’ve upgraded the fan in the mean time, so the problem and the fix here very likely only apply to the older units)

First, you’ll most likely have to remove the heatsink in order to oil the fan. Next, look for a little plastic plug on the back of the fan assembly (see picture). Pop this off with a knife. Next, just send a few drops of 3-in-1 or similar machine oil down the hole and spin the fan a bit to work it in. That’s it! Reassemble everything (you may want to freshen the heatsink compound if you overclock your CPU), and you should be good to go.

Microsoft = anti-virtualization?

I’m convinced yet again that Microsoft’s licensing weenies will be the first against the wall when the revolution comes. I just read an article by Paul Thurrott that describes licensing changes for Windows Vista, and it’s really, really disappointing to me. In short, Microsoft is cranking down licensing terms to specifically restrict which versions of Vista will be “legal” to run in a VM.
In his article, Paul carefully explains that the Vista licensing terms aren’t all that different in effect than XP’s, and that the VM restrictions exist because VM users are “fringe” use cases. This is, in my opinion, short-sighted at best, and destructive at worst.

History Revisited

Rewind five years to the release of Windows XP. Virtualization was, at that point, a toy technology. A reasonably computer-savvy person could imagine the potential that virtualization might eventually yield, but the benefits were clearly over the horizon. There was no such thing as dual-core, let alone quad-core, and the idea of CPU’s with virtualization accelerators built into them was inconceivable to the average technologist.

Windows XP’s licensing terms were, therefore legitimately onerous to computer enthusiasts who changed PC components frequently and faced the headache of repeated product activations, but there we really no virtualization customers against whom Microsoft could discriminate. Bottom line: Microsoft was about as abusive as they thought they could get away with towards the users they knew about at the time.

Back to today. Now, Vista’s licensing terms remain similarly invasive for “regular” users, but now, Microsoft’s also targeting VM users. Paul points out that most VM users are either professionals or enthusiasts, implying that this is why only the “Pro” and “Ultimate” versions are licensed for use in a VM. If you want to run Vista Home Basic or Home Ultimate, you’re out of luck. Can’t go there.

This is ludacris from so many angles, it’s hard to know where to start railing, but I’ll give it a shot.

Special Users

“Currently, the majority of Microsoft’s virtualization users fall into exactly two groups: business customers and enthusiasts,” says Paul.

Ok — I’ll buy that, subject to one caveat — “currently”. If we’ve learned anything about virtualization over the last five years, it’s that it’s here to stay, and, in fact, its use is going to grow. Where do you think we’ll be on the adoption curve in another five years?

Next problem: just because your VM users are professionals and enthusiasts, that doesn’t tell you squat about the OS’s they want to run. In fact, I’d bet that these groups are more likely than other groups to *want* other versions of Vista. The two biggest reasons for VM adoption today are operational management of production servers and testing. Production servers aren’t generally going to use desktop Vista, so I’ll leave them for another day. Let’s talk about testing.

The use of VM’s for testing has a number of benefits. One of them is that you can set up a VM for a specific customer configuration. Another is that you can replicate combinations of hardware and software that exhibit “interesting” traits with respect to the software you’re testing. Finally, using VM’s let you power-on machines when you need them and power them off when you don’t, and in the time behind runs, they take up zero extra space in your test lab / office, and they consume zero additional electricity.

This implies that you need to run whatever edition of Vista your customers are running. If you build an application that supposedly runs on Vista Home Basic, you should be expected to test on it. If you’re a software developer, that means no VM for you!

If you use a VM’s to support custom installs or configurations for your customers, you probably have VM’s for each customer, and most of them are probably turned on only when you’re chasing a customer support problem. That’s a lot of Windows licenses sitting idle most of the time.

What’s so Hard?

Now that we know that virtualization is here to stay, why in the world can’t we look it in the eye and deal with it? What, exactly, is Microsoft trying to protect itself from by cranking down these licensing terms?

Piracy.

How do you tell the difference between piracy and the legitimate use of VM’s for testing, as I described it above?

When you use VM’s for testing, most of them aren’t on most of the time.

Once more, in small words so the licensing lawyers can get it: Please don’t make me pay for an OS that only gets turned on once a year.

Microsoft’s got how many gazillion engineers now? I’m pretty sure that Microsoft could divert a couple dozen people off the splash screen team and fix this once and for all. We need concurrent licensing for Microsoft OS’s — especially when run in VM’s. It’s that simple. VMWare and Virtual PC both have “VM tool” installs they provide for guest OS’s to “enhance” operation of the guests. Use these to hook into a licensing service provided at the host level, and (for bonus points) make it networked. VM’s don’t chew up a license when they’re not on.

So, Microsoft, go find the guys who are writing Zune software and put them to work on Vista licensing for VM’s. After all, Zune’s walking dead.

Other takes

A couple other pundits have words on this subject, too.

Dialogue Box by Chris Pirillo: Vista Will Double Appleā€™s Market Share

Koroush Ghazi: Windows Vista’s Enthusiastic Licensing Restrictions