Beta Testing: A glimpse inside the MS sausage factory

Windows Home Server Console
Image via Wikipedia

I saw a pretty entertaining post this week on one of the Channel 9 forums. A guy there reacted to some Windows Home Server bug counts from their beta program. Personally, I found this guy's reaction to be humorous all by itself:

I definitely like this topic:

Of the bugs that have been addressed, Sullivan said that only 15% have actually been fixed. The remainder are issues that are in the server by design (13%), not reproducible (21%), will be postponed to later versions (11%) or likely won't be fixed (7%).

Bloody hell... ALL REPRODUCIBLE BUGS MUST BE FIXED! (Or never made!). There is no such thing as "bug by design". It was invented by big corps to cover own failures, as "excuse".

An energetic, uninformed rant is always worth a little chuckle, but in this case, my interest extends beyond the first reaction. You see, this person represents a real constituency out there who know enough about a software development process to be dangerous.

To ship software, just fix every bug until none remain. Any software that ships with > 0 bugs is proof that the software company is just a money-grubbing corporate blight on the face of the industry.

Simple, right?

Ok, fess up - how many of you have really shipped commercial software? How many have shipped software out for testing, and then fielded unsolicited "help" from these testers. I've done these things on a smaller scale than Microsoft is doing here, and the numbers don't startle me a bit.

In fact, if you develop commercial software or plan to do so someday, I'd recommend you start watching the Windows Home Server Blog. I know I was critical of the opacity of the initial marketing launch for this product, but the subsequent transparency of the development process is really impressive. Pay attention, because you're getting a glimpse inside the sausage factory.

You see, when you open testing up beyond a tightly-controlled QA environment, there are only two ways to get a grip on your bugs. You either throttle them before they come in, or you sort and categorize them after they're reported. The former method is almost always impractical, not to mention making the tester feel oh, so valuable ("Thanks for the report. I'll just put it here on this stack to review before I enter it into the system. We'll be sure to let you know if it's entered as a 'real' bug").

Categorization is really your only option for a beta program like this, and when you let anybody report any bug, you're going to get a lot of chaff. Again, the distribution reported by the WHS team looks pretty reasonable, and the team is doing a great job of explaining the numbers along the way.

I guess it's hard to fault Microsoft for being opaque sometimes when openness like this is turned against them by a misinformed public. Still, I hope we see more of this face of Microsoft.

Reblog this post [with Zemanta]

Miguel de Icaza: Another blog you need to read

If you're a .Net developer, manager, or architect, you've got to read this guy's blog. Who is Miguel de Icaza, and why should you care?

Miguel is a the driving force behind Mono, the open-source port of Microsoft's .Net CLR. I've been watching this project since Miguel first began working on Mono 1.0, and his progress has been really impressive.

To be honest, it seemed like an uphill battle when he started, and when he hit 1.0, Microsoft was releasing 2.0, so I figured there was no way he was going to be able to keep up. But he and the other Mono developers (with some support from Novell along the way, if I'm not mistaken), have kept on driving, and the movement is growing.

They're supporting .Net 2.0 features now, and along the way, they've picked up some of the UI tools needed to support WinForms, plus support for VB.Net. Perhaps more importantly, though, are some of the ancillary projects popping up around Mono. There are open-source IDE's, and a great product called Grasshopper that lets you deploy your .Net app on a Tomcat Java application server running on Linux (Grasshopper is built by a company called Mainsoft who sells a commercial product that lets you do the same thing, but deploy it to Enterprise-class app servers on any hardware platform).

So why is Miguel's blog a must-read?

It's all about understanding your options. As a developer / manager / architect, you need to understand how the technology you use connects to the needs of your business. Mono could be an important part of that in the future.

Microsoft has paced the field in development tools for a long time, and by most accounts, they're still tops. It's pretty hard to top .Net's development experience, despite the great gains by Eclipse over the years. The weak link in an all-Microsoft development strategy is that it implies an all-Microsoft deployment strategy. This can feel burdensome from time to time, and like like cement shoes whenever there's a new OS release (upgrade or pressure).

What would it be worth to your company to develop in .Net and distribute onto Linux servers? How about for a hosting company? Can Mono turn into the next LAMP?

Miguel doesn't post often, but when he does, it's worthwhile. Sign up with your favorite RSS reader, and you'll know the next time he has some news. Check out today's post, The Race to Linux 2.0, where he talks up an upcoming contest to pump up Mono's 2.0 support.

Scott Adams is watching you!

Scott Adams (of Dilbert fame) has a blog, and I've been a subscriber for a while. In today's blog, he's gotten himself pretty wound up about Google's ability to email you an alert whenever your keywords appear for the first time in its massive index. Pretty cool, I'll admit.

He's asked his readers to chime in on a technology that they think is cool. Normally, of course, I'd just drop a comment on his blog, but in this case, I think I'll just let Google send him an email. 😉

So, what do I think is cool?

I'm a software guy. The mechanics of an individual piece of software frequently impress me because I can appreciate the work it takes to get an application to jump through hoops. I was awe-stricken when I saw Google Suggest for the first time. But that feeling quickly fades. The relentless advance of software marches on, and pretty soon, you can get Ajax free in a box of breakfast cereal.


But here's what's cool. There's finally enough standards-based infrastructure, and enough collective development gray matter, and enough boardroom comfort with technologies like XML, web services, RSS, Ajax, and Flash that people are moving past the wow of the technologies and beginning to build some pretty useful applications.

Once, as a friend of mine used to say, it wasn't necessary for the dancing bear to dance well for it to be impressive. Any dancing bear was a good show, or so I've been told. Today, though, if your bear isn't ready for "Dancing with the Stars", you're yesterday's news. The bar has been raised.

So the cool thing for me is to be aware of the immersion that we broadly tag "Web 2.0". Here are some of the things that are so cool, we don't even stop to appreciate them anymore:

  • RSS News Readers. I've loved RSS since way before it was cool to love RSS. I'm a news junkie, and RSS is a mainline into my frontal lobe. The cool part is that it's seamless now. When RSS started, it was rare to find a feed, and hooking it up was accomplished with all the grace of - well, a dancing bear. Now, every site you go to lights up a little orange button on your browser, and when you hit the button, you're done - no drama at all.
  • Social networking that works. The phenomenon that was MySpace tuned everyone in to the scale that social networks could achieve. There were online bulletin boards and email lists, of course, back in the dark ages of the internet. While this technology was found exclusively in dark rooms warmed only by the glow of computer monitors, it never had a chance of becoming mainstream. Today, though, social networks are fun, but they're also real business tools. LinkedIn is at the top of my list, and MyBlogLog is gaining ground. I've made several quality acquaintances through these networks; really top-notch business leaders who I very likely would never have met before.
  • Virtual Machines. Ok, you're not going to get too warm and fuzzy about this one unless you're a technophile, but I think it's pretty cool, and you will, too eventually. I've experienced the breathtaking excitement of shoving thirty-some diskettes into a computer, one-by-one, for hours on end to get OS/2 or Windows-Stonehenge-Edition set up, only to find that you bungled the video setup, and you were going to have to start over because you foolishly chose a refresh rate used only in an obscure part of Lower Elbonia, and you were now sadly unable to read the screen where this error was blinking helpfully. I've also endured ten years of developers telling me, "it works on my machine!" -- as if this somehow mitigates the fact that the QA system has turned into a boat anchor. There's something to be said for setting up software by copying a Virtual Machine file to a server and powering it up, and for restoring a bricked system by copying yesterday's image back over the smoking remains of today's.

So those things are pretty cool. We can tell that a technology's really made it, though, when we start seeing new tech built on yesterday's cool tech:

  • Podcasts. I hesitate to call this new tech, but it's a perfect example of cool tech that's built on platform tech -- in this case, RSS. To the untrained eye, RSS is nowhere to be found in your typical Podcast, but that's just the way it should be. This is a harbinger of things to come -- we're now starting to see video-cast feeds, completing the multimedia sweep.
  • Web Services. You can now snipe an eBay auction or tag your MP3 collection with album art from Amazon using web services, and never be aware of how it's happening. That's seamless, and that's cool.
  • Yahoo Pipes. This one is out there a little bit, but it shows how multiple Web 2.0 technologies can do when they come together. Sadly, Pipes is so cutting-edge that it's down from time to time. Like now.
  • Mashups. Not a day goes by without a half dozen new Google Maps mashups. Or a Technorati / Flickr combo. Or any other crazy combination that someone can think of. The apps are sometimes pretty goofy, but the fabric that enables this Frankenstein-inspired architecture is big-time serious, and it's ready to make a difference in your business.
  • One-touch link blogging. When I'm reading your RSS feed in Google Reader, I press one button if I like your article, and it's instantly re-published on my Google Shared Feeds link. Delicious and CoComment sit on my toolbar and store my links and track the comments I leave on other peoples' sites.

And we almost take all of this stuff for granted. It's infrastructure; it's on all the time at the flick of a switch. And *that* is wicked-cool.

So, by the time you're through reading this article, Scott Adams says his Blackberry will be buzzing. What's up, Scott?