Different is Interesting

Last week, I was reminded of a lesson I learned from one of my mentors many years ago: when you’re regression testing, “different” is all you really need to look for.

A little context would probably help, here.  Nearing the end of a sprint, we’d tested the new features in the sprint, but we really weren’t sure whether we had good regression coverage.  Testing from prior sprints had part of the answer, but we couldn’t really afford the time to run all the tests for this sprint and all the tests from our prior sprints.  Thus, the conversation turned to regression testing, automation, and tactics to speed it all up a bit.

The naive approach to regression testing, of course, is to run every test you’ve ever run all over again, checking each value, examining the dots on the i’s and the crosses on the t’s.  It’s absolutely irrefutable and in the absence of automated tools, it’s also completely impractical.  With automated tools, it’s merely incredibly difficult, but fortunately, in most cases, there’s a better way.

Segue back to my years-ago epiphany, in which I was struggling with a similar problem, and the aforementioned mentor gave me a well-aimed shove in the right direction.  He pointed out that once I’d achieved “accepted”, and thus, “good”, all I needed to look for when regression testing was “different”.   All by itself, this didn’t do much to help me, because looking for “different” still sounded like a whole lot of looking.   Combined with the fact that our application was able to produce some artifacts, however, this idea vaulted our testing forward.

Our application, it turned out, supported importing and exporting to and from excel files, so we were able to use this to radically speed up the search for differences — we saved an export from a known good test, and future tests would just import this file, do some stuff, and export it back out again.  At this point, a simple file compare told us whether we’d produced the same output we’d previously validated as “good”.

Armed with this technique, we began looking for other testable artifacts — to the point where we built in some outputs that were used only for testing.  At the time, it was time well spent, because the maturity of testing tools made the alternatives pretty prohibitive.

And what do you do about a difference when you find one?  First reactions notwithstanding, it’s not necessarily a bug.  A difference could arise as a result of an intended change, so it’s not at all unusual to be able to attribute a difference to a change you’re actually looking for; in which case, you throw out the old baseline and replace it with a new one.  Just remember you’ll need to explain each difference every time you find one, which brings me to a final issue you’ll want to sort out if you’re going to use this technique.

Any artifact you want to use in this way must be completely idempotent — that is, when you run through the same steps, you must produce the same output.  This seems like a gimme, but you’ll find that a lot of naturally-occurring artifacts will have non-idempotent values like ID’s, timestamps, machine names, and so on — you’ll need to address these in order to have artifacts you can use effectively for testing.

Once you get past this little hurdle, you’re likely to find this idea powerful and super-easy to live with, and what more can you ask for in regression testing?

 

The Software Iceberg

I spoke with a software manager recently who’s considering selling some software that’s working great for their company. “We’re about 90% done,” he mentioned, with the implication that he’s nearly ready to unleash his creation in the marketplace, and I immediately recalled one of my earliest lessons in “product”, and one that’s echoed over and over through the years.

When it comes to productizing software, “done” is usually just a good start.

The prototypical software-to-product path starts with an application that’s built to solve a business problem. Once the software is proven, the leap to selling that software seems like a minor increment.

If you didn’t build productization into your work from the onset, though, you’ll need to address a whole host of concerns. Not all of these will apply to each new bit of software, but if you don’t have all this support in your company, you’ll need to develop them.

  • Branding. If you’re a widget company, you can be the best widget company in the world and still find it difficult to sell whosiwhatsits. Don’t count on a lot of umbrella support for a software product unless your main business bears a strong resemblance to that software, and if it does, be prepared for nervousness among your prospects – they’ll be (rightly) worried about you gobbling up their businesses once you’ve sold them your software. Branding, marketing and sales are first-class concerns if you’re really going to sell software.
  • Support. This is huge — your life will change as soon as you sign a customer to a software deal. You’ll need more than a guy to man emails and phones — you’ll need to be able to replicate problems your users are seeing, help them use your software to solve problems you’ve never encountered, and educate them on all aspects of your platform. If your software features any integration, be prepared for many multiples more demanded of you and your team.
  • Maintenance. Somewhat related to support, you’ll need a plan to maintain software over time if you’re going to keep your customers happy. This is no accident; your ability to release updates and new software versions across a number of customers can only occur with careful planning and precise execution.

Each of these needs is worthy of more pages of discussion, but suffice it to say these are all contributing reasons why software products are so much more challenging than internal applications. In my opinion, this also brings greater reward.

Microsoft HomeOS

I’m not sure how I missed this up till now, but Microsoft is apparently looking into the same home automation space that Google is trying to gobble up with its Nest acquisition.  Microsoft’s project (still in their research lab) is called HomeOS, with an accompanying software kit for devices called Lab of Things.

I’m really not sure whether to laugh or cry yet, because having long been a proponent of Microsoft to go kill this market, it really seems like they’re still missing the real bread & butter features needed to start printing money.  I watched the demo video on the HomeOS page, and just about the whole damned demo is about seeing appliances blink on and off in response to arbitrary events — all on a Windows phone, of course.  They finally touched on user access at nine and a half minutes into the ten-minute video, despite that being arguably the most important feature of all.

So listen, Microsoft — if there’s anyone out there still interested in making money, here’s how you do it.

  • HomeOS (or whatever this product winds up being called) needs a form of ActiveDirectory.  One place — for real — to set up and administer users, including extended family members, friends, guests, etc. — with a single interface so I can light up guest WiFi and TV streaming for a guest, for instance, without needing to hit four or five different devices.  Once this is in place, I promise I’ll never buy another device that doesn’t authenticate against this directory, ok?
  • Speaking of devices, I’m actually way less concerned about electronic doorbells than I am about XBoxes, NAS devices, and routers.  Get this stuff to work together really, really seamlessly, and you might single-handedly save BestBuy.
  • Continuing on the theme of devices, I’ll repeat my recommendation to buy Drobo.  Host your HomeOS on it, including AD, and package it to look like (and live next to) an XBox, including high-speed interconnection.  BOOM!  Microsoft owns the living room.  The damned thermostat is an afterthought after that, you know?
  • Let me tunnel to the HomeOS box I buy for Mom & Dad.  They’re not going to administer their own smart home, even if they can accomplish incredible things in less than 250 lines of C#.  (Are you kidding me???  Is that *really* a feature??)
  • If you’re going to let me tunnel to other HomeOS locations, it’s a pretty short leap to go ahead and buy a router company, too, isn’t it?  Plug that bad-boy right into my X-Drobo-thing, and configure it from the same interface.  Make it easy enough for my nephew to set up, and keep it secure, too, if you think you can pull that off.  That’d be great.
  • If you require me to use a Windows phone to use all this warm fuzziness, you can flush it all down the pot.  You’re too late on this one, and you’re going to have to embrace Android and maybe even IOS.  You should have listened to me earlier on that one.

Seriously, Microsoft — if you manage to snatch defeat from the jaws of victory on this one, I’m not going to feel too sorry for you.  This should be a big ‘ol gimme — just by re-assembling some tech you’ve already got.  Give it a shot — I’ll bet there’s plenty of time to sell brilliant doorbells later.

Home PC administration — another lost opportunity

I’ve written in the past about places where Microsoft could absolutely *own* the infrastructure of the home by establishing a beachhead in the living room — not to mention the previous assertions about their development tools.

I still believe quite strongly that a well-targeted home computing platform is just a couple of software tweaks away for Microsoft.  Today’s edition is all about authentication.  I’ve got a bunch of PC’s at home, including some VM’s.  I’ve also got a Drobo 5N and a PS3 and a bunch of networking equipment.

You know what stinks?  I need to set up logins on every single one of these devices individually, and they’re not connected to one another (so “Fred” on one box isn’t really the same login as “Fred” on another box).

Stupid.

Microsoft, give me a lightweight Active Directory for the home — something I can obtain without buying a Windows Server license, okay?  Here’s a hint: if you built this into thew new XBox, I’d buy one, and I bet a bunch of other people would, too.  Let me use this for DNS, so I can type “router” into my browser and actually get my router, instead of making me set up a HOSTS file on every single PC I own.  By the way, how many average consumers would even know that’s possible??

I fully expect that the new XBox, when it arrives, will let me stream photos and music off my Drobo, but if you want to really take this idea to the next level, how about selling us a Pogoplug -type of device I can give to my Mom & Dad so I can (1) set up user names for them, and (2) let them see photos that I don’t plan on uploading to Flickr, etc.?  The idea here, by the way, since I’m spelling everything out in excruciating detail, is that just about every family has one or more members somewhere who (a) own a gaming system, and (b) understand enough about computers to be the family SysAdmin.

Get it??

Oh, and by the way, since you’ve given up on Windows Home Server for reasons I’ve never quite been able to fathom, and since you now aspire to be a “devices + services” company, why don’t you just go ahead and buy Drobo and make their stuff work with yours?  I’d happily plug mine into a new XBox.

I swear, if Microsoft were able to get their collective heads out of whatever orifices they’re lodged within long enough to make an XBox that actually acted like it was part of a family, they’d crank up another WinTel-style monopoly to last them a good dozen more years.

Enhanced by Zemanta

Why is it so hard to buy a standard developer workstation?

It’s spring, 2013, and Intel has just released its Haswell processors and chipsets.  Motherboard vendors are touting their new wares and all the manufacturers are announcing new products.  As luck would have it, it’s about time to refresh workstations in our office, too, so it’s a fine time to take stock of the current state of the art for desktop systems.

So, without further ado, if you’re a developer, you want a system along these lines:

  • Intel Core i7 — Haswell is great, but Ivy Bridge would be fine.
  • At least 16 GB RAM.
  • An SSD boot drive – 128-256 GB.
  • Mirrored 7200rpm data drives – around 2-3 TB.
  • Integrated graphics are ok unless you do any serious graphics work.

As an aside, I checked my notes from 2009, and these are almost identical to the specs I put together the last time I looked at systems.  We’ve gone through a couple generations of CPUs and chipsets, and the “sweet spot” for storage buys about double the capacity now, but the rough idea is about the same.  I figure the cost for a system like this is down a third since then, too.

Incidentally, if you’re a professional in a content-generation field (web design, illustration, photography, video), this is a decent starting spot for you, too, though you’ll probably want to toss in a stout video card to help with the graphics.  Although you might be tempted to save a few bucks here or there, every single one of these elements is there because it adds value for a professional who relies on his equipment.  Nothing about this configuration is exotic or surprising.

Despite this, I continue to be astounded that nobody sells systems that look like this.  Obviously, you can build it yourself, and if you know the first thing about computers, I highly recommend this — you’ll wind up getting better parts, and there’s something to be said for knowing your kit is built right.  Some shops would obviously rather buy PC’s than pay people to assemble them, though, and so it is here.  Off to Dell.com, then.

Before I start ripping Dell, I’ve got to point out that I’ve generally been a fan of theirs.  I’ve used something approaching dozen of their PC’s and laptops at work over the years, and I’ve got a Dell laptop at home that’s recently been retired because it stopped charging. I used to have a Dell 1U server in my basement running VM’s, in fact.  Noisy, but a nice little machine.  I’ve got no ax to grind with Dell, per-se.  Despite this, they’re dead to me now.

I’ve always found their product lines to be a bit too complex, and their configurator is just about as much fun as a poke in the eye with a sharp stick.  I’d forgotten about this, but I shopped their site back in 2009 to see if they could build a system along the lines I indicated up front.  They couldn’t.  Back then, this was disappointing, but not terribly surprising.  Intel’s RAID-enabled chipsets were fairly new, so practical mirroring was fairly novel, and SSD’s were just beginning to trickle down from enthusiast systems.  Today, both of these things should be considered absolutely mainstream.  I honestly don’t understand how both of these features aren’t considered standard equipment for anyone who makes more than minimum wage.

On top of not being able to build the system I wanted, the web site absolutely blew chunks.    As soon as I visited the site, I got one of those “Will you leave your opinion?” pop-ups, and every time I selected a system, the page scrolled over to the top-right, where a “Chat with Dell” window appeared, offering me help.  Offer accepted.  Needless to say, “Brandon” wasn’t able to help me: “…and no workstations that we have will allow an SSD boot hard drive and then mirrored 2nd and 3rd hard drives.”  No kidding.  I also participated in their feedback session.  At the end, they asked if I had any additional notes for them.  I did:

The last time I tried shopping for a system here was 2009.  At that time, I was looking for a Core i7 desktop with around 16GB RAM, an SSD boot drive and mirrored data drives.  I couldn’t find one.  Today, I tried shopping for the same thing, and guess what — can’t get there from here.  This is a standard configuration for developers, and you literally can’t buy it from Dell.  You guys might want to worry a little less about how your buyout is going and more about building PC’s.  Just sayin’.

So, that was Dell.  The good news is that HP fared better.  I opened the site without drama, and found a desktop right off the bat that would support the configuration I wanted. The system I wound up configuring used an Ivy Bridge processor instead of Haswell, but at least I could get something close.  Dell, I know you probably sell far more PC’s to secretaries and call center workers than you do to developers, but if you don’t hold the high ground, you’re going to get your head handed to you on commodity systems.

Anyway, it’s been nice knowin’ ya, Dell.

Enhanced by Zemanta

In design, details matter

Have you ever experienced a cascading menu that seemed to run away from you as you navigated it?  This is one of those subtle usability failings that can lead to a disembodied hatred of a site or application.  Very few people will notice what’s actually going wrong, let alone what should be done to fix it.

Galileo
Galileo (Photo credit: dglambert)

Ben Kamens noticed when Amazon got this right.  Not only did he notice, he wrote up what he found and developed a jQuery menu you can use on you own site to achieve the same fix.  The improved implementation, by the way, has its origins in noticing what direction your mouse is moving — if it’s headed toward a sub-menu, this implementation gives you a chance to catch it.

The moral of the story?  When you get this stuff right, most people never notice the details, but they’ll notice the feeling of quality in the product.  Nobody loves an Apple iAnything because the edges are chamfered exactly so or because the icons are rounded just a bit, but they notice that it feels solid and sorted out.  I’ll bet you’d have a hard time finding people who can tell you exactly why a BMW feels better than a Chevy, but most of them will agree that it does.

As a designer, it’s important that you do, in fact, notice the little stuff, and that you understand how these details contribute to the quality of your product.  With any luck at all, you’ll work for an organization that also gets this stuff, because you’ll also find it pretty frustrating to try to explain details like this to a bean-counter that hasn’t got any awareness of this relationship.

Enhanced by Zemanta

Pardon the inturruption

A while back, I experienced a small hacking disaster here, and although it took a while to get that all sorted out, I think I’m finally back up and running.  I honestly had no idea it’d been this long, though…

In any event, I hope to get back to illustrating some of my more colorful software journeys here, so if you’re rejoining me after the break, welcome back, and if you’re joining me for the first time, thanks for the visit!

Five keys to IT Operations

Any developer who builds a new system dreams of the day their software goes live, and real users start pounding on their application.  This is perhaps the most tangible validation of their work that some developers ever have.  On that day, it’s pretty common for developers to either be the team responsible for running the system, or at least to be working shoulder-to-shoulder with the people who are running it.

For those lucky souls who “grow into” this role, though, it may be helpful to have a sense of perspective about what success in Operations means.  Although specific technical and design details are aligned with your implementation, your job ultimately boils down to some simple objectives:

1. Understand what’s supposed to happen

Eugene F. "Gene" Kranz, provided by ...
Gene Kranz – Image via Wikipedia

While Development is all about designing, building, and testing, Operations is all about Execution.  Go rent Apollo 13, or better yet, pick up a copy of Failure is Not an Option by Gene Kranz.  Happiness in Operations means understanding what’s going to happen before it happens, and there’s no such thing as a good surprise.  You can see this in the checklists that all the NASA engineers used, and when someone tries to tell you that you’re too smart for checklists, remember that those guys were, in fact, rocket scientists.  Nuff said?

Virtually nothing in Operations is of any use whatsoever unless everyone who’s touching a keyboard knows the exact same stuff, which implies that checklists are written down in a way that ensures everyone’s got the same information.  If you don’t have anything better to start with, get a whiteboard and write, “8am: Make sure the servers are still on,” and then start filling in from there.

2. Know what’s actually happening

This is the IT equivalent of watching the gauges on your car’s dashboard.  Failure to pay attention to the gauges could prove costly when your engine starts shooting connecting rods through the hood.

In operations, you’re watching for failures and performance problems — hopefully in time to react to them before your customers start complaining, and you’re watching for unusual activity that could indicate problems with other systems you interface with or even hacker activity.  When you get more sophisticated about what you’re watching, you may even be able to provide design guidance on what features your customers are using the most, or whether there are parts of your application(s) that users seem to be struggling with, but please make sure you’re covering the basics first – server uptime, exceptions, and application performance.

As usual, tools help here.  It’s a whole lot more efficient and effective to have a tool checking to make sure servers are responding properly.  Fortunately, there are all sorts of tools like this, including some free ones.

Important:  Be sure to understand the difference between IT Operations and Business Operations.  These can, in some cases, be co-resident, but remember that one is focused on your systems and the other is focused on the business.  These two aspects of Operations should communicate liberally back and forth, but it’s important to understand the difference between technical status and management and business status and management.

3. Communicate status

Since it’s Operations’ job to know what’s happening, they therefore serve as a fount of knowledge for other departments.  In a lot of cases, proactive communication is more effective than “pull” communication, and again, whenever you can drive decision-making out of the process, it’s a good thing.  Therefore, operations should know in advance what sort of events should trigger communication, and to whom they’d be communicating.  Some of this could, in fact, be automated.

Status is typically focused on what’s happening right now, but a complete understanding of status also includes a sense for whether measures are trending in one direction or another.  Data about how our system performs over time, for instance, can tell you a lot about whether a performance metric you’re seeing right now is a blip or part of a trend that’s moving steadily toward a big problem.  This sort of long-term information should also help us see performance or resource constraints in time to react to them before they affect customers.

4. Handle catastrophes

Sometimes, bad things happen to good applications.  When the sh*t hits the fan, it’s absolutely imperative that the cure isn’t worse than the disease.  Go watch Apollo 13 again.  Since everything that normally happens in operations should happen according to a checklist or procedure, it should be glaringly obvious to everyone (to the point of discomfort) that you’re now operating off-script.

I’ve heard pilots describe their jobs as “hours of boredom punctuated with seconds of sheer terror.”  This is when you want to open the cockpit door and see Sully sitting there.  Sully uses checklists, too, by the way.

5. Maintenance and planning

Since operations has done such a good job of ensuring our system is running like a Swiss watch, they’ve got some time left to plan for future improvements.  With any luck, this might include stuff like:

  • Preparing and managing hardware and/or virtual servers.
  • Planning infrastructure changes for upcoming software releases.  This is actually a very important form of developer support, because this is where operations and development work together to make sure you can deploy the things you’re building without any undue drama.
  • Tuning / tweaking system monitoring and management tools.
  • Analysis to assist development – where are your servers stressed, what custom tasks do you deal with today that might be built into the application, etc.

This list is just a start, of course, but it’s a pretty good start.

What tips would you add?

Enhanced by Zemanta

Reason #358 why I hate Flash

I use four PCs on a regular basis (two work PCs, plus a laptop and desktop at home), and all three run Windows — one Windows Server 2003, one Server 2008, and two Windows 7.  All of these boxes are either on 24×7 or hibernated between uses, so the only time I reboot them is to install Windows updates.

And every… single…. time… I reboot any of these machines, I see one of these:

I typically go ahead and let Flash do what it wants to do, and yet it keeps coming back, over and over and over again.  Based on this, I’m forced to conclude that either (1) Flash isn’t really updating correctly, or (2) it really does have a new update to install every single time I reboot.

Neither of these is acceptable.  Adobe, you’re not building an OS here.  Get it right and get out of my way.  If there’s  a *real* new version or a *real* security disaster, then let me know about it, but I just refuse to believe that there are really that many emergencies that you need to install something every single time I reboot.

If you’re wondering why folks like Apple have made such a big stink about getting Flash off their systems, this is exactly the sort of issue they had in mind.

The bug “event horizon”

Black holes are fearsome astronomical phenomenon that are so dense and have a gravity well so deep that their infinitely-large mass is compressed into an infinitely small space.  Anything that approaches these monsters is almost certain to be sucked into the hole, and there is a region surrounding each of them where escape is impossible, even for light itself;  it is called the Event Horizon.

The supermassive black holes are all that rema...
Image via Wikipedia

Bugs are like that, too.

A Bug’s Life

First-year software development students (and even MBA’s) learn that bugs in software are more time-consuming (and thus, expensive) to fix the further along in the software development process they’re caught.  The easiest bug to fix is the one you prevent in the first place with proper architecture and design.  Sadly, these quantum bugs are hard to quantify, so good design rarely gets to take credit for these precognitive fixes.

Of the bugs that are actually found, the ones caught during development (perhaps by TDD or unit tests) are wonderfully cheap.  In many cases, mere seconds pass between detection and eradication, the developers’ fingers never lifting from the keyboard. These bugs, in fact, are the next best thing to the ones that never existed in the first place, since most of them are never recorded in a bug tracking system and are probably never seen beyond the desk of the developer who found and fixed them.

After code is checked into a version control system, bugs move through a build process and on to testers, while simultaneously being propagated to other developers’ desktops via the source code.  Bugs found at this point are slightly more costly to fix, because you’ve involved other people, and there’s very likely a process and tracking system engaged to help keep track of the critters at this point.

But there’s another reason these bugs are more costly to fix — context switches.  As soon as a developer checks in his code, he begins moving on to his next task.  He’ll forget all about the code he was working on, including in some cases tearing down whatever virtual infrastructure existed to develop that code.  When he’s just about eyeball-deep in his next task, that bug comes home to bite him — a big context switch.  If the bug is urgent, he’s going to have to drop everything and get back up to speed on that code again, leaving behind any progress he managed to make on the new project.

The New Normal

Given that a developer’s job is to develop, you’d like him to be able to devote his full attention to creating solid designs and stout code.  In most cases, the work we’re asking of these people is really fairly difficult to do well, and it can be nearly impossible if he’s not able to concentrate.  When design is interrupted by a bug or two per week, there’s very likely an impact in terms of productivity as the developer changes contexts, but concentration and software design shouldn’t be adversely affected to a large degree.

As interruptions become more frequent and prolonged, however, there can be an impact in the quality of new code that’s produced, as well.  Design becomes disjointed and inconsistent.  Code becomes sloppy.  More new bugs are introduced.  We’ve introduced a vicious cycle of downwardly-spiraling quality.

As bug counts grow, they can start to have an impact beyond development and QA areas.  Help desks can become buried in calls; bugs begin to go un-recorded and uncorrected.  The noise level caused by poor software quality can contribute further to the downward spiral of quality.

Singularity

When code quality suffers so completely that the organization cannot produce high-quality code, the transformation of the organization is complete.  Buggy code eventually results in even buggier code, and software collapses upon itself as if it was a black hole.

Is the Demise Inevitable?

Unlike a real black hole, the problem of software quality can be stemmed — especially if it’s caught early.  Much like bugs themselves, though, the later this issue is addressed, the more difficult and expensive it will be to fix the problem.  When you consider the cost of bugs in your organization, don’t forget the cumulative effects of bugs on the quality of the work you’re able to produce.

 

Enhanced by Zemanta