Different is Interesting

Last week, I was reminded of a lesson I learned from one of my mentors many years ago: when you're regression testing, "different" is all you really need to look for.

A little context would probably help, here.  Nearing the end of a sprint, we'd tested the new features in the sprint, but we really weren't sure whether we had good regression coverage.  Testing from prior sprints had part of the answer, but we couldn't really afford the time to run all the tests for this sprint and all the tests from our prior sprints.  Thus, the conversation turned to regression testing, automation, and tactics to speed it all up a bit.

The naive approach to regression testing, of course, is to run every test you've ever run all over again, checking each value, examining the dots on the i's and the crosses on the t's.  It's absolutely irrefutable and in the absence of automated tools, it's also completely impractical.  With automated tools, it's merely incredibly difficult, but fortunately, in most cases, there's a better way.

Segue back to my years-ago epiphany, in which I was struggling with a similar problem, and the aforementioned mentor gave me a well-aimed shove in the right direction.  He pointed out that once I'd achieved "accepted", and thus, "good", all I needed to look for when regression testing was "different".   All by itself, this didn't do much to help me, because looking for "different" still sounded like a whole lot of looking.   Combined with the fact that our application was able to produce some artifacts, however, this idea vaulted our testing forward.

Our application, it turned out, supported importing and exporting to and from excel files, so we were able to use this to radically speed up the search for differences -- we saved an export from a known good test, and future tests would just import this file, do some stuff, and export it back out again.  At this point, a simple file compare told us whether we'd produced the same output we'd previously validated as "good".

Armed with this technique, we began looking for other testable artifacts -- to the point where we built in some outputs that were used only for testing.  At the time, it was time well spent, because the maturity of testing tools made the alternatives pretty prohibitive.

And what do you do about a difference when you find one?  First reactions notwithstanding, it's not necessarily a bug.  A difference could arise as a result of an intended change, so it's not at all unusual to be able to attribute a difference to a change you're actually looking for; in which case, you throw out the old baseline and replace it with a new one.  Just remember you'll need to explain each difference every time you find one, which brings me to a final issue you'll want to sort out if you're going to use this technique.

Any artifact you want to use in this way must be completely idempotent -- that is, when you run through the same steps, you must produce the same output.  This seems like a gimme, but you'll find that a lot of naturally-occurring artifacts will have non-idempotent values like ID's, timestamps, machine names, and so on -- you'll need to address these in order to have artifacts you can use effectively for testing.

Once you get past this little hurdle, you're likely to find this idea powerful and super-easy to live with, and what more can you ask for in regression testing?