Tochi's Hut

Someone posted on Slashdot today that Google started a new blog on the all-important subject of the software testing. Cutely named, "Testing on the Toilet", the blog will feature weekly episodes of improving testing coverage by presenting some not-so-obvious examples in which perfectly innocent looking code can fail.

Writing testcases is probably one of the most daunting tasks in any software project. As a developer, I always have a hard time coming up with good set of testcases for any code I write. You usually design code so that it's elegant, modular, and efficient. Unfortunately, more often than not, there are bastard edge cases which crop up after I declare the code "perfect". Fixing these "bugs" often makes my code look fugly. :( Knowing how poor of a tester I am, over the years, I have learned to appreciate the awesomeness of a good test team.

The first "project" in which I paid dearly for the lack of good test coverage was in some 400 level semester-long project in university. While I won't mention much specifics, it didn't really help that we were pulling consecutive all-nighters, and were writing weird code like:

while(true) {
if(xxx)
if(yyy)
break;
}

(Somehow, this code was claimed to have "worked")

A little disappointing in the end. Our program was able to run all the simple testcases, but failed spectacularly on the final set of benchmarks (I think it was 0/8). However, I would have to say we got our "revenge" the following year. ;-)

Recently, I cooked up a little Firefox extension. When I decided to make it generally available, I wasn't sure how I was going to sufficiently test it. The only saving grace is that my extension was provided AS-IS, with absolutely no warranties. If there was a bug, I wasn't necessarily going to have to fix it. While I was fairly confident in my code, some weird bugs slipped through. Someone had issues with the extension on Linux (that somehow resolved itself after uninstall/reinstall about 8 times). I don't even have a linux box.. no way I was going to figure this one out. Another case was when all the UK folks reported extension problems. Weird that only UK people had problems, until I realized I had set the en-GB manifest to a non-existent set of locale files. I had tested en-US, fr-FR and zh-CN. I wrongly thought that since en-GB used the same set of files as en-US, it can never be busted (unless I pointed to non-existent en-GB files.) :)

Now that I'm working on a much larger project, with dedicated test teams, I have really learned how important it is to test your own code before your release it. You really don't want to waste the testers' time with simple problems that you should have caught. Usually, by the time they flag the bug and go through all the process of documenting how to reproduce the failure, the bug has already been fixed.

On these large projects, the most amazing thing is when the testers stress test your program with these super duper test cycles that include runs for hundreds of hours. It's always an anxious time waiting for the results from these runs. If our program survives the testing,everyone's off partying. If it doesn't.... (/cry and cancel those weekend plans).