Sometimes, when you need some extra confidence in your algorithm, it doesn’t hurt to employ random data and statistics. You may be able to take your testing a step further by throwing literally every input—or at least a random sample of every input—at your program in an attempt to raise an error or unexpected success.

I used this technique today, with a low-level program that accepts structured binary input (including variable length fields, nested records, and some other non-trivial parsing requirements). I had already tested as many execution paths and pathological corner cases as I could design tests for, but I wanted some extra assurance that my program would not segfault or produce unexpected results, no matter what a user might throw at it.

Representative Sample?

So, naturally, I threw /dev/random at it.

In a few minutes, I wrote a script to run my program against about one thousand completely random data sets, which, barring a miracle, should have been complete gibberish to my parser. I was looking for program crashes, but thankfully my parser handled all of the random inputs gracefully.

I did get an unexpected result, however: Within the first few hundred iterations, my parser had actually picked out small sections of valid data. Over the thousand-or-so runs, it pulled out six valid values. This looked fishy to me, so I did the math. If my parser was working correctly, the probability of a match was about 1:18,000,000. So, six hits out of a thousand was a real red flag. After all, I can never even win a free lottery ticket.

When I inspected one of the random files that contained the supposed valid data, I quickly discovered a subtle bug in my parser that made it far too permissive in some cases. The bug was easy to fix, but would have been very difficult to spot otherwise.

So, don’t forget to throw some complete rubbish at your programs during testing once in a while.

[ Photo credit: Nice balls by RcktManIL, on Flickr ]