Smart monkey testing

In daily practice software engineers focus on validating the code’s correct operation: the so-called ‘happy path’. Happy-path testing does a good job at achieving high line coverage, that is, the number of source code lines exercised by the test. However, many long code paths through multiple source files and libraries normally triggered by customers remain uncovered. And how about testing the code that checks input data validity? Even experienced test engineers often underestimate the importance of thoroughly testing error handling and error recovery code.

Various testing tools can create new test cases for your software. Tools based on formal methods, for example, can prove that certain invariants do not hold for your code and can provide a test case that triggers the assertion. However, they don’t scale to handling millions of lines of source code. Other tools merely examine a function’s input and output and generate a trivial test code template that misses its actual functionality. It saves typing, but not thinking.