19 Dec 2004 pfdietz   » (Master)

James Andrews has recently done an experiment with 'Coveraged-checked Random Unit Testing' (CRUT) (see also here. ) He took two sets of libraries with basic data structures (hash tables, trees, heaps, and so on) from Sourceforge, and wrote harnesses to make random function calls. The result logs were checked using an automated log checking system using an algebraic description of the expected results.

He found 6 major bugs (with no workarounds) in the two libraries and 12 additional bugs (either with workarounds or of lesser severity.) The random test generators achieved better than 90% statement coverage (as measured by gcov), which is even more impressive when one realizes that most of the uncovered lines were dead or untestable.

I'm not terribly surprised by these results -- it's easier to achieve high coverage with unit tests, and there are fewer low probability hoops for the random inputs to jump through -- but I'm troubled that free software packages labeled 'production/stable' had so many serious defects.

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!