On writing test-cases and testsuites.
Last night I mostly patched my local copy of less to build and link against the PCRE regular expression library.
I've wanted to do that for a while, and reading Raymond Chen's blog post last night made me try it out.
The patch was small and pretty neat, and I'm familiar with GNU less having patched it in the past. But it doesn't contain tests.
Test cases are hard. Many programs, such as less, are used interactively which makes writing a scaffold hard. Other programs suffer from a similar fate - I'm not sure how you'd even test a web browser such as Firefox these days - mangleme would catch some things, eventually, but the interactive stuff? No clue.
In the past MySQL had a free set of test cases, but my memory is that Oracle locked them up. SQLite is famous for its decent test coverage. But off the top of my head I can't think of other things.
As a topical example there don't seem to be decent test-cases for either bash or openssl. If it compiles it works, more or less.
I did start writing some HTTP-server test cases a while back, but that was just to automate security attacks. e.g. Firing requests like:
GET /../../../etc/passwd HTTP/1.0 GET //....//....//....//etc/passwd HTTP/1.0 etc
(It's amazing how many toy HTTP server components included in projects and products don't have decent HTTP-servers.)
I could imagine that being vaguely useful, especially because it is testing the protocol-handling rather than a project-specific codebase.
Anyway, I'm thinking writing test cases for things is good, but struggling to think of a decent place to start. The project has to be:
- Non-interactive.
- Open source.
- Widely used - to make it a useful contribution.
- Not written in some fancy language.
- Open to receiving submissions.
Comments welcome; but better yet why not think about the test-coverage of any of your own packages and projects...?