Well I just logged in to advogato and being of an academic bent am wondering if this distributed certification process would be a better way to make help tenure appointements than the current black art. :)
Specifically, it is possible to write a valuable, widely used piece of software inducing a large number of downloads and not make any academically significant contribution. Developers at IBM, Microsoft, Sun, etc. do this every day. In such cases the software is literally new, as might be expected in a synthetic field, but it has been created within the known state-of-the-art. It is not "better" by embodying new ideas or techniques, as Brooks requires. It may be improved, but anyone "schooled in the art" would achieve similar results.
Quantitative data may not imply all that is claimed for it, and it can be manipulated. Downloads do not imply that the software is actually being used, nor do Web hits imply interest. There are techniques, such as the Googol page-rank approach [http://www.google.com], that may produce objective information about Web usage, for example, but caution in using numbers is always advised.
So should being highly ranked on adovgato "count" during tenure review? If it did, you could get lots of tenure-track faculty to probably write more useful free software. The current "economic" incentives in a academic environment make writting useful cutting edge software difficult.