Project Success - Measuring it/Facilitating it

Posted 1 Mar 2002 at 00:50 UTC by renster Share This

I've just been through a series of discussions with various people about how to predict success of Free Software projects and what factors predict success. I emailed a list of experienced Free Software/Open Source developers/leaders/gurus and compiled their responses. Yes, it's raw - very raw. This also appears on my site.

I have pasted below the email I sent back to the original pool of people I questioned. I simply asked them to list 3-5 measures of project success and 3-5 factors that predict success. This resulted in a mixture of direct responses, links to other resources and a series of emails discussing particular points. I selected people I knew to have some experience in Free Software/Open Source software development.

I am still looking for feedback. If you would like to comment on the list below or on any other sections (particularly section 3 :)) please feel free to do so. Any elaboration on items listed below would be very handy.


The responses are in no particular order - they are in fact in a fair bit of disorder. I attempted to keep the comments largely as they were by extracting the concept and listing them. I didn't want to begin condensing the list at this stage - that will come later (and hopefully higher order constructs will magically fall out).

As expected there was a lot of overlap between measures of success and factors that contribute to success. I'm not too phased by looks like I can create a decent argument for a list of measures of project success. The major concern in presenting a coherent argument in my research will be issues surrounding causality. Therefore some items will be moved from success factor to indicator or vice versa.

I will now go away and attempt to create some measures to be included in a survey to be sent to project leaders.




Project Success

  1. Growth - of developer/contributor base
  2. Growth - of user base (non-developers)
  3. Number of contributors
  4. Developer activity - and is it visible?
  5. Openness of project - allows others into the development process
  6. Openness of project - communicates/indicates progress
  7. Quality of documentation - web site, changelogs, manuals
  8. Number of discussion/mailing list posts
  9. Number of websites discussing/linking
  10. Number of downloads
  11. Number of page views
  12. Number of other projects basing their project on it
  13. Number of commercial entities basing their business on it
  14. Adoption and support/investment from commercial entities - ranges from lip service to actual direct support
  15. Rate of stable release
  16. Number of bugs reported/resolved - robust under unexpected uses - stability (# of bugs reported dependent on project popularity)
  17. Debian popularity-contest package list

Personal Success/Outcomes

  1. Learning - extended programming skills
  2. Enjoyment/fun - increased
  3. Employment outcomes
  4. Reputation outcomes


  1. Active marketing/promotion of project -an evangilist is helpful?
  2. Nice Website
  3. Willingness of founders to grant commit access to others
  4. Acceptance of ideas/viewpoints of others - and incorporate if appropriate
  5. Architecture of the project prevents coordination problems - embedded scripting language, plug-in architecture
  6. Leadership - technically proficient (a good programmer?)
  7. Leadership - personable
  8. Usefulness of software - fills a niche or is better than current
  9. Competent technical core of developers
  10. A sense of community
  11. Stability
  12. Documentation and support
  13. Sex appeal - allows creativity and growth, excitement about the software and end product, some attraction.
  14. Ownership by developer community
  15. Commmunication among developers
  16. Communicaton to user community
  17. Open Development Cycle
  18. Low Barrier to entry for developers - easy to get in
  19. Redundant Developer roles
  20. Clear dispute resolution mechanism - not a management hierachy
  21. No Flat organization
  22. User base size



The following is from "The Boston Consulting Group/OSDN Hacker Survey" ( You can see some overlap in responses with the list above. I've added this to present a more complete list of factors/items.


  1. Intellectually stimulating
  2. Work functionality
  3. Obligation from use
  4. Work with team
  5. Professional status
  6. Other
  7. Open Source reputation
  8. Beat proprietary software
  9. Code should be open
  10. Non-work functionality
  11. Improves skill
  12. License forces me to


  1. Create initial code base
  2. Integrate submissions
  3. Determine tasks
  4. Help people get started
  5. Open minds to options
  6. Provide motivation
  7. Delegate tasks
  8. Provide specific help
  9. Create promise/vision
  10. Initiate dialogue
  11. Contribute code
  12. Recruit contributors
  13. Other
  14. Manage timing



In order to capture some of the time related information I have begun to organise your responses across stages in project life cycles (that I made up off the top of my head). Some of the comments received clearly related to the beginning stages of projects whereas others were directed more towards established projects.

This may or may not be useful to later match strategy types to particular project stages.


  1. An initial code base
  2. A useful piece of software - fills a niche or improves on existing
  3. Founder - technically proficient

Catalyst to next stage

  1. Founder - free software principles, recruits/markets, makes project open, personable, initiate dialogue, motivates
  2. Sex appeal
  3. Low barriers to entry


  1. Marketing/promotion
  2. Competent technical core of developers
  3. Architecture of the project organised to reduce coordination/development issues
  4. Open development model
  5. Extensible software
  6. A good leadership team - coordinate, delegate, openness, integrate, motivate, removes bottle necks.
  7. Leadership team that can provide focus and prioritise - strategic direction - direction of the developmental process (e.g.
  8. Communication among developers
  9. Mentoring and training developers/contributors

Catalyst to next stage

  1. Critical mass issues - developer and user base of certain size X?
  2. Stability and security
  3. Mechanism for dispute resolution
  4. Increased reputation X?
  5. Movement away from flat organization
  6. Attracts external resource support - financial, capital
  7. Support mechanism to buffer user negativity


  1. Community leader
  2. Sense of ownership across project contributors
  3. Adoption by other projects/commercial entities
  4. Formal/clear documentation and support
  5. Stable core team that doesn't break if others leave/take a break


  1. Failure of succession - founder bows out, no project hand over
  2. Failure to grow - no sex appeal, software not useful, or initial code base not useable
  3. External factors - forking.

Project Success - Antithesis of the Risk of Failure, posted 1 Mar 2002 at 05:15 UTC by vivekv » (Journeyer)

Nice collection of information. I ran a similar thread sometime back on Risks of running open source projects where I defined risks as anything that would upset success. You can look up that information too.

Re: Project Success - Antithesis of the Risk of Failure, posted 1 Mar 2002 at 05:39 UTC by renster » (Journeyer)

Yeah, I have already thanks :) The information in there was helpful. I'm trying to get together a survey at the moment but the closer I look the more complicated it gets.

My case studies, posted 1 Mar 2002 at 09:27 UTC by raph » (Master)

I would like to respond by dissecting successes and failures from my own free software career.

premail: modest success

My very first free software effort was premail, a client for anonymous remailers and general email encryption. It achieved a reasonable user base, significant developer contributions, and was fun. I haven't maintained it in many years, but it is still used a bit, and its close companion, the remailer list, lives on.

I think premail was a success because it was a good idea and did something useful. It probably failed to achieve greater success because of the fundamental limitations of the concept: port 25 is actually not a good platform for trying to do encrypted messaging. Further, there were serious usability and "configuration science" problems resulting in the integration with email clients. Lastly, the implementation was flawed in many ways: it was started in Perl4 and kept much of the design from that era; it forked off PGP processes for all crypto; and, at ultimately 8k lines, became unwieldy.

gzilla: mostly a failure

My next big free software effort was gzilla, a "small, simple" web browser. I had big plans for it - I was hoping for it to become the web browser for Linux. This obviously didn't happen.

I tried to recruit a larger development team, but aside from a few patches and a bit of work from one other developer, this never really happened. A big part of the problem is that the code contained too much rocket science - it was structured to be very asynchronous, but without making use of threads. This was an interesting style of programming for me, but I think too difficult for others to follow.

The end of my involvement with gzilla was the announcement by Netscape that they were going to open-source their browser. It felt like there was no more need for a free browser. As we all know, they still haven't shipped a 1.0, and it is clear that a niche exists for the original gzilla goal of a small, simple browser.

The gzilla codebase did get adopted as the basis for Armadillo, then Dillo, which is currently a live project. The early days of the handoff were rather unpleasant: I felt that the new developers didn't understand what I was trying to do, and many of their changes were deprovements (the introduction of SIGIO remains a particularly bitter memory). Fortunately, the current team seems to be pretty smart, and the project is reasonably vital.

Another frustration is that people kept implementing small and simple Gtk+ browsers even after gzilla staked out the territory - gtkhtml and, today, Yelp come to mind.

GdkRgb: great success

The free software project I am currently most proud of is GdkRgb. The idea was very good and very simple: provide an API for drawing 24-bit RGB images in windows. At the time, most applications that wanted to draw images did their own implementations in terms of the underlying X concepts: X images, colormaps, and visuals. As a rule, these implementations sucked. GdkRgb, by contrast, does not suck.

GdkRgb started life as a fragment from gzilla (gzwimage, to be precise). While coding gzilla, it became obvious to me that something like GdkRgb was needed. Since then, it has become an integral part of Gtk+. In a beautiful example of dramatic irony, it is now the way Mozilla draws images on X.

GdkRgb meets all the criteria for success - it has a huge user base, it is being actively maintained, it is being ported to lots of other platforms, and it is more or less gradually displacing all the other sucky things people were doing to display images on X. In general, people are not NIH'ing new GdkRgb-like things, either.

It's interesting to note that while the implementation contains some highly nontrivial rocket science (I'm particularly proud of the 16-bit dithering, which I believe is unmatched), the interface is very easy to understand.

Gill: failure

My next big project was Gill, which was an attempt to make a Gtk+ vector editor application. The major idea was to structure the entire program around W3C standards, most especially SVG and DOM. This turned out to be the big mistake - the latter, in particular, is a terrible piece of technology. For some reason, I became very enthusiastic about it, and even tried to evangelize it to the broader Gnome community. Boy, was I wrong!

Big pieces of Gill live on, however. librsvg is a batch (SAX rather than DOM) renderer for an SVG subset, and is used for rendering vector icons and the like in Nautilus. In addition, the ksvg team is borrowing liberally from the Gill codebase, and is also in good communication with me. And, of course, there's...

Libart: modest success

One of the big pieces needed to implement Gill was a modern 2D graphics engine - one that could handle PostScript-style shapes, antialiasing, and alpha transparency. Libart was my implementation of this engine.

Quite a few graphics applications use Libart, but it hasn't exactly taken over the world either - it doesn't have the same central role on the Linux desktop that Quartz has on Mac OS X. Also irritating is the fact that people keep NIH'ing Libart-like things. The latest is the Anti-Grain Geometry library, which at least lists Libart as an influence.

I've gotten some developer contributions on Libart, but have had to do most of the work myself. This is largely due to the fact that some of the internals make use of serious rocket science. Fortunately, the external interface is much easier to understand, and I generally get positive feedback from people trying to use it.

I guess my sense of Libart not meeting its full potential has to do with the fact that it's quite a low level module, and there hasn't (yet) been a module at the next level up that's really excellent. The Gnome Canvas was an attempt at this, but is flawed in many ways. Sometimes, good things simply need time to become successful. Hopefully, this is the case with Libart.

rcss: bitter failure

Rcss was an attempt to build a very efficient CSS engine. Among other things, it was to be used in Gill, but my hope is that it would be adopted by others with the need to process CSS, such as HTML widgets and browsers.

Technically, the ideas in rcss were (and still are) very good. It uses standard computer science concepts such as lazy expansion of a nondeterministic finite automaton into a deterministic one. It also made use of clever ideas such as incrementally keeping track of the CSS state as the client traverses the document structure. If fully implemented, the result would have been extremely efficient in terms of both memory and execution time. Further, it would have had some nice features such as being able to tell you very quickly which properties changed when the style sheet was updated.

By contrast, CSS implementations such as Mozilla's are buggy (implementing the greedy shortcut, when the spec clearly calls for nondeterminism), slow, bloated, and not very featureful. I thought rcss would be a valuable contribution to the world.

Two things happened, though, that killed the project dead. First, as I tracked the proposed changes to CSS for level 3, I found that the W3C was well on their way to producing another bloated, overcomplex standard. A number of the new features (the "contains" selector in particular) would break the model that rcss uses to achieve its efficiency. Further, I couldn't (and still can't) see a compelling justification for these new features. It was clear to me that the W3C didn't have an efficient implementation in mind at all, just sucky ones.

Second, as I communicated with people in the HTML world (including Mozilla), I found a stunning lack of interest. While it certainly is true that CSS efficiency is much more critical in the SVG world than the HTML one, this was disheartening to say the least.

This isn't a complete list. Other important projects include Advogato (modest success; too early to tell long term), Wet Dream (nonsuccess), and of course my current Ghostscript work (success, at least by my criteria), including IJS (which looks well on its way to being a success). I won't try to draw conclusions here either - hopefully these case studies will be grist for the discussion mill.

Avoidance of W3C standards a project success indicator?, posted 1 Mar 2002 at 15:27 UTC by dan » (Master)

<facetious value="50%">

Raph: I notice that in general you've described the projects depending on W3C standards as having failed, and the others as successful. Can we propose this as a general rule?

My opinion on the article, posted 4 Mar 2002 at 14:18 UTC by shlomif » (Master)

I'm not sure such a long and undescriptive list is useful for anybody. Here's my take of what makes a project successful:

Usually the number of people who actually contribute to the project is very small. Therefore a project must depend on having at least one person who is interested in it enough to contribute. A good start for a project is for the one organizing to write a small codebase himself and release it as is.

I believe projects that start this way tend to end up better than those projects that someone opens a project for it in SourceForge and enough people would ne interested.

My own Freecell Solver project started this way. I took some time on a vacation to write a version of it in C, and when I had a stable codebase ("Wow, it actually works") I released version 0.2 of it and announced it on Freshmeat. FCS 0.2's design and implementation was seriously broken in all so many ways, a thing which improved in later versions, and I did %99 of the work.

As another example: syscalltrack went to a slow start when it was first announced. Only when <person>mulix</person joined in and Guy Keren decided to invest more effort in it, did it start to become actively developed. I'm not sure Guy, Muli and the other guys that develop it can every hope to see another active participant in the project (I myself am just a lurker in its mailing list). So can't I for Freecell Solver.

This is just one factor that makes a project successful. But I believe a list of such factors with explanations will be more instructive than what was presented here.

Refer here for a very nice and well-written document by Paul Graham on what makes a computer language successful. I think one should do something like that when trying to enumerate the factors that make a project successful.

re: My opinion on the article, posted 5 Mar 2002 at 04:46 UTC by renster » (Journeyer)

Thanks for the reply shlomif.

I am actually working on it from both ends (data gathering and theory building) and I think that is the best way to proceed. I provided the information knowing full well that it is raw and cumbersome. I think it is best to give out what I have now, even if you find it of little use, rather than wait until I have moved to the next step. Some people will take it, others can leave it. I am happy that some people relate (as you have) how this raw information applies or does not apply to their experience.

I think it is valuable to gather raw information from people that know more than me about being involved in free software projects and what makes them tick. In this way the information is grounded in what people know, not what theory does or does not tell me. I agree that raw information is not knowledge. I have engaged in more in-depth discussion with individual experts about how these pieces work in a project (ie attempted to create knowledge out of information). These email discussions were very useful. The real intent of posting the information here, in raw form, rather than presenting preformed conclusions about 'the way things work', was to achieve some similar discussion.

As I mentioned previously, I am now relating what I have been told back to theoretical constructs and attempting to get some sort of match so I can go out a survey projects. This requires description as you point out and with description comes an understanding of how these pieces of information fit together and interact. For example, answering questions as to why mulix and Guy Keren decide to invest more time in syscalltrack. Is there some theory about individuals or about how the project was managed that we can point at to explain or not explain the success of syscalltrack. Was it because your project was based on a useful piece of software? Did it fill a niche? How did you market it and go about developing it? Are you a charismatic leader? etc...

This is part of theory building and sensemaking. Not that I hope to reach the lofty heights of creating a theory. The aim is for my PhD thesis to be a "..very nice and well-written document" that moves a small step closer to understanding free software. You can read my messy and inaccurate draft documents for the little they are worth on my webpage :)

My experiences, posted 5 Mar 2002 at 16:39 UTC by julesh » (Master)

Just to let you know my own take on this, when I started working on NASM I knew pretty much from the beginning that it would be a success, as long as we kept it 'right'.

I knew this because it was easy to see that there were many people who would use it. At the time I regularly read the comp.lang.asm.x86 newsgroup, and a thread about what people would want in a new assembler was the most popular thread on the group for months. The group of us that set out to write NASM were the result of that thread. Of course NASM doesn't have all of the features that came out of that discussion, but it got pretty close fairly quickly.

At one point, things were looking a little pessimistic, though. The number of active developers dwindled from about 10 to only 2 before even the first releases were made. But we hung on. We knew that we had to make it useful for general users before it would take off, and as soon as we did - with the release of the 0.3 version - the user base exploded and we started to receive patches every so often and bug reports more often :-)

Aside: I think the number of (resolved) bug reports is probably a good indicator of success...!

I think the greatest factor in NASMs success was the demand for it in the first place. It was the first free product of its kind, its only real competitors were Eric Isaacson's shareware A86 and commercial products like MASM and TASM, and all of those were DOS only products...

And that bears out my other experiences - my other projects have been in areas with little demand, mainly because either they perform unusual tasks (an emulation of the precise semantics of DOS's 'REN' command under Linux) or because better software was already out there when I started (a web site mirroring system, an operating system kernel).

Include concrete reasons people join certain projects, posted 6 Mar 2002 at 00:42 UTC by tk » (Observer)

To add to shlomif's point, I think it'll be nice if the article can look at the open source scene from a different point of view: that of the onlooker who's deciding which project to join, rather than the maintainer/developer. That is, include specific reasons why developers join -- or decide not to join -- specific projects. Saying e.g. "fills a niche" is just too general.

For example, julesh mentioned that NASM succeeded because there was already demand for such an assembler in the first place. Linux probably achieved its current success because it was initially targeted at the Minix community, and a large portion of the community was looking for something better than Minix. And so on.

Such data, even if it's very raw, will be very useful and interesting.

(By the way, I'm an NASM user too. :)

re: Include concrete reasons people join certain projects, posted 6 Mar 2002 at 03:07 UTC by renster » (Journeyer)

I also think it would be nice, informative and helpful if an article was written to help onlookers decide on which project to join. But, I'm sorry, my PhD isn't directly about that. I am concerned with other issues relating to strategic management and free software development. Due to my PhD's focus on the management of projects, particurlarly, resources and dynamic capabilities of projects, it will necessarily be focussing on those who take on the roles of project leadership. It would be a bonus to be able to include all the contributors in my research as well, but i don't expect that to happen given the logistical problems.

My research might not exclude making findings that will be in some way beneficial for onlookers (potential contributors) who wish to select a project to be involved with. Unfortunately, the focus I have tends to wash away the meaning of my results for onlookers. However, I am trying to identify general characteristics that predict success for different types of free software projects. Whether a certain set or configuration of resources and capabilities is useful for onlookers to be interested in is difficult to answer. Even being able to externally observe a project and identify the configuration of resources and capabilities is also questionable.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page