Project Success - Measuring it/Facilitating it

Nice collection of information. I ran a similar thread sometime back on Risks of running open source projects where I defined risks as anything that would upset success. You can look up that information too.

Yeah, I have already thanks :) The information in there was helpful. I'm trying to get together a survey at the moment but the closer I look the more complicated it gets.

I would like to respond by dissecting successes and failures from my own free software career.

premail: modest success

My very first free software effort was premail, a client for anonymous remailers and general email encryption. It achieved a reasonable user base, significant developer contributions, and was fun. I haven't maintained it in many years, but it is still used a bit, and its close companion, the remailer list, lives on.

I think premail was a success because it was a good idea and did something useful. It probably failed to achieve greater success because of the fundamental limitations of the concept: port 25 is actually not a good platform for trying to do encrypted messaging. Further, there were serious usability and "configuration science" problems resulting in the integration with email clients. Lastly, the implementation was flawed in many ways: it was started in Perl4 and kept much of the design from that era; it forked off PGP processes for all crypto; and, at ultimately 8k lines, became unwieldy.

gzilla: mostly a failure

My next big free software effort was gzilla, a "small, simple" web browser. I had big plans for it - I was hoping for it to become the web browser for Linux. This obviously didn't happen.

I tried to recruit a larger development team, but aside from a few patches and a bit of work from one other developer, this never really happened. A big part of the problem is that the code contained too much rocket science - it was structured to be very asynchronous, but without making use of threads. This was an interesting style of programming for me, but I think too difficult for others to follow.

The end of my involvement with gzilla was the announcement by Netscape that they were going to open-source their browser. It felt like there was no more need for a free browser. As we all know, they still haven't shipped a 1.0, and it is clear that a niche exists for the original gzilla goal of a small, simple browser.

The gzilla codebase did get adopted as the basis for Armadillo, then Dillo, which is currently a live project. The early days of the handoff were rather unpleasant: I felt that the new developers didn't understand what I was trying to do, and many of their changes were deprovements (the introduction of SIGIO remains a particularly bitter memory). Fortunately, the current team seems to be pretty smart, and the project is reasonably vital.

Another frustration is that people kept implementing small and simple Gtk+ browsers even after gzilla staked out the territory - gtkhtml and, today, Yelp come to mind.

GdkRgb: great success

The free software project I am currently most proud of is GdkRgb. The idea was very good and very simple: provide an API for drawing 24-bit RGB images in windows. At the time, most applications that wanted to draw images did their own implementations in terms of the underlying X concepts: X images, colormaps, and visuals. As a rule, these implementations sucked. GdkRgb, by contrast, does not suck.

GdkRgb started life as a fragment from gzilla (gzwimage, to be precise). While coding gzilla, it became obvious to me that something like GdkRgb was needed. Since then, it has become an integral part of Gtk+. In a beautiful example of dramatic irony, it is now the way Mozilla draws images on X.

GdkRgb meets all the criteria for success - it has a huge user base, it is being actively maintained, it is being ported to lots of other platforms, and it is more or less gradually displacing all the other sucky things people were doing to display images on X. In general, people are not NIH'ing new GdkRgb-like things, either.

It's interesting to note that while the implementation contains some highly nontrivial rocket science (I'm particularly proud of the 16-bit dithering, which I believe is unmatched), the interface is very easy to understand.

Gill: failure

My next big project was Gill, which was an attempt to make a Gtk+ vector editor application. The major idea was to structure the entire program around W3C standards, most especially SVG and DOM. This turned out to be the big mistake - the latter, in particular, is a terrible piece of technology. For some reason, I became very enthusiastic about it, and even tried to evangelize it to the broader Gnome community. Boy, was I wrong!

Big pieces of Gill live on, however. librsvg is a batch (SAX rather than DOM) renderer for an SVG subset, and is used for rendering vector icons and the like in Nautilus. In addition, the ksvg team is borrowing liberally from the Gill codebase, and is also in good communication with me. And, of course, there's...

Libart: modest success

One of the big pieces needed to implement Gill was a modern 2D graphics engine - one that could handle PostScript-style shapes, antialiasing, and alpha transparency. Libart was my implementation of this engine.

Quite a few graphics applications use Libart, but it hasn't exactly taken over the world either - it doesn't have the same central role on the Linux desktop that Quartz has on Mac OS X. Also irritating is the fact that people keep NIH'ing Libart-like things. The latest is the Anti-Grain Geometry library, which at least lists Libart as an influence.

I've gotten some developer contributions on Libart, but have had to do most of the work myself. This is largely due to the fact that some of the internals make use of serious rocket science. Fortunately, the external interface is much easier to understand, and I generally get positive feedback from people trying to use it.

I guess my sense of Libart not meeting its full potential has to do with the fact that it's quite a low level module, and there hasn't (yet) been a module at the next level up that's really excellent. The Gnome Canvas was an attempt at this, but is flawed in many ways. Sometimes, good things simply need time to become successful. Hopefully, this is the case with Libart.

rcss: bitter failure

Rcss was an attempt to build a very efficient CSS engine. Among other things, it was to be used in Gill, but my hope is that it would be adopted by others with the need to process CSS, such as HTML widgets and browsers.

Technically, the ideas in rcss were (and still are) very good. It uses standard computer science concepts such as lazy expansion of a nondeterministic finite automaton into a deterministic one. It also made use of clever ideas such as incrementally keeping track of the CSS state as the client traverses the document structure. If fully implemented, the result would have been extremely efficient in terms of both memory and execution time. Further, it would have had some nice features such as being able to tell you very quickly which properties changed when the style sheet was updated.

By contrast, CSS implementations such as Mozilla's are buggy (implementing the greedy shortcut, when the spec clearly calls for nondeterminism), slow, bloated, and not very featureful. I thought rcss would be a valuable contribution to the world.

Two things happened, though, that killed the project dead. First, as I tracked the proposed changes to CSS for level 3, I found that the W3C was well on their way to producing another bloated, overcomplex standard. A number of the new features (the "contains" selector in particular) would break the model that rcss uses to achieve its efficiency. Further, I couldn't (and still can't) see a compelling justification for these new features. It was clear to me that the W3C didn't have an efficient implementation in mind at all, just sucky ones.

Second, as I communicated with people in the HTML world (including Mozilla), I found a stunning lack of interest. While it certainly is true that CSS efficiency is much more critical in the SVG world than the HTML one, this was disheartening to say the least.

This isn't a complete list. Other important projects include Advogato (modest success; too early to tell long term), Wet Dream (nonsuccess), and of course my current Ghostscript work (success, at least by my criteria), including IJS (which looks well on its way to being a success). I won't try to draw conclusions here either - hopefully these case studies will be grist for the discussion mill.

<facetious value="50%">

Raph: I notice that in general you've described the projects depending on W3C standards as having failed, and the others as successful. Can we propose this as a general rule?

I'm not sure such a long and undescriptive list is useful for anybody. Here's my take of what makes a project successful:

Usually the number of people who actually contribute to the project is very small. Therefore a project must depend on having at least one person who is interested in it enough to contribute. A good start for a project is for the one organizing to write a small codebase himself and release it as is.

I believe projects that start this way tend to end up better than those projects that someone opens a project for it in SourceForge and enough people would ne interested.

My own Freecell Solver project started this way. I took some time on a vacation to write a version of it in C, and when I had a stable codebase ("Wow, it actually works") I released version 0.2 of it and announced it on Freshmeat. FCS 0.2's design and implementation was seriously broken in all so many ways, a thing which improved in later versions, and I did %99 of the work.

As another example: syscalltrack went to a slow start when it was first announced. Only when <person>mulix</person joined in and Guy Keren decided to invest more effort in it, did it start to become actively developed. I'm not sure Guy, Muli and the other guys that develop it can every hope to see another active participant in the project (I myself am just a lurker in its mailing list). So can't I for Freecell Solver.

This is just one factor that makes a project successful. But I believe a list of such factors with explanations will be more instructive than what was presented here.

Refer here for a very nice and well-written document by Paul Graham on what makes a computer language successful. I think one should do something like that when trying to enumerate the factors that make a project successful.

Thanks for the reply shlomif.

I am actually working on it from both ends (data gathering and theory building) and I think that is the best way to proceed. I provided the information knowing full well that it is raw and cumbersome. I think it is best to give out what I have now, even if you find it of little use, rather than wait until I have moved to the next step. Some people will take it, others can leave it. I am happy that some people relate (as you have) how this raw information applies or does not apply to their experience.

I think it is valuable to gather raw information from people that know more than me about being involved in free software projects and what makes them tick. In this way the information is grounded in what people know, not what theory does or does not tell me. I agree that raw information is not knowledge. I have engaged in more in-depth discussion with individual experts about how these pieces work in a project (ie attempted to create knowledge out of information). These email discussions were very useful. The real intent of posting the information here, in raw form, rather than presenting preformed conclusions about 'the way things work', was to achieve some similar discussion.

As I mentioned previously, I am now relating what I have been told back to theoretical constructs and attempting to get some sort of match so I can go out a survey projects. This requires description as you point out and with description comes an understanding of how these pieces of information fit together and interact. For example, answering questions as to why mulix and Guy Keren decide to invest more time in syscalltrack. Is there some theory about individuals or about how the project was managed that we can point at to explain or not explain the success of syscalltrack. Was it because your project was based on a useful piece of software? Did it fill a niche? How did you market it and go about developing it? Are you a charismatic leader? etc...

This is part of theory building and sensemaking. Not that I hope to reach the lofty heights of creating a theory. The aim is for my PhD thesis to be a "..very nice and well-written document" that moves a small step closer to understanding free software. You can read my messy and inaccurate draft documents for the little they are worth on my webpage :)

Just to let you know my own take on this, when I started working on NASM I knew pretty much from the beginning that it would be a success, as long as we kept it 'right'.

I knew this because it was easy to see that there were many people who would use it. At the time I regularly read the comp.lang.asm.x86 newsgroup, and a thread about what people would want in a new assembler was the most popular thread on the group for months. The group of us that set out to write NASM were the result of that thread. Of course NASM doesn't have all of the features that came out of that discussion, but it got pretty close fairly quickly.

At one point, things were looking a little pessimistic, though. The number of active developers dwindled from about 10 to only 2 before even the first releases were made. But we hung on. We knew that we had to make it useful for general users before it would take off, and as soon as we did - with the release of the 0.3 version - the user base exploded and we started to receive patches every so often and bug reports more often :-)

Aside: I think the number of (resolved) bug reports is probably a good indicator of success...!

I think the greatest factor in NASMs success was the demand for it in the first place. It was the first free product of its kind, its only real competitors were Eric Isaacson's shareware A86 and commercial products like MASM and TASM, and all of those were DOS only products...

And that bears out my other experiences - my other projects have been in areas with little demand, mainly because either they perform unusual tasks (an emulation of the precise semantics of DOS's 'REN' command under Linux) or because better software was already out there when I started (a web site mirroring system, an operating system kernel).

To add to shlomif's point, I think it'll be nice if the article can look at the open source scene from a different point of view: that of the onlooker who's deciding which project to join, rather than the maintainer/developer. That is, include specific reasons why developers join -- or decide not to join -- specific projects. Saying e.g. "fills a niche" is just too general.

For example, julesh mentioned that NASM succeeded because there was already demand for such an assembler in the first place. Linux probably achieved its current success because it was initially targeted at the Minix community, and a large portion of the community was looking for something better than Minix. And so on.

Such data, even if it's very raw, will be very useful and interesting.

(By the way, I'm an NASM user too. :)

I also think it would be nice, informative and helpful if an article was written to help onlookers decide on which project to join. But, I'm sorry, my PhD isn't directly about that. I am concerned with other issues relating to strategic management and free software development. Due to my PhD's focus on the management of projects, particurlarly, resources and dynamic capabilities of projects, it will necessarily be focussing on those who take on the roles of project leadership. It would be a bonus to be able to include all the contributors in my research as well, but i don't expect that to happen given the logistical problems.

My research might not exclude making findings that will be in some way beneficial for onlookers (potential contributors) who wish to select a project to be involved with. Unfortunately, the focus I have tends to wash away the meaning of my results for onlookers. However, I am trying to identify general characteristics that predict success for different types of free software projects. Whether a certain set or configuration of resources and capabilities is useful for onlookers to be interested in is difficult to answer. Even being able to externally observe a project and identify the configuration of resources and capabilities is also questionable.