You've worked with a team for a while, everything is going good, but you are at a point where everyone disagrees.
How to fork a project the right way, and pitfalls to avoid..
A bit of advice from a successful forker.
I'm the lead developer of an opensource web-based game called Nova Gaming System - The Kabal Invasion.
I didn't start from scratch however.. our project is a fork from the BlackNova Traders game.
The way we ended up here is fairly typical of other forks. I joined the development of BNT, had a different vision, and had a personality clash with a developer on the team.
That single paragraph took over three months to accomplish, so was not undertaken lightly. We tried numerous times to keep me in the fold, and to compromise our visions. In the end, we just had different goals.
Once I decided to fork, I made a few key decisions:
- I would continue to make clear my deep respect for the coders that contributed to the code so far - I would do my best to maintain a close friendship with the code team - I would make it clear to players that my fork was unique and distinct from the original - it should be, if there is sufficient reason to do a fork.
I succeeded in two.
In breaking from the pack, I freed myself of alot of administrative/consensus issues. I was able to make a slew of architectural changes that I felt were absolutely critical to the project.
I gained two developers right away, who felt the same frustration with the development of the game having little direction. We coded frantically, designing and developing a vision, and a codebase to get us to that vision.
We fulfilled my first goal - we went out of our way to ensure that we gave plenty of credit and respect to the BNT team. We still do.
We fulfilled the second goal - we remained (still do) on the BNT dev mailing list, and we were able to share ideas, concepts, and concerns. It has definitely made both team's code stronger.
However, the elusive third goal, distinguishing ourselves, proved to be challenging.
The installed base of BNT is considerable. Five seperate slashdot stories, numerous ad campaigns, and three years of publicity has made them very popular.
The further we deviated from that installed base, the more 'unique' users we got that were very dedicated to the vision of NGS. That's a good thing. However, the further we deviated, the smaller the portion of the existing userbase - the very players we enjoyed playing with and coding for - was interested in our project.
Things have changed lately. We've chosen to take on a backwards fork, of sorts. We are attempting to use the now much more stable and clean NGS codebase to create a feature-for-feature BNT compatible version.
We are not giving up on our vision - far from it. We have wanted all along to have NGS be a platform on which multiple game types can be built. However, its drawing our very limited resources away from a truly unique and faithful crowd at a time when our code has in turn been forked!
My advice to developers considering forking:
Consider the ramifications. Sit down and think about which user community you want to serve, and be consistent in your goal to serve them. We took the misstep of wanting to break away from the pack first, when we should have focused on getting a stable platform that could replace BNT - feature for feature - and then work to offer that AND customizable/configurable differentiation.
Think through how you could 'return to the fold' in time. No matter the level of differences, and they may seem insurmountable, a year or two can move mountains.
Most importantly of all - be honest. Be upfront, be humble, and be realistic.
Our project is at a crossroads.. we may return to the fold a full year after we left. We have competition that is drawing some of the very unique userbase we established, and we have the challenge of reverting our code in numerous ways to gain back the BNT-faithful.
With over 2,000 changes, and 10,000 lines of code changed in a single year, I can't help but consider our fork successful. Its played by over 2,000 players on a regular basis, and it has been translated in 4 languages.
But all of that is nothing without the users. Fork carefully.
This was an interesting article, but I found that most of the insights here don't specifically apply to forking. For example,
Most importantly of all - be honest. Be upfront, be humble, and be realistic.
Maybe the right thing to do is to consider every open source project a "fork" of some previous effort. After all, you can't write a (halfway-decent) webserver these days without learning from the Apache project, even if your goals and codebase are completely unrelated to theirs. You can't do much with a browser without looking at Mozilla. The rules that iamsure proposes sound good for any project:
In fact, forkers have an advantage over people starting new projects, because they already have the experience of working with a functional codebase, and a developer- (and hopefully user-) community. Developers of new projects often lack this insight and take longer to get off the ground because of it.
I know I did :-). I will try to put these guidelines more strongly into practice for myself in the future.
if any group of individuals feel the need to fork, then it has more to do with a failure of the way the project is managed than anything else.
consider that it's possible with cvs to do "double-branching", as outlined in an earlier advogato post on double-checkouts under two separate tags in the same directory (files from one tag mixed with files from another).
consider now that we have subversion, which provides "double-branching" even on directories, not just files.
consider also that a web site has the ability to be edited such that it may provide links to two closely related projects just as easily as it can provide one, or is there something about the web server that project developers are using such that it can only manage to present one project?
why, when there is no technical restriction on what may be done, why the bloody hell should you have to fork?
basically, if you have to fork, it means that the people you are forking from are incapable of being flexible, and that both you and they have failed to negotiate a perfectly reasonable sharing of space.
in other words, someone has all the power, and doesn't want to give any of it away.
in other words, you should look at putting in place a "development charter" similar to the Apache Software Foundation one, which mandates a number of things - not least that developers must respect each other and also that developments may only be rejected on technical grounds and no other.
if it's a REALLY large project, of over several million lines of code consisting of dozens of separate protocols and programs, you may wish to modify that to "technical or strategic grounds".
if any group of individuals feel the need to fork, then it has more to do with a failure of the way the project is managed than anything else.
consider that it's possible with cvs to do "double-branching", as outlined in an earlier advogato post on double-checkouts under two separate tags in the same directory (files from one tag mixed with files from another).
consider now that we have subversion, which provides "double-branching" even on directories, not just files.
consider also that a web site has the ability to be edited such that it may provide links to two closely related projects just as easily as it can provide one, or is there something about the web server that project developers are using such that it can only manage to present one project?
why, when there is no technical restriction on what may be done, why the bloody hell should you have to fork?
basically, if you have to fork, it means that the people you are forking from are incapable of being flexible, and that both you and they have failed to negotiate a perfectly reasonable sharing of space.
in other words, someone has all the power, and doesn't want to give any of it away.
in other words, you should look at putting in place a "development charter" similar to the Apache Software Foundation one, which mandates a number of things - not least that developers must respect each other and also that developments may only be rejected on technical grounds and no other.
if it's a REALLY large project, of over several million lines of code consisting of dozens of separate protocols and programs, you may wish to modify that to "technical or strategic grounds".
lkcl wrote "why, when there is no technical restriction on what may be done, why the bloody hell should you have to fork?"
In my experience, not all changes can easily coexist with the old codebase, even with advanced source code control.
I forked SBCL from CMU CL in order to make the build process start from scratch each time (as conventional software does) instead of incrementally modifying a running version of itself (as Lisp systems and, I think, Smalltalk systems sometimes do). This required a lot of rearrangement of the system so that things are defined in the order in which they're physically needed, and a lot of new definitions to clarify what's being done when. Given that I'm not a sufficiently good diplomat or technical advocate to convince the CMU CL maintainers that building from scratch is worth supporting, it's not reasonable to expect these changes to live in their project. From the point of view of a system which continues to build as the original CMU CL does, the new ordering constraints and new distinctions (e.g. between host compiler and cross compiler) required to support SBCL would be a major distraction and maintenance headache with no benefits. Conversely, if SBCL developers had to do their maintenance in the CMU CL codebase, trying to construct appropriate "bootstrap files" and other cleverness needed to do surgery on a running CMU CL without killing the patient would be pointless distractions, since we can and do avoid the issue by routinely killing the last patient before we start on the new one.
I don't know much about other real forks, but at least theoretically I'd expect the same sorts of issues could arise in other cases where one group of developers wanted to add some sort of globally intrusive design principle: e.g. thread safety or security-motivated constraints. If the original maintainers can't be convinced that the features were sufficient motivation for constraints on the code, then they're not going to be happy trying to respect the constraints involved; but unless they respect the constraints involved, they're not going to be able to smoothly share code with the fork. Or, another change which would involve global changes would be moving to an upwardly compatible implementation language, where the code sharing couldn't be maintained without severe self-imposed coding constraints by the people on the "let's move to the new language" side of the disagreement.
So I think when lkcl proposes to solve the problem by agreement that "developments may only be rejected on technical grounds and no other" he blurs over the way that sometimes global design changes can, in effect, cause subsequent technical considerations to differ between forks. Maintenance which is natural and important for the SBCL branch, the thread-unsafe branch, or the secure branch may be pointless or unreasonably disruptive or both for the CMU CL branch, the single-thread branch, or the all-users-are-trusted branch. (and vice versa)
lkcl wrote "why, when there is no technical restriction on what may be done, why the bloody hell should you have to fork?"
(ie, lets discuss the technical aspects of forking)
There is to a degree.
I reorganized a good third of the files in the game, and totally redid how the majority of game functions worked. While it would be possible to contain both in the same CVS tree, I could also keep an entirely different project under the same CVS tree. It doesnt make it an intelligent choice.
At some point a fork becomes considerably different from its parent, and is difficult to consider in relation to the original.
There is a roughly 15,000 line diff from us to our parent, out of 33,000 lines of code. There isnt a whole lot thats the same anymore.
So at what point - from a technology point of view - should we recognize the differences as fundamental, and embrace two different cvs trees?
It sounds like lkcl doesnt at all, and its just unrealistic. It makes documentation difficult, coding difficult, and a host of other issues difficult. Is it possible, sure, but at what cost? When you have a shortage of developer resources, why sacrifice them towards maintaining a relationship that has proven to be fruitless?
lkcl also wrote "in other words, you should look at putting in place a 'development charter' similar to the Apache Software Foundation one, which mandates a number of things - not least that developers must respect each other and also that developments may only be rejected on technical grounds and no other."
Sure, but that requires a consensus as well. The problem we had in the BNT team was that consensus was never reached. There was no definitive list of who was an active dev and who wasnt.. some devs would say/do nothing for six months at a time, and yet would pop up and want their voice counted during a discussion.
Further, that ignores the possibility of two equally correct solutions, that are incompatible. It happens often in the game, and I am forced to choose...
The lack of structure and responsiveness - and especially consensus - was a major driver for the fork.
In addition, lkcl makes the statement "in other words, someone has all the power, and doesn't want to give any of it away."
But in this case NO ONE had all the power, and thus, no one could give it away, OR use it to help keep things moving.
For technical and personality reasons, it was more effective and more productive - by a huge degree - than to stay 'in the house' so to speak.
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
del.icio.us
Digg
Google bookmark
reddit
Simpy
StumbleUpon
Furl
Newsvine
Technorati
Tailrank