Against the Grain: Getting Projects To Work Together

Posted 6 May 2003 at 14:28 UTC by gregorrothfuss Share This

by Gregor J. Rothfuss and Paul Everitt

Last year we wrote a piece here questioning whether open source projects were motivated to work with each other. This article was in the context of our work with OSCOM and its attempts to bring together open source content management projects and developers.

The article stimulated some interesting questions. Now that we've had some time to take some steps and try some ideas, we thought it appropriate to report back.

In summary, OSCOM interop has caught on more than expected, but the hard work remains ahead. Also, other open source interop efforts such as freedesktop.org and Linux Standards Base have shown momentum despite the odds. We have found that open source projects collaborate much less than the open access to source code would suggest.

Why does open source fall into the Not Invented Here syndrome so often?

As before, the number one barrier to open source interop is cultural. Does open source make closed minds? How do tribes form a nation? Tackling this issue isn't in the domain of software engineering, rather, social engineering.

A Recap of the Previous Discussion

The Interop in the Bazaar article described three disincentives to interop: unfunded scope creep, tribalism as a cultural issue, and the boring challenge of writing standards documents. We then listed interop's beneficiaries and upsides before making some predictions.

Let's review the accuracy of our guesses:

Prediction One: Interop between open source projects is a fool's errand.
100% true, as we're fools, and we're still at it. :^)
Prediction Two: If we stay practical and focus on small steps, we can provide value with lower risk.
Prediction One notwithstanding, we're finding within OSCOM that practical, bite-sized interop is interesting to implementors. Bottom line: If you can whip up a prototype in a day, and brag about it over beers that same evening, then chances for success go up considerably.
Prediction Three: We'll stumble across the Big Idea that is the bait to get the fish (project leaders) on the hook in a big way.
Below we discuss how we used an OSCOM sprint in Zurich to jointly develop an app that gives incentive to implement WebDAV.
Prediction Four: Somebody will get sued over a patent infringement and we'll all move to Norway.
No news yet from Interwoven. :^)

So What Happened?

As noted previously, the first step (a will to work together) is the hardest. This is discussed below. Suffice it to say, open source does not always equate to open minds. Thus, we have a real accomplishment here.

OSCOM 2 in Berkeley proved that OSCOM was still interesting. The conference proved smaller than Zurich, but more intense. We found a deeper connection to our users and learned the most relevant directions to pursue.

From the press coverage, we also found that OSCOM is, in many ways, a better story than our individual projects. Journalists are bombarded by story pitches from the teeming masses of CMS vendors and projects. Everything sounds the same. However, OSCOM -- with its pitch of cooperating to better serve customers -- is a fresh story. Fresh means that journalists will write stories, and thus we can attract the desperate sustenance of attention.

For OSCOM 2, we launched a project called SlideML to encode and produce presentations. SlideML was a good, appropriately-sized first step, and we learned from it. Primarily, projects only work when someone cares enough to work on it, even if nobody else does. In this case we mean Roger Fischer from BitFlux. These champions become the glue that promotes and accumulates sporadic efforts from contributors.

OSCOM also tackles marketing issues, and one glaring need was an authoritative, independent grid of CMS projects. This is the CMSML project at OSCOM. It was more ambitious, as it was more about marketing than software development. CMSML has also exceeded expectations, as other similar efforts were interested to collaborate.

CMSML marks an effort where we crossed another cultural barrier, that between Free / Open Source and proprietary software. We're still alive to talk about it. One roadblock that we found was that many projects do not want to be compared head-to-head with their proprietary competitors (and vice versa). Lesson: Don't be afraid of the proprietary world, and seek collaboration for open standards with it wherever possible.

We also hosted a developer "sprint" in Zurich. This was the first attempt at a close-hand development effort between different projects. We were nervous that the sprint would devolve into two days of mail-reading. Instead, it greatly exceeded expectations.

Twingle, an OSCOM project to build a common CMS authoring tool, was a more ambitious attempt to pool energy to meet a common need. Though very young, Twingle sparked interest and was quickly able to spur small steps towards interoperability. Specifically, it helped incent servers to move WebDAV support up on the priority list. Since Twingle uses Mozilla's RDF engine, it brought CMS attention to the Semantic Web. Twingle reinforced the lesson that quick, tangible efforts are ideal for bringing unity.

Culture Wars

As mentioned in the lead-in, the number one challenge open source interop faces is cultural biases within communities. Why work with the other guy? Tribal forces are particularly strong in open source.

Tackling this first requires an understanding of the open source motivation. Open source projects do not have much besides their social fabric. All "payment" is in the form of reputation within that small subculture. Thus the subculture is vigorously defended against intruders, and cross-culture efforts like OSCOM need to build their own culture and reputation systems to compete.

The memetic explanation for these effects is that autoimmune responses kick in to defend the culture of a project against threats from the outside.

Stated differently, open source interop needs to seed the interop effort with the same upside as traditional open source projects. Without a strong community that rewards interop, project collaborations are doomed to fail.

Lessons Learned

  • Developers care more than expected about working together.
  • Working together bears fruit when it is perceived as fun, such as a sprint.
  • We all have a lot to learn from each other.
  • We're too busy to launch ambitious standards agendas, and that's not our strength anyway. Focus on practical standards that overlap our common agenda.
  • The shared agenda is good for customers, in a way that proprietary vendors can't match without facing a shareholder lawsuit. (For related material, see Permanence, Replacability, and Inventor's Disease.)
  • The real opportunity with OSCOM is that these are the people actually making the systems. If something is agreed to , and they have the time, then it's done. And when it's done, it becomes part of several large communities, reducing the time needed to hit the tipping point.
  • Journalists have told us that the common story (OSCOM interop) will sell better than the individual stories (Apache Lenya, Midgard, Zope, etc.). It's a crowded market and we need to attract attention. In this case, "attention together" is better than "ignored alone".

A Common Culture

OSCOM is a place where CMS implementors hang out. Sure, we have an agenda: make the mainstream CMS market aware of the open source advantage, foster inter-project interop, share the cost of common needs, etc.

But the first and hardest task is in the bag. We have created a community. We do things together, we know each other, we look forward to seeing each other. Inside OSCOM, we really set aside our tribes and we rarely jockey for position by promoting our narrow interest. "Me" yields to "we".

What's Next

OSCOM 3 is focused on "Semantic Web meets CMS". This brings OSCOM into an important, albeit contentious arena. Also, we're changing the conference format to be more about participating than listening.

Twingle, if it reaches its goal of improving the user experience, can be a reward for work spent on interoperability.

Conclusion

The first article posed the question, "Do open source developers want interop?" It remains an open question. Still, OSCOM is making a bet that the answer, when the right circumstances are created, can be yes.

This second article shows OSCOM at its first steps in the journey. We've created a small community and culture of CMS implementors. We all know each other now, we enjoy each other's company, and we expend real effort towards mutual success.

With this base, we are tackling practical interop between open source CMS projects. We are also tackling common marketing initiatives.

However, the question remains. To leave a mark and make a real difference to projects and customers, we must deliver results to the question, "Do open source developers care about working together?"


Do open source developers care about working together?, posted 7 May 2003 at 16:12 UTC by etrepum » (Journeyer)

Yes, of course they do. That's why there are many projects out there with quite a few developers involved that have done great things together. Many of the developers enjoy working with each other. This is usually something that is intraproject rather than interproject. The real question is: Do developers from project A want to make their software work with project B? This really breaks down to quite a few more questions. Here's a few off the top of my head:

  • Do the developers from project A even know about project B, if so, have any of them tried to use it?
    This is definitely one case of Not Invented Here, they have never seen nor used the other project

  • Does project B work? Is it finished enough to care about?
    Nobody wants to use software that doesn't work, unless they have a vested interest to make it work.

  • Is it hard to make project A to work with project B?
    Most of the open source developers I know seem to work on things that are interesting or useful to them (for fun or profit). Integrating two products isn't usually top of the list if it's excruciating to do.

  • Are project A and project B license compatible?
    For example, If project A is BSD and project B is GPL, the project A developers may want to develop a BSD alternative of project B. GPL rubs a lot of people the wrong way, especially the ones with jobs. Personally, I use GPL software just like I use proprietary software. If the software is LGPL, BSD, etc. I'm much more likely to contribute development hours, personally or professionally. There are exceptions, but GPL is a strike against many open source projects for me, especially when it's a library or framework.

  • Are any developers shared between project A and project B?
    If there's common developers, there's a good chance someone will decide to integrate the two, because you have people familiar with the source code on both sides of the fence.

  • Do any project A developers need the additional functionality provided by integrating with project B?
    If it's not useful to them, they probably won't do it.

  • Is project B written in the same programming language?
    This usually has something to do with the difficulty of making them talk, but sometimes it's more philisophical. For example, a lot of developers don't like Java, and a lot of developers ONLY like Java. These two groups of developers probably won't make their software cooperate. Also, you have languages like Perl and Python where the development philosophies and coding styles are so different the developers from either project may not understand the other at all.

  • Is it easier or more worthwhile to just make project A encompass the functionality of project B, perhaps doing it better in the process?
    A classic case of Not Invented Here. Sometimes emulating project B is a heck of a lot easier than just using project B.. and sometimes it makes a lot of sense, because project B might just not have been done well, or not developed in a compatible way with project A. This happens ALL the time, especailly with event driven frameworks where threaded implementations are just not compatible (even if they can be made compatible, it's usually not a good idea unless absolutely necessary) with non-blocking or polling implementations.

OSS interoperability better than closed source, posted 7 May 2003 at 16:37 UTC by jrobbins » (Master)

I think the title article is provocative and a bit misleading: I see a much higher level of interoperability among open source projects than among closed source products.

There will always be some projects that have not achieved everything that they wanted, so listing a few projects that have had difficulties does not make the argument. We could very easily list major open source projects that have played very important roles in interoperability and standards. In general, the Apache Software Foundation is at the top of that list, along with mozilla, IMHO. One specific example is the subversion project, it a very important step toward the use of more standards in CM systems and a useful tool right now.

On the closed source side, you see many interoperability failures in mature products, as well as a vested interest in proprietary file formats or protocols as a way to try to lock in users. In some cases you even see non-interoperability used in anti-competitive practices, e.g., websites that reject certain browsers that are not made by the same company.

Success Stories, posted 7 May 2003 at 19:49 UTC by ncm » (Master)

The best way to find out how to get Free Software projects to work together on interoperability is to study the ones that did it successfully, and do what they did. Talk to the ones who made it happen. They have thought about it a lot more deeply than anybody who just wishes it would. They've tried things they thought would work, and found what didn't, and then what did. Write down what they say. Then, show it to groups you wish would work together better.

Maybe something will happen, or maybe the integration was doomed from the start. Either way, no time is wasted.

working at working together, posted 7 May 2003 at 23:24 UTC by pate » (Apprentice)

I'm steering/herding/stirring the pot of Koha, a free software library automation system. There are a few other similar projects ot there, and a few of us (thanks Art), are trying to push for a higher level of cooperation. So far, it's not happening.

I think that ego and wildly diverging intent are mostly to blame, but I'd certainly like to find the bait needed to get the various development groups to at least talk to each other. One of the things that I think would really help would be to run some kind of a summit/con is the way to go, but I'm not sure how to best go about it. Any thoughts?

names/bashing/summits, posted 9 May 2003 at 00:15 UTC by slef » (Master)

I think one of the main problems may be mutual incomprehensibility: specialist applications sometimes grow their own jargon and it can be difficult to understand that one program's sproidal is another's wornik. I'm not sure how that can be overcome without mutual developers.

Some of these collaboration efforts have poor names. OSCOM is a good example, giving the impression that free software is not welcome or a marginalised minority.

That said, I'm sad to see GPL-bashing in a reply above. GPL is not anti-job any more than BSD is donating your work to an evil corporation. Please don't make wild claims that have little to do with the topic.

pate, if you want a free software library summit to happen, I'd like to help once the conf I'm currently organising is out of the way. That said, I'm not sure that the UK is such a great venue. Maybe regional summits are useful anyway?

re: names/bashing/summits, posted 9 May 2003 at 20:54 UTC by etrepum » (Journeyer)

Linking to or integrating GPL code into a proprietary codebase is not license compatible. Linking to LGPL code from a proprietary codebase is license compatible. Linking to or integrating BSD code into a proprietary codebase is license compatible. Some of us have to work with partially proprietary codebases during the day which means GPL is mostly out of the question.

That said, when favorably licensed project ends up in whole or in part in a proprietary project of that I'm involved with I almost always end up being a significant contributor to the project and/or contract out related parts of the proprietary project itself or just feature requests to developers of said favorably licensed project. It's win-win just about any way you slice it. Unless it's just for fun on my own time, I generally can't do this with GPL software.

re: names/bashing/summits, posted 10 May 2003 at 08:41 UTC by slef » (Master)

etrepum, nothing in what you write is evidence that GPL is anti-job in any way, just that it is a resource not available to proprietary software that wants to link it in. I think most people know that. You could use GPL code if you negotiated a GPL-compatible licence for the proprietary parts of your codebase. I'm sorry if your job doesn't allow you to do that, but surely that just means that your job is anti-GPL? (and possibly anti-copyleft)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page