Extreme Programming and Open Source Software

Posted 13 Nov 2000 at 23:08 UTC by apm Share This

These are two of the hottest topics in software development today. How do they fit together? This is an important question to me, since at Suneido we are trying to embrace both. Here is my look at each of the elements of Extreme Programming (XP) and how they fit with Open Source Software (OSS). If you don't know about XP there should be enough here to introduce you, and to perhaps interest you in looking into it further.

"XP is a lightweight, efficient, low-risk, flexible, predictable,scientific, and fun way to develop software" -- Kent Beck, Extreme Programming Explained

"The basic idea behind open source is very simple. When programmers on theInternet can read, redistribute, and modify the source for a piece ofsoftware, it evolves. People improve it, people adapt it, people fix bugs. Andthis can happen at a speed that, if one is used to the slow pace ofconventional software development, seems astonishing." -- opensource.org

The four values of XP are:

  • communication
  • simplicity
  • feedback
  • courage

I think these all fit remarkably well with OSS. Most OSS projects rely onabundant communications via email, mailing lists, newsgroups, and discussionforums. Although the geographic separation of OSS teams might seem to hampercommunications, it may actually have a positive effect because thecommunication that does go on is normally written, recorded, and open toobservation. To me, simplicity is a given, although, in XP it has a specialsignificance that I think we can learn from. In XP "feedback" refers to anumber of things, including feedback from customers, tests, and otherprogrammers. Again, it seems like OSS projects often benefit from a lot offeedback. Courage means diving in and doing the work. OSS programmers arefamous for their courage. It obviously takes a lot of courage to jump into anongoing project and start contributing. But if everyone sat back, nothingwould get done.

The fundamental principles of XP are:

  • rapid feedback
  • incremental change
  • quality work
  • assume simplicity

Rapid feedback and incremental change fit well with the OSS tradition of"release early, release often". And of course, OSS projects pride themselveson their higher quality work as opposed to much flawed commercial software. According to Beck, "Assume Simplicity" means "do a good job of solving today'sjob today and trust your ability to add complexity in the future where youneed it." This applies to many OSS projects that start small and grow asneeded.

But the core of XP is it's practices:

The planning game

XP assumes you have a customer (or a proxy) as part of the project. OSSprojects generally don't have a well defined "customer". Nor is there asingle voice for their users. Instead, OSS projects tend to be guided by acombination of the vision of their key developers, and the "votes" of theirother developers and users. And of course with OSS, if anyone wants somethingbadly enough they can always write it themselves!

Small releases

Release early, release often. What more can I say?

Metaphor

The idea here is to have a coherent picture of the system that helpseveryone understand the basic elements and their relationships. I think theshared "vision" of many OSS projects is the equivalent of XP's metaphor. Withthe wide variety of developers on most OSS projects, anything that helps keepeveryone on track could obviously be beneficial.

Simple design

XP says do the simplest thing that could possibly work. This fits wellwith OSS development. After all no one is paying for a lot of up-frontdesign. In fact, OSS is sometimes criticized for its lack of "analysis anddesign". But according to Beck "If you believe that the future is uncertain,and you believe that you can cheaply change your mind, then putting infunctionality on speculation is crazy. Put in what you need when you needit." It seems that maybe OSS had the right approach all along.

Testing

Automated tests are a key part of XP. They are what give you the couragefor refactoring and collective ownership. They are what allow you to releaseearly and often. Although many OSS projects include test suites, I think thisis probably the practice that OSS could most benefit from. An OSS projectthat incorporated XP style tests would have enormous advantages. It wouldrequire less oversight and review because you'd know things worked. It wouldencourage more people to contribute because they would have immediate feedbackon whether their changes worked or not. And it would ensure that as thesoftware changed and grew and ported, that it wouldn't break. Although it'snot as extensive as we'd like our Suneido project has automated tests both inthe C++ code and in the standard library. They have proved invaluable.

Refactoring

"Refactoring is the process of changing a software system in such a waythat it does not alter the external behavior of the code yet improves itsinternal structure." -- Martin Fowler, Refactoring. Many OSS projects dorefactoring, but often reluctantly, and in large scale bursts. In XP yourefactor all the time, so that the design stays as clean as possible. Thefear, of course, is that you're going to break something, but this iscounteracted by having automated tests, and pair programming. I think OSScould benefit from adopting more XP style refactoring.

Pair programming

This is obviously not so easy when programmers are spread out around theworld. Some people have interpreted pair-programming in this context to meancode is always a result of two people's work, through a combination ofcollaboration and review.

Collective ownership

A broad interpretation of "open source" would say that by definition thecode is open to everyone. But in XP collective ownership means anyone canchange any part of the project - programmers are not restricted to a certainarea of expertise. Most large OSS projects have some restrictions on who isallowed to make changes directly, and what process is required for others tosubmit changes. This is necessary because of the large numbers ofcontributors. XP is generally aimed at teams of 10 programmers or less.

Continuous integration

OSS projects vary on this issue. Some integrate contributions continuouslyand others "batch" them up. Some projects allow anyone to commit changes e.g.via CVS whereas others require changes to be submitted for review. But on anindividual level, most OSS developers do practice continuous integration. After all, they want to see if their changes work! At Suneido we integrateinternally as often as several times a day, and update our web-site every weekor two.

40-hour week

It's hard to apply this to OSS developers that are working in their spare time. But one of the key differences is that most OSS developers are doing it because they want to, and because they enjoy it. This is a whole different ball-game from trying to force employees to work mega overtime.

On-site customer

Most OSS projects don't have a physical "site" or a specific "customer" let alone an "on-site customer". But the essence of this practice is to have good communications with your users, and as I've already discussed, I think most OSS projects have this.

Coding standards

Most OSS projects are pretty good at adhering to coding standards, for the same reasons that XP requires them. If you are going to have different programmers at different times messing with the code, then you have to follow some coding standards. On smaller OSS projects, the standards are often informally enforced by the key developer who massages contributions into his own style. Large OSS projects like GNOME usually have an explicit set of standards.

I hope this has given you some idea what XP is all about, and my thoughts on how XP fits with OSS. Although I'm not sure it's possible for an OSS project to adhere strictly to XP, I do think there are a lot of good ideas that can be applied. For an example of an OSS project that is trying to embrace XP, check out Suneido

Andrew McKinlay

Suneido Software

Resources

Extreme Programming Explained - Kent Beck

The Cathedral and the Bazaar - Eric S. Raymond

Refactoring - Martin Fowler

www.opensource.org

www.xprogramming.com

www.extremeprogramming.org


cynicism, posted 14 Nov 2000 at 00:32 UTC by apgarcia » (Journeyer)

xp is just another overhyped set of best practice type things, preached as though it's the silver bullet everyone's looking for, flogged to death in as many books as they can possibly crank out on the subject while it's still a popular buzzword and plenty of people still want to get in on it first, or if not first, at least before plenty of other people do, until it's been milked for all it's worth, but by then it doesn't matter anymore, cuz they've made enough money to spend the rest of their days on a yacht.

oss? no thanks, i prefer the free software perspective.

Embrace change, posted 14 Nov 2000 at 01:50 UTC by Pseudonym » (Journeyer)

Bazaar development (as opposed to Open Source, which is a superset; there are plenty of open source projects which do not employ bazaar development) is a natural for Extreme Programming. Remember that for all of the rules, principles and guidelines in XP, there's really only one rule: If something is worth doing, it's worth doing to extremes.

For example, KISS is good. So you should always "do the simplest thing that could possibly work". Design is good. So "refactor mercilessly". And so forth and so on.

Some of the principles designed for in-house development don't carry across well (e.g. pair programming), but XP allows you to drop some principles if they don't work in your situation. With some exceptions. For example, it's not extreme if you "do the simplest thing that could possibly work" and don't "refactor mercilessly". On the other hand there is no "forty hour week" on a volunteer project.

Extreme Programming has a lot of good advice, posted 14 Nov 2000 at 03:46 UTC by Bram » (Master)

The project I'm currently working on, Mojo Nation has a number of principles taken from (or at least voted for by) Extreme Programming. Most notably we probably wouldn't be doing pass/fail unit tests if I hadn't toyed with JUnit a bit. Simple design, quick releases, and scheduling based on dates of release rather than features to be done, while not advocated by Extreme Programming alone, were all ideas I understood much better and believed in much more after reading Extreme Programming Explained.

That said, Extreme Programming isn't the methodology - each project must in practice be carried out in it's own slightly different way. The Extreme Programming crowd are rather pedantic about the way, but they do have some good ideas.

I advocate developing a consistent methodology by codifying what you wind up doing in practice. There is no need to invent something a priori. My experience has been that if an activity feels like a waste of time it probably is a waste of time.

I'm a guinea pig too!, posted 14 Nov 2000 at 13:27 UTC by mvw » (Journeyer)

XP has been chosen as software development methodology at my work place. Most the ideas seem familiar.

The one I still have to get used to is pair programming.

This is a tighter coupling of developers than the usual "n developers in a room together" scheme, while I would rather switch to a looser coupling over email and common web sites, the one used in open source development, if I could. :-)

Not only because I am rather short-sighted (in the sense of myoptic :-) but because I believe I can concentrate better on written stuff and can answer better in written form. And that communication 's inherent delay helps too.

Contrast that to two folks sitting in front of a box, both having to work at same speed, concentrated on the same issue.

Right now, in this early stage of experience, I am not convinced yet. Possibly due to the present shortage of man power too - I simply don't buy that two developer's combined work is always as good or better than two developers working seperately.

But possibly it might turn useful for certain situations. Who knows.

eXtreme programming and pair programming, posted 14 Nov 2000 at 18:58 UTC by thomasd » (Journeyer)

I've been looking at XP methodologies for a while (and I know several people who are very keen on the idea). Some ideas are definitely good -- I know I should write many more test cases than I currently do. But I'm worried that the `Do the simplest thing which works' aspect of it can be taken as an excuse for not having any design at all.

When I'm coding, I always try to make clear distinctions between interfaces and implementations (I guess that having Java as my current primary language influences this a bit). I don't have any problem with writing throw-away implementations when they are expedient, but when a new public interface is added, this is always something to be pondered (and debated with the rest of the development group, if it's a collaberative project). Implementations can (and should!) come and go, but I've always seen a good, stable, set of interfaces as the key to making that work.

As for pair programming, I guess it's got it's ups and downs. Extra eyes certainly find bugs -- I often `code swap' with office-mates to track down irritating problems. But I have trouble being tied to someone else for more than a few minutes at a time. I'm not a terribly focused coder -- unless I'm writing something that's algorithmically difficult, I'll quite often sit with an editor buffer on one half of the screen and a mail reader or web-browser on the other and flick between the two. Probably not the kind of practice which is likely to get incorporated into a formal coding methodology any time soon, but it works for me, and keeps my concentration much higher than if I were staring at the same block of code for hours on end.

My thoughts...., posted 14 Nov 2000 at 23:10 UTC by dnm » (Master)

Actually, I was just about to post on this (XP + open source). I'm glad other people are looking at it.

I think Extreme Programming is a great idea. This doesn't necessarily mean it fits in every situation (no idea does), but inside of it's problem domain, I think it's excellent. Perhaps the reason there's a lot of hype about it is simply because the conditions which XP tries to attack (small teams of programmers developing software where the requirements are loosely defined and/or change rapidly) is becoming more the mode (which I think is true of most small software companies) these days.

Practical software design (whereby you produce a functional requirements doc, design, and then implement) is pretty rigid, but in large software projects or even in small projects where there are clearly defined constraints and goals that don't change (perhaps a la JPL), that rigidity can be good, so XP isn't for everyone. I think the XP chorus is accepting of that and has no problem with it -- use what works.

Also, XP grew out of highly object-oriented software development projects and the use of certain languages/environments in those projects (Smalltalk, mostly), so while I certainly wouldn't say that XP doesn't work for any non-OO language, I would say that the use of a complete OO language makes XP much easier (I can't imagine XP on assembly code being necessarily all that more useful over regular design proceedures, for instance). By the same token, what I find interesting is that a number of XP practices are things I've been doing all along in functional programming (I'm a big Lisp person, and a functional programming adovcate in general). Regular design proceedures tend to constrain people who use functional langauges, since the natural evolution of software in functional languages is (usually) much different than that of proceedural languges, which I especially notice when doing things in Common Lisp (I like to take advantage of things like macros, the MOP, and full reflective-ness).

XP and distributed development, posted 15 Nov 2000 at 00:36 UTC by Dacta » (Journeyer)

I'm a big fan of XP, but I'm not convinced that conventional XP will work in the normal distributed OSS project.

Firstly, XP relies on always doing the simplest thing to get something working, and continual refactoring of other components. These two habits seem to be at odds with the idea of a distributed group of people working of separate modules of code. This is tied into the XP idea of group ownership of code - no one is responsible for a single part of code. In most OSS projects, it is the total opposite - a single person is responsible for driving the development of their module.

The Pair Programming part of XP obviously does not occur in a distributed OSS environment, but in some cases this if offset by the many eyeballs factor - while you may program alone, many people look at your work and find bugs. This depends on the size of your development community, though, of course.

The XP emphasis on testing is a good thing, and some OSS projects do similar things already.

I think where most OSS programmers would have a problem with XP is the do the simplest thing philosophy. That is totally against the huge plans most OSS projects make (the BDUF - Big Design Up Front - philosophy).

This difference in philosopy may be caused by the areas that XP and OSS specialise in. Most XP proponants write business applications for non-software companies (ie, in a corporate environment). This means they have control over the release procedure, so they can put out incompaitible releases - they know there will never be any old versions around. In the OSS environement (with no real control over who uses which release), compatibility is always important, so it is sometimes better to put stuff in at the start of the project, then fix it later - that way there won't be so many incompatibilities - just bugs and fixes.

Serial Pair Programming, posted 15 Nov 2000 at 01:24 UTC by sej » (Master)

I think we could use one twist on the Pair Programming concept, something like Serial Pair Programming as opposed to Parallel Pair Programming. There are many quality foundations of free software available where the original developers would be glad to mentor new programmers but no longer have the time to proactively extend and maintain what they started. Not every creator of open source can indefinitely continue to work on a project at full tilt . But any quality piece of software needs constant maintenance and extension to remain viable to its users.

So we see many older software projects wither as the life circumstances of the original developers change, and amazingly similar yet disconnected new projects get started where an enormous effort is required to replicate the work of others. With Serial Pair Programming, the knowledge-rich time-poor original programmers could assist the time-rich apprentices as they get up to speed. Theory would say that these collaborations would take us all a lot farther than the preponderance of born-yesterday vanity projects you see at places like sourceforge.

I have strong caveats to what I just wrote. I believe in diversity and freedom as much as I believe in continuity and continual evolution. If you think you have a better idea, or you just want to do your own thing, please do. But if you believe you might have something to learn from someone else's software or programming experience, why not find a source package you like and apprentice yourself?

Twisted Pair Programming, posted 15 Nov 2000 at 01:30 UTC by MisterBad » (Master)

I AM 3XTR33M, D00D

well, posted 15 Nov 2000 at 13:37 UTC by jdube » (Journeyer)

Considering the first time I saw the book eXtreme programming the first thing that popped into my head was someone with a laptop base jumping while hacking Perl, this has opened me up a little bit to what it really is. While it does seem a little, um, buzz-wordy, the ideals you posted here sound nice. I admit, I will miss the chance to be able to hang out with my friends in a coding session and leap up at random times shouting EXTREME!!! WHOO!! now that I know what it is.

Do The Simplest Thing != no design at all, posted 15 Nov 2000 at 15:05 UTC by lalo » (Journeyer)

A lot of people react badly to "Do The Simplest Thing That Works" because of a gut feeling on the importance of design. Well, so do your design. What XP tells you is to not waste lots of hours in design trying to predict the future, because you WILL be wrong. Design for today's needs, don't add a lot of hooks for tomorrow. Then when you need to change the interface tomorrow, if there is already third-party software using this interface, refactor in a backwards-compatible way.

Granted, a lot of XP advocates will tell you not to design at all. But most of the serious ones do some design. In special, I'd point you to Fowler's work on XP, and some of the stuff by Cockburn.

Kind of like the "Infinite Monkeys" idea...., posted 15 Nov 2000 at 18:07 UTC by mrorganic » (Journeyer)

Extreme Programming reminds me of the "Infinite Monkeys" idea -- i.e., that if you put an infinite number of monkeys with an infinite number of typewriters into a room, eventually they will produce the collected works of Shakespeare. Each monkey does the simplest thing -- bang on a typewriter key -- and somehow all these simple events will lead up to the greatest work of literature the world has ever known.

The only problem is, statistically, this wouldn't happen before the heat-death of the universe. I suspect the same is true of XP.

Yet another ill-conceived fad meant to line the pockets of IT consultants. God save us all!

another link, posted 15 Nov 2000 at 18:09 UTC by apm » (Journeyer)

Mats Henricson pointed out that Refactoring also has a website:

www.refactoring.com

Thanks Mats.

XP and Free Software, posted 15 Nov 2000 at 23:51 UTC by avi » (Apprentice)

for those that have been reading my diary here, you'll know that i have a new Free software project starting soon that will be sold commercially as well. it also happens that i recently read the first XP book, and found it really interesting - if somewhat lacking in the "how-to" area. last week i stumbled across the new, unpublished XP book on the web. it is available here (ftp) or here (http). it is an excellent read, even if some of the parts of the book are in their final draft - so good in fact that we are going to use XP for our project to see what it is like in practise. each chapter focuses on a different practical aspect of how to implement XP.

even though no development methodology is perfect, i think XP will be able help us create a better product in a shorter amount of time.

Misunderstanding of "the simplest thing", posted 16 Nov 2000 at 03:25 UTC by Pseudonym » (Journeyer)

lilo, mrorganic: A lot of people misunderstand the "do the simplest thing that could possibly work" maxim. It doesn't work unless you also "refactor mercilessly". The idea is that the design of your software evolves. You mutate it to add a new feature or fix a bug, then you refactor to improve the design of what you have. And you do it mercilessly. The point is not to avoid design. Far from it. Design is a great thing! So in XP you do it to extremes: you design ALL the time.

BTW, the subtitle of the refactoring book is "Improving the design of existing software".

new XP books, posted 16 Nov 2000 at 13:50 UTC by apm » (Journeyer)

Both new XP books are now out:

Extreme Programming Installed - Ron Jeffries, Ann Anderson, Chet Hendrickson

Planning Extreme Programming - Kent Beck and Martin Fowler

(Addison-Wesley)

XP and design, posted 17 Nov 2000 at 01:35 UTC by avi » (Apprentice)

Check out Martin Fowler's keynote from the XP2000 conference here. The speech is entitled "is design dead?", and is quite a good read.

Re: "Kind of like the "Infinite Monkeys" idea...., p, posted 17 Nov 2000 at 13:27 UTC by mvw » (Journeyer)

Saw one .signature once that had a nice answer to the Infinity Monkey scenario:
"The internet proves otherwise" :-)

Another take on requirements analysis, posted 19 Nov 2000 at 18:27 UTC by rlk » (Journeyer)

This article by Lisa G. R. Henderson of the Industrial Engineering Department at Mississippi State University has some interesting ideas about how requirements analysis might be performed in a free source environment.

Simple?, posted 20 Nov 2000 at 07:51 UTC by sergent » (Journeyer)

So I like to think I was doing a number if not all of these things before I had heard of XP. I disagree with the comments made above about design, however.

Simple design does not mean no design. Simple designs don't preclude creating complex systems; in fact, they facilitate them. Things that are not designed well end up having interfaces that either not well suited to the problem at hand or are unstable because they are continuously changing since the requirements were not managed. Trying to put things together that don't have interfaces that work together is hard.

Someone--I forget whom--said that UNIX pipes were the only component model that ever worked. I think the underlying statement is that interfaces that are more complicated than UNIX files (forget ioctl and sockets for now; I think these are things that were hacked on later that don't fit the original model, and are a symptom of the problem mentioned above) are too complicated to be successful. I think this is a bit of an exaggeration, but it is not orders of magnitude off.

Reference, posted 19 Oct 2001 at 23:07 UTC by her » (Journeyer)

There is also a Wiki page on this subject: CombiningOpenSourceAndXp

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page