Extreme Programming and Open Source Software
Posted 13 Nov 2000 at 23:08 UTC by apm
These are two of the hottest topics in software development today.
How do they fit together? This is an important question to me, since
at Suneido we are trying to
both. Here is my look at each
of the elements of Extreme Programming (XP) and how they fit with Open
Source Software (OSS). If you don't know about XP there should be
enough here to introduce you, and to perhaps interest you in looking
into it further.
"XP is a lightweight, efficient, low-risk, flexible,
predictable,scientific, and fun way to develop software" -- Kent Beck,
Extreme Programming Explained
"The basic idea behind open source
is very simple. When programmers on theInternet can read, redistribute,
and modify the source for a piece ofsoftware, it evolves. People
improve it, people adapt it, people fix bugs. Andthis can happen at a
speed that, if one is used to the slow pace ofconventional software
development, seems astonishing." -- opensource.org
values of XP are:
I think these all fit
remarkably well with OSS. Most OSS projects rely onabundant
communications via email, mailing lists, newsgroups, and
discussionforums. Although the geographic separation of OSS teams
might seem to hampercommunications, it may actually have a positive
effect because thecommunication that does go on is normally written,
recorded, and open toobservation. To me, simplicity is a given,
although, in XP it has a specialsignificance that I think we can learn
from. In XP "feedback" refers to anumber of things, including feedback
from customers, tests, and otherprogrammers. Again, it seems like OSS
projects often benefit from a lot offeedback. Courage means diving in
and doing the work. OSS programmers arefamous for their courage. It
obviously takes a lot of courage to jump into anongoing project and
start contributing. But if everyone sat back, nothingwould get
The fundamental principles of XP are:
- incremental change
- quality work
- assume simplicity
Rapid feedback and incremental change
fit well with the OSS tradition of"release early, release often". And
of course, OSS projects pride themselveson their higher quality work as
opposed to much flawed commercial software. According to Beck, "Assume
Simplicity" means "do a good job of solving today'sjob today and trust
your ability to add complexity in the future where youneed it." This
applies to many OSS projects that start small and grow
But the core of XP is it's practices:
XP assumes you have a customer (or a proxy)
as part of the project. OSSprojects generally don't have a well
defined "customer". Nor is there asingle voice for their users.
Instead, OSS projects tend to be guided by acombination of the vision
of their key developers, and the "votes" of theirother developers and
users. And of course with OSS, if anyone wants somethingbadly enough
they can always write it themselves!
Release early, release often. What more can I
The idea here is to have a
coherent picture of the system that helpseveryone understand the basic
elements and their relationships. I think theshared "vision" of many
OSS projects is the equivalent of XP's metaphor. Withthe wide variety
of developers on most OSS projects, anything that helps keepeveryone on
track could obviously be beneficial.
XP says do the simplest thing that could possibly
work. This fits wellwith OSS development. After all no one is paying
for a lot of up-frontdesign. In fact, OSS is sometimes criticized for
its lack of "analysis anddesign". But according to Beck "If you
believe that the future is uncertain,and you believe that you can
cheaply change your mind, then putting infunctionality on speculation
is crazy. Put in what you need when you needit." It seems that maybe
OSS had the right approach all
Automated tests are a key part
of XP. They are what give you the couragefor refactoring and
collective ownership. They are what allow you to releaseearly and
often. Although many OSS projects include test suites, I think thisis
probably the practice that OSS could most benefit from. An OSS
projectthat incorporated XP style tests would have enormous
advantages. It wouldrequire less oversight and review because you'd
know things worked. It wouldencourage more people to contribute
because they would have immediate feedbackon whether their changes
worked or not. And it would ensure that as thesoftware changed and
grew and ported, that it wouldn't break. Although it'snot as extensive
as we'd like our Suneido project has automated tests both inthe C++
code and in the standard library. They have proved
"Refactoring is the
process of changing a software system in such a waythat it does not
alter the external behavior of the code yet improves itsinternal
structure." -- Martin Fowler, Refactoring. Many OSS projects
dorefactoring, but often reluctantly, and in large scale bursts. In XP
yourefactor all the time, so that the design stays as clean as
possible. Thefear, of course, is that you're going to break something,
but this iscounteracted by having automated tests, and pair
programming. I think OSScould benefit from adopting more XP style
obviously not so easy when programmers are spread out around theworld.
Some people have interpreted pair-programming in this context to
meancode is always a result of two people's work, through a combination
ofcollaboration and review.
A broad interpretation of "open source" would
say that by definition thecode is open to everyone. But in XP
collective ownership means anyone canchange any part of the project -
programmers are not restricted to a certainarea of expertise. Most
large OSS projects have some restrictions on who isallowed to make
changes directly, and what process is required for others tosubmit
changes. This is necessary because of the large numbers
ofcontributors. XP is generally aimed at teams of 10 programmers or
vary on this issue. Some integrate contributions continuouslyand
others "batch" them up. Some projects allow anyone to commit changes
e.g.via CVS whereas others require changes to be submitted for review.
But on anindividual level, most OSS developers do practice continuous
integration. After all, they want to see if their changes work! At
Suneido we integrateinternally as often as several times a day, and
update our web-site every weekor two.
It's hard to apply this to OSS developers that are working in their
time. But one of the key differences is that most OSS developers are
because they want to, and because they enjoy it. This is a whole
ball-game from trying to force employees to work mega overtime.
Most OSS projects don't have a physical "site" or a
specific "customer" let
alone an "on-site customer". But the essence of this practice is to
communications with your users, and as I've already discussed, I think
OSS projects have this.
Most OSS projects are pretty good at adhering to coding standards,
same reasons that XP requires them. If you are going to have different
programmers at different times messing with the code, then you have to
some coding standards. On smaller OSS projects, the standards are often
informally enforced by the key developer who massages contributions
own style. Large OSS projects like GNOME usually have an explicit set
I hope this has given you some idea what XP is all about, and my
on how XP fits with OSS. Although I'm not sure it's possible for an OSS
project to adhere strictly to XP, I do think there are a lot of good
that can be applied. For an example of an OSS project that is trying to
embrace XP, check out Suneido
Extreme Programming Explained - Kent Beck
The Cathedral and the Bazaar - Eric S. Raymond
Refactoring - Martin Fowler
cynicism, posted 14 Nov 2000 at 00:32 UTC by apgarcia »
xp is just another overhyped set of best practice type things, preached as though it's the silver bullet everyone's looking for, flogged to death in as many books as they can possibly crank out on the subject while it's still a popular buzzword and plenty of people still want to get in on it first, or if not first, at least before plenty of other people do, until it's been milked for all it's worth, but by then it doesn't matter anymore, cuz they've made enough money to spend the rest of their days on a yacht.
oss? no thanks, i prefer the free software perspective.
Embrace change, posted 14 Nov 2000 at 01:50 UTC by Pseudonym »
Bazaar development (as opposed to Open Source, which is a superset;
there are plenty of open source projects which do not employ bazaar
development) is a natural for Extreme Programming. Remember that for
all of the rules, principles and guidelines in XP, there's really only
one rule: If something is worth doing, it's worth doing to extremes.
For example, KISS is good. So you should always "do the simplest
thing that could possibly work". Design is good. So "refactor
mercilessly". And so forth and so on.
Some of the principles designed for in-house development don't carry
across well (e.g. pair programming), but XP allows you to drop some
principles if they don't work in your situation. With some
exceptions. For example, it's not extreme if you "do the simplest thing
that could possibly work" and don't "refactor mercilessly". On the
other hand there is no "forty hour week" on a volunteer project.
The project I'm currently working on, Mojo Nation has a number of
principles taken from (or at least voted for by) Extreme Programming.
Most notably we probably wouldn't be doing pass/fail unit tests if I
hadn't toyed with JUnit a bit. Simple design, quick releases, and
scheduling based on dates of release rather than features to be done,
while not advocated by Extreme Programming alone, were all ideas I
understood much better and believed in much more after reading Extreme
That said, Extreme Programming isn't the methodology - each
project must in practice be carried out in it's own slightly different
way. The Extreme Programming crowd are rather pedantic about the
way, but they do have some good ideas.
I advocate developing a consistent methodology by codifying what you
wind up doing in practice. There is no need to invent something a
priori. My experience has been that if an activity feels like a waste of
time it probably is a waste of time.
XP has been chosen as software development methodology at my work place.
Most the ideas seem familiar.
The one I still have to get used to is pair programming.
This is a tighter coupling of developers than the usual "n developers in
a room together" scheme, while I would rather switch to a looser
email and common web sites, the one used in open source development,
if I could. :-)
Not only because I am rather short-sighted (in the sense of myoptic
but because I believe I can concentrate better on written stuff and can
better in written form. And that communication 's inherent delay helps
Contrast that to two folks sitting in front of a box, both having to
work at same speed, concentrated on the same issue.
Right now, in this early stage of experience, I am not convinced yet.
Possibly due to the present shortage of man power too - I simply don't
buy that two developer's combined work is always as good or better
than two developers working seperately.
But possibly it might turn useful for certain situations. Who knows.
I've been looking at XP methodologies for a while (and I know several
people who are very keen on the idea). Some ideas are definitely good --
I know I should write many more test cases than I currently do. But I'm
worried that the `Do the simplest thing which works' aspect of it can be
taken as an excuse for not having any design at all.
When I'm coding, I
always try to make clear distinctions between interfaces and
guess that having Java as my current primary language influences this a
bit). I don't have any problem with writing throw-away implementations
when they are expedient, but when a new public interface is added, this
always something to be pondered (and debated with the rest of the
group, if it's a collaberative project). Implementations can (and
and go, but I've always seen a good, stable, set of interfaces as the
making that work.
As for pair programming, I guess it's got it's ups and downs. Extra
certainly find bugs -- I often `code swap' with office-mates to track
irritating problems. But I have trouble being tied to someone else for
than a few minutes at a time. I'm not a terribly focused coder --
writing something that's algorithmically difficult, I'll quite often sit
an editor buffer on one half of the screen and a mail reader or
on the other and flick between the two. Probably not the kind of
which is likely to get incorporated into a formal coding methodology any
time soon, but it works for me, and keeps my concentration much higher
than if I were staring at the same block of code for hours on end.
My thoughts...., posted 14 Nov 2000 at 23:10 UTC by dnm »
Actually, I was just about to post on this (XP + open source). I'm glad other people are looking at it.
I think Extreme Programming is a great idea. This doesn't necessarily mean it fits in every situation (no idea does), but inside of it's problem domain, I think it's excellent. Perhaps the reason there's a lot of hype about it is simply because the conditions which XP tries to attack (small teams of programmers developing software where the requirements are loosely defined and/or change rapidly) is becoming more the mode (which I think is true of most small software companies) these days.
Practical software design (whereby you produce a functional requirements doc, design, and then implement) is pretty rigid, but in large software projects or even in small projects where there are clearly defined constraints and goals that don't change (perhaps a la JPL), that rigidity can be good, so XP isn't for everyone. I think the XP chorus is accepting of that and has no problem with it -- use what works.
Also, XP grew out of highly object-oriented software development projects and the use of certain languages/environments in those projects (Smalltalk, mostly), so while I certainly wouldn't say that XP doesn't work for any non-OO language, I would say that the use of a complete OO language makes XP much easier (I can't imagine XP on assembly code being necessarily all that more useful over regular design proceedures, for instance). By the same token, what I find interesting is that a number of XP practices are things I've been doing all along in functional programming (I'm a big Lisp person, and a functional programming adovcate in general). Regular design proceedures tend to constrain people who use functional langauges, since the natural evolution of software in functional languages is (usually) much different than that of proceedural languges, which I especially notice when doing things in Common Lisp (I like to take advantage of things like macros, the MOP, and full reflective-ness).
I'm a big fan of XP, but I'm not convinced that conventional XP will
work in the normal distributed OSS project.
Firstly, XP relies on always doing the simplest thing to get
something working, and continual refactoring of other components. These
two habits seem to be at odds with the idea of a distributed group of
people working of separate modules of code. This is tied into the XP
idea of group ownership of code - no one is responsible for a single
part of code. In most OSS projects, it is the total opposite - a single
person is responsible for driving the development of their module.
The Pair Programming part of XP obviously does not occur in a
distributed OSS environment, but in some cases this if offset by
the many eyeballs factor - while you may program alone, many
people look at your work and find bugs. This depends on the size of
your development community, though, of course.
The XP emphasis on testing is a good thing, and some OSS projects do
similar things already.
I think where most OSS programmers would have a problem with XP is
the do the simplest thing philosophy. That is totally against
the huge plans most OSS projects make (the BDUF - Big Design Up Front -
This difference in philosopy may be caused by the areas that XP and
OSS specialise in. Most XP proponants write business applications for
non-software companies (ie, in a corporate environment). This means
they have control over the release procedure, so they can put out
incompaitible releases - they know there will never be any old versions
around. In the OSS environement (with no real control over who uses
which release), compatibility is always important, so it is sometimes
better to put stuff in at the start of the project, then fix it later -
that way there won't be so many incompatibilities - just bugs and
I think we could use one twist on the Pair Programming concept,
something like Serial Pair Programming as opposed to Parallel Pair
Programming. There are many quality foundations of free software
available where the original developers would be glad to mentor new
programmers but no longer have the time to proactively extend and
maintain what they started. Not every creator of open source can
indefinitely continue to work on a project at full tilt . But any
quality piece of software needs constant
maintenance and extension to remain viable to its users.
So we see
many older software projects wither as the life circumstances of the
original developers change, and amazingly similar yet disconnected new
projects get started where an enormous effort is required to replicate
the work of others. With Serial Pair Programming, the knowledge-rich
time-poor original programmers could assist the time-rich apprentices as
they get up to speed. Theory would say that these collaborations would
take us all a lot farther than the preponderance of born-yesterday
vanity projects you see at places like sourceforge.
I have strong caveats to what I just wrote. I believe in diversity and
freedom as much as I believe in continuity and continual evolution. If
you think you have a better idea, or you just want to do your own thing,
please do. But if you believe you might have something to learn from
someone else's software or programming experience, why not find a source
package you like and apprentice yourself?
well, posted 15 Nov 2000 at 13:37 UTC by jdube »
Considering the first time I saw the book eXtreme programming the first
thing that popped into my head was someone with a laptop base jumping
while hacking Perl, this has opened me up a little bit to what it
really is. While it does seem a little, um, buzz-wordy, the ideals you
posted here sound nice. I admit, I will miss the chance to be able to
hang out with my friends in a coding session and leap up at random
times shouting EXTREME!!! WHOO!! now that I know what it is.
A lot of people react badly to "Do The Simplest Thing That Works"
because of a gut feeling on the importance of design. Well, so do your
design. What XP tells you is to not waste lots of hours in design trying
to predict the future, because you WILL be wrong. Design for
today's needs, don't add a lot of hooks for tomorrow. Then when you need
to change the interface tomorrow, if there is already third-party
software using this interface, refactor in a backwards-compatible way.
Granted, a lot of XP advocates will tell you not to design at all. But
most of the serious ones do some design. In special, I'd point
you to Fowler's work on XP, and some of the stuff by Cockburn.
Extreme Programming reminds me of the "Infinite Monkeys" idea -- i.e.,
that if you put an infinite number of monkeys with an infinite number
of typewriters into a room, eventually they will produce the collected
works of Shakespeare. Each monkey does the simplest thing -- bang on a
typewriter key -- and somehow all these simple events will lead up to
the greatest work of literature the world has ever known.
The only problem is, statistically, this wouldn't happen before the
heat-death of the universe. I suspect the same is true of XP.
Yet another ill-conceived fad meant to line the pockets of IT
consultants. God save us all!
another link, posted 15 Nov 2000 at 18:09 UTC by apm »
Mats Henricson pointed out that Refactoring also has a website:
XP and Free Software, posted 15 Nov 2000 at 23:51 UTC by avi »
for those that have been reading my diary here, you'll know that i have
a new Free software project starting soon that will be sold commercially
as well. it also happens that i recently read the first XP book, and
found it really interesting - if somewhat lacking in the "how-to" area.
last week i stumbled across the new, unpublished XP book on the web. it
is available here (ftp)
(http). it is an excellent read, even if some of the parts
of the book are in their final draft - so good in fact that we are going
to use XP for our project to see what it is like in practise. each
chapter focuses on a different practical aspect of how to implement XP.
though no development methodology is perfect, i think XP will be able
help us create a better product in a shorter amount of time.
lilo, mrorganic: A lot of people misunderstand the "do the simplest
thing that could possibly work" maxim. It doesn't work unless you also
"refactor mercilessly". The idea is that the design of your software
evolves. You mutate it to add a new feature or fix a bug, then you
refactor to improve the design of what you have. And you do it
mercilessly. The point is not to avoid design. Far from it. Design is
a great thing! So in XP you do it to extremes: you design ALL the
BTW, the subtitle of the refactoring book is
"Improving the design of existing software".
new XP books, posted 16 Nov 2000 at 13:50 UTC by apm »
Both new XP books are now out:
Extreme Programming Installed - Ron Jeffries, Ann Anderson, Chet
Planning Extreme Programming - Kent Beck and Martin Fowler
XP and design, posted 17 Nov 2000 at 01:35 UTC by avi »
Check out Martin Fowler's keynote from the XP2000 conference here.
The speech is entitled "is design dead?", and is quite a good read.
Saw one .signature once that had a nice answer to the Infinity Monkey
"The internet proves otherwise" :-)
by Lisa G. R. Henderson of the Industrial Engineering
Department at Mississippi State University has some interesting ideas
about how requirements analysis might be performed in a free source
Simple?, posted 20 Nov 2000 at 07:51 UTC by sergent »
So I like to think I was doing a number if not all of these things before I had heard of XP. I disagree with the comments made above about design, however.
Simple design does not mean no design.
Simple designs don't preclude creating complex systems; in fact, they facilitate them.
Things that are not designed well end up having interfaces that either not well suited to the problem at hand or are unstable because they are continuously changing since the requirements were not managed. Trying to put things together that don't have interfaces that work together is hard.
Someone--I forget whom--said that UNIX pipes were the only component model that ever worked. I think the underlying statement is that interfaces that are more complicated than UNIX files (forget ioctl and sockets for now; I think these are things that were hacked on later that don't fit the original model, and are a symptom of the problem mentioned above) are too complicated to be successful. I think this is a bit of an exaggeration, but it is not orders of magnitude off.
Reference, posted 19 Oct 2001 at 23:07 UTC by her »