I wanted to post this as an Article but I chickened out. I wrote this some time ago and had it on my website. My website is no more so I shall archive it here.
Steps to Programming Success
In this document I outline the steps for successfully developing a programming
project. By programming project I mean one or more pieces of software that makes
up something like a library, an application or a set of
utilities, for either a personal computer or an embedded system. By success
I mean the result of ending up with bug free, working, quality software.
The intended audience is that of a lead programmer or architect in a small
groups of developers, or a programmer who is a member in a group of developers,
working at a small software company. As I wrote this open source projects kept
coming to mind so I think that my thoughts here will work for both commercial
and open source software.
The following steps should be generally useful when starting a new, not too
large, programming project. But I outline overall programming practices that
should scale quite a bit. Please keep in mind that I do not provide an exact
recipe for success, but provide an outline which can be applied to the process
of software development.
I have developed these ideas over many years of writing code, creating programs
small and large, and interacting with people one-on-one, through e-mail and as
a member of various programming teams.
I will not say that what I have written here is the one best way. However,
I will say that I have had my share of both failures and successes so that
I feel that I am not completely ignorant in these matters. I also have nothing
to gain from what I say here -- I am not an author and am doing this solely
for my own personal gratification. (This is basically a compilation of notes
I have written down in an old notebook over the summer, so do not expect a
great piece of literature.)
We start with zero, or, more importantly, before step one.
0. Have a clear goal.
There always must be a clear goal. Without one there can be nothing to
succeed in. Success is reaching your goal. You cannot have success without
a goal.
Obviously, wistful goals are not goals. A goal of "success" or "fame" or
"fortune" is not a goal. A goal of "the best Web application" is unrealistic
and unreachable. Goals cannot be vague, they must be specific.
A goal to learn a second spoken language is not unlike a goal to write
a computer program -- as long as you want to learn a specific language and
write a specific program.
Scratching an itch is just a vulgar way of saying to write something that
does not exist (or that you do not have), to accomplish something that you
need or want, but the relief of that itch is the goal in that the result is
a specific program that you want to use for a specific reason.
For open source software, a project will not succeed by just opening doors and
expecting people to come. It takes a clear goal with a clear result as the
reward to attract people.
1. Design your design.
The first thing to do once you have a goal is to document a design. As
with a goal, specifics are needed. It will not be enough to simply
say "fast," "flexible," "versatile," or "massively multiplayer." Software
cannot be "powerful" -- it cannot push your car up a hill. Software is
limited to manipulating data and computer peripherals.
A design is more than just an idea or a list of ideas. As a filmmaker uses a
storyboard, a writer an outline, and a builder a blueprint, a programmer needs
a design.
The design should be in proportion to the size of the project. Small
programs that can be (and are) written by a single programmer can be
designed "on the fly" in the programmer's head as he or she writes
the code. Larger programs may need a design from a page or two in a notebook
to several documents including many diagrams. The documentation is not
directly proportional to size but to complexity. A program that only interacts
locally -- keyboard, screen, file system -- may be smaller than the same
program doing the same thing interacting with multiple connections
over the Internet, but it probably will not require a larger design or
more documentation.
The design may also be influenced by hardware or software interfaces such as
device drivers. Designing the UI may be sufficient for some programs.
But a design should never be too detailed. Do not over document! If you
try to document beyond data structures and APIs, and cover how to code
down to for loops vs. while loops or other code aspects, you can wind
up documented into a corner, losing the creativity and inspirations of
the coders. When a code comes up with a more efficient way of doing
something you will be faced with either adhering to inefficient code,
having to rewrite the documentation or having inaccurate documentation.
Once you have the basics down the code will follow.
There is no need for a detailed design of the implementation down to the
level of each function call before coding can begin. If you
have to document how you are going to implement the code you are then
essentially writing the
code twice -- the "how I am going to implement the code" document and
then code itself. You will then be in the situation of having to
maintain two code bases, the implementation documentation and the
code itself.
So, what is a design? How would one know what is sufficient? Well, perhaps
this will be seen as a cop out, but answering those questions will require
another essay of itself. Let me at least try to offer some suggestions. A
design could range from a list of features the software is to accomplish in
the case of a service or to something that would be similar to the software
manual in the case of a an application. A design should answer the question,
"What does the software do?"
2. Keep a strong lead.
One experienced person needs to be in charge, to oversee things, to have
final say in disputes. The design needs to be communal, but the overall
control needs to be top down. A strong lead needs to be experienced in knowing
that no one will always be right. The lead must know when a change to the design
is necessary, and when to let others have their way when they are right.
A leader who demands total control over the design risks alienating the
coders, slowly killing a project. A leader that demands that the design
strictly follow a textbook, rather than treating textbooks as guidelines,
will quickly kill a project.
But too many people all trying to design, and arguing over details, can slow
a project down, risking making the project difficult to manage.
Adding a little Cathedral to the Bazaar can be a good thing. One person
designing everything, instructing everyone, will probably not work well
unless that person is extremely good. How many of us are that good? But one
person, keeping the team together, getting the team to design together,
delegating, asking for input and ideas, making the "tie-breaking" decisions,
will work and work well. How many of us can do that? Perhaps more than we
think if they understand the process.
In the Bazaar model, the "community" of developers spread across the Net are
not all directly involved in the design; in actuality, one person or a few
people (the small "core" group), does most of the work, while fielding
suggestions, bug fixes and patches from the rest of the "community" (taking
the best solutions, rejecting the rest).
3. Let the coders implement to the design.
Once the design is done, unknowns marked down, and the coders start coding,
the team spirit must be maintained but only on the large scale. The
individual coders must be able to act and program like individual coders;
they must be able to converse and consult among themselves and their peers
when making implementation decisions.
A weekly meeting to see how things are going, to see if the work is
progressing within the time frame, to see if anyone needs anything, is
quite adequate. The leader's role is important, needing to be supportive and
not intrusive. A leader working one-on-one with each coder in separate little
meetings is a recipe for disaster.
The reason why the design should not go into excessive detail will become
apparent during implementation. Very often, as the coding starts, a better
idea comes along and a different algorithm takes over an original idea --
this should never be prevented. Trying to justify a new algorithm to an
inexperienced manager who had you over-design in the first place is not
fun, to say the least.
Programmers have many resources at their disposal: books, magazines, the
World Wide Web, Usenet and colleagues. A good programmer will takes advantage
of all of these. So there is a little of the Bazaar in the Cathedral too as
programmers working in a closed source environment take advantage of
everyone else's published works. With today's Web, all "in house" closed
source projects are helped indirectly by a world wide community of developers.
All free software is available to all.
(In looking at the competing aspects of closed source vs. open source, for
closed source projects there can be less of a sustained programmer base for
programmers as closed source companies are subject to programmer turnover, as
a result there may be little or limited peer review (of code and coding
practices).)
4. Relaxation and reflection are important.
Sometimes there is an end to a project, more often so for closed source
(when a delivery is made, or a release is published). But while
developing one needs to have some sort of closure from time to time
-- a period to rest, idle time to think about and clean-up the code,
perhaps polishing it a bit here and there. (Remember that algorithm that
you know can be more efficient?)
If there is no rest or room for reflection, especially if there is
feature creep, chances are that the code will remain in a "just
finished" state, or "just good enough," and it will never get
cleaned-up, let alone polished. Debugging statements may get left
in, test code may be left in but commented out, comments may remain
wrong or misleading.
Time needs to be set aside, during development, for relaxing and
reflecting on the code -- allowing the coders to clean-up and
polish.
I realize that such an R & R process during development will be quite
controversial and many people will immediately call me crazy. And
I do understand the importance of scheduling and of delivering on
time. But there is importance in quality code and in quality time.
If a project has a tight schedule or is running late there
will be no time for pauses in the development will be one argument.
Another will be that if everything is designed properly there
will be no need to re-code anything in the middle of development.
Late projects will always, perhaps, be an exception. But scheduling
can easily accommodate a few days a month toward this. Since the
entire code base can not be designed fully and must be implemented
in sections there may always be room for improvement. And I am
certainly not advocating any sort of basic re-design every once in
a while -- code clean-up and polishing should consist of comment
clean up, the improvement of a conditional branch here or the
re-organization of a for-loop there. Along with the ever important
process of catching bugs.
As for re-implementing an algorithm based solely on its efficiency,
this is where a strong lead and weekly meetings shows its most important
role. If, during development, someone thinks of a way of re-implementing
something non-trivial, she can bring it to the table where it can be
discussed. If re-implementation would effect the schedule there would
be no need to carry it out.
This need not be thought of R & R if you do not like that term.
Thinking of it as a code review process may help in understanding
this. Improving quality and preventing bugs is extremely important.
Why leave code review and clean-up to the end of a project?
Afterword
Sometimes I think this is all right on the money, other times... it
seems like a bunch of hooey... I don't know. My programming projects
are getting smaller and smaller these days; PERL, PHP and a few
scripting languages are all I use these days. Well, perhaps there is something good here. If not... Well, no harm done. (I hope. ;-)
Bibliography
The following books have influenced me or I feel that they having something
important to say on this subject.
1. The Art of Programming, Donald Knuth
Although the works have not effected much of my experiences of the management
process, the three volumes that make up The Art of Programming has had by far
the most influence on my coding style.
2. The C Programming Language, Brian Kernighan and Dennis Ritchie
The second most influential book on my coding style.
3. Software Tools, Brian Kernighan and P. J. Plauger
4. The Practice of Programming, Brian Kernighan and Rob Pike
Two more books explaining the basis of quality programming and program
development.
5. Literate Programming, Donald Knuth
6. The Cathedral and the Bazaar, Eric Raymond
7. The Psychology of Computer Programming, Gerald Weinberg
8. The Mythical Man-Month, Frederick Brooks