19 Mar 2005 prozac   » (Journeyer)

I wanted to post this as an Article but I chickened out. I wrote this some time ago and had it on my website. My website is no more so I shall archive it here.

Steps to Programming Success

In this document I outline the steps for successfully developing a programming project. By programming project I mean one or more pieces of software that makes up something like a library, an application or a set of utilities, for either a personal computer or an embedded system. By success I mean the result of ending up with bug free, working, quality software.

The intended audience is that of a lead programmer or architect in a small groups of developers, or a programmer who is a member in a group of developers, working at a small software company. As I wrote this open source projects kept coming to mind so I think that my thoughts here will work for both commercial and open source software.

The following steps should be generally useful when starting a new, not too large, programming project. But I outline overall programming practices that should scale quite a bit. Please keep in mind that I do not provide an exact recipe for success, but provide an outline which can be applied to the process of software development.

I have developed these ideas over many years of writing code, creating programs small and large, and interacting with people one-on-one, through e-mail and as a member of various programming teams.

I will not say that what I have written here is the one best way. However, I will say that I have had my share of both failures and successes so that I feel that I am not completely ignorant in these matters. I also have nothing to gain from what I say here -- I am not an author and am doing this solely for my own personal gratification. (This is basically a compilation of notes I have written down in an old notebook over the summer, so do not expect a great piece of literature.)

We start with zero, or, more importantly, before step one.

0. Have a clear goal.

There always must be a clear goal. Without one there can be nothing to succeed in. Success is reaching your goal. You cannot have success without a goal.

Obviously, wistful goals are not goals. A goal of "success" or "fame" or "fortune" is not a goal. A goal of "the best Web application" is unrealistic and unreachable. Goals cannot be vague, they must be specific.

A goal to learn a second spoken language is not unlike a goal to write a computer program -- as long as you want to learn a specific language and write a specific program.

Scratching an itch is just a vulgar way of saying to write something that does not exist (or that you do not have), to accomplish something that you need or want, but the relief of that itch is the goal in that the result is a specific program that you want to use for a specific reason.

For open source software, a project will not succeed by just opening doors and expecting people to come. It takes a clear goal with a clear result as the reward to attract people.

1. Design your design.

The first thing to do once you have a goal is to document a design. As with a goal, specifics are needed. It will not be enough to simply say "fast," "flexible," "versatile," or "massively multiplayer." Software cannot be "powerful" -- it cannot push your car up a hill. Software is limited to manipulating data and computer peripherals.

A design is more than just an idea or a list of ideas. As a filmmaker uses a storyboard, a writer an outline, and a builder a blueprint, a programmer needs a design.

The design should be in proportion to the size of the project. Small programs that can be (and are) written by a single programmer can be designed "on the fly" in the programmer's head as he or she writes the code. Larger programs may need a design from a page or two in a notebook to several documents including many diagrams. The documentation is not directly proportional to size but to complexity. A program that only interacts locally -- keyboard, screen, file system -- may be smaller than the same program doing the same thing interacting with multiple connections over the Internet, but it probably will not require a larger design or more documentation.

The design may also be influenced by hardware or software interfaces such as device drivers. Designing the UI may be sufficient for some programs.

But a design should never be too detailed. Do not over document! If you try to document beyond data structures and APIs, and cover how to code down to for loops vs. while loops or other code aspects, you can wind up documented into a corner, losing the creativity and inspirations of the coders. When a code comes up with a more efficient way of doing something you will be faced with either adhering to inefficient code, having to rewrite the documentation or having inaccurate documentation.

Once you have the basics down the code will follow.

There is no need for a detailed design of the implementation down to the level of each function call before coding can begin. If you have to document how you are going to implement the code you are then essentially writing the code twice -- the "how I am going to implement the code" document and then code itself. You will then be in the situation of having to maintain two code bases, the implementation documentation and the code itself.

So, what is a design? How would one know what is sufficient? Well, perhaps this will be seen as a cop out, but answering those questions will require another essay of itself. Let me at least try to offer some suggestions. A design could range from a list of features the software is to accomplish in the case of a service or to something that would be similar to the software manual in the case of a an application. A design should answer the question, "What does the software do?"

2. Keep a strong lead.

One experienced person needs to be in charge, to oversee things, to have final say in disputes. The design needs to be communal, but the overall control needs to be top down. A strong lead needs to be experienced in knowing that no one will always be right. The lead must know when a change to the design is necessary, and when to let others have their way when they are right.

A leader who demands total control over the design risks alienating the coders, slowly killing a project. A leader that demands that the design strictly follow a textbook, rather than treating textbooks as guidelines, will quickly kill a project.

But too many people all trying to design, and arguing over details, can slow a project down, risking making the project difficult to manage.

Adding a little Cathedral to the Bazaar can be a good thing. One person designing everything, instructing everyone, will probably not work well unless that person is extremely good. How many of us are that good? But one person, keeping the team together, getting the team to design together, delegating, asking for input and ideas, making the "tie-breaking" decisions, will work and work well. How many of us can do that? Perhaps more than we think if they understand the process.

In the Bazaar model, the "community" of developers spread across the Net are not all directly involved in the design; in actuality, one person or a few people (the small "core" group), does most of the work, while fielding suggestions, bug fixes and patches from the rest of the "community" (taking the best solutions, rejecting the rest).

3. Let the coders implement to the design.

Once the design is done, unknowns marked down, and the coders start coding, the team spirit must be maintained but only on the large scale. The individual coders must be able to act and program like individual coders; they must be able to converse and consult among themselves and their peers when making implementation decisions.

A weekly meeting to see how things are going, to see if the work is progressing within the time frame, to see if anyone needs anything, is quite adequate. The leader's role is important, needing to be supportive and not intrusive. A leader working one-on-one with each coder in separate little meetings is a recipe for disaster.

The reason why the design should not go into excessive detail will become apparent during implementation. Very often, as the coding starts, a better idea comes along and a different algorithm takes over an original idea -- this should never be prevented. Trying to justify a new algorithm to an inexperienced manager who had you over-design in the first place is not fun, to say the least.

Programmers have many resources at their disposal: books, magazines, the World Wide Web, Usenet and colleagues. A good programmer will takes advantage of all of these. So there is a little of the Bazaar in the Cathedral too as programmers working in a closed source environment take advantage of everyone else's published works. With today's Web, all "in house" closed source projects are helped indirectly by a world wide community of developers. All free software is available to all.

(In looking at the competing aspects of closed source vs. open source, for closed source projects there can be less of a sustained programmer base for programmers as closed source companies are subject to programmer turnover, as a result there may be little or limited peer review (of code and coding practices).)

4. Relaxation and reflection are important.

Sometimes there is an end to a project, more often so for closed source (when a delivery is made, or a release is published). But while developing one needs to have some sort of closure from time to time -- a period to rest, idle time to think about and clean-up the code, perhaps polishing it a bit here and there. (Remember that algorithm that you know can be more efficient?)

If there is no rest or room for reflection, especially if there is feature creep, chances are that the code will remain in a "just finished" state, or "just good enough," and it will never get cleaned-up, let alone polished. Debugging statements may get left in, test code may be left in but commented out, comments may remain wrong or misleading.

Time needs to be set aside, during development, for relaxing and reflecting on the code -- allowing the coders to clean-up and polish.

I realize that such an R & R process during development will be quite controversial and many people will immediately call me crazy. And I do understand the importance of scheduling and of delivering on time. But there is importance in quality code and in quality time.

If a project has a tight schedule or is running late there will be no time for pauses in the development will be one argument. Another will be that if everything is designed properly there will be no need to re-code anything in the middle of development.

Late projects will always, perhaps, be an exception. But scheduling can easily accommodate a few days a month toward this. Since the entire code base can not be designed fully and must be implemented in sections there may always be room for improvement. And I am certainly not advocating any sort of basic re-design every once in a while -- code clean-up and polishing should consist of comment clean up, the improvement of a conditional branch here or the re-organization of a for-loop there. Along with the ever important process of catching bugs.

As for re-implementing an algorithm based solely on its efficiency, this is where a strong lead and weekly meetings shows its most important role. If, during development, someone thinks of a way of re-implementing something non-trivial, she can bring it to the table where it can be discussed. If re-implementation would effect the schedule there would be no need to carry it out.

This need not be thought of R & R if you do not like that term. Thinking of it as a code review process may help in understanding this. Improving quality and preventing bugs is extremely important. Why leave code review and clean-up to the end of a project?

Afterword

Sometimes I think this is all right on the money, other times... it seems like a bunch of hooey... I don't know. My programming projects are getting smaller and smaller these days; PERL, PHP and a few scripting languages are all I use these days. Well, perhaps there is something good here. If not... Well, no harm done. (I hope. ;-)

Bibliography

The following books have influenced me or I feel that they having something important to say on this subject.

1. The Art of Programming, Donald Knuth

Although the works have not effected much of my experiences of the management process, the three volumes that make up The Art of Programming has had by far the most influence on my coding style.

2. The C Programming Language, Brian Kernighan and Dennis Ritchie

The second most influential book on my coding style.

3. Software Tools, Brian Kernighan and P. J. Plauger

4. The Practice of Programming, Brian Kernighan and Rob Pike

Two more books explaining the basis of quality programming and program development.

5. Literate Programming, Donald Knuth

6. The Cathedral and the Bazaar, Eric Raymond

7. The Psychology of Computer Programming, Gerald Weinberg

8. The Mythical Man-Month, Frederick Brooks

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!