30 Mar 2011 idcmp   » (Journeyer)

Technical Debt

Our build came up recently in a planning session at work. It's a standard big ball of mud, with hidden knowledge all over. It's frail, inconsistent and unhelpful, it's complicated to work with and almost nobody ever gets their local build past "good enough for now" to "working well".

This has been an issue for a while and it hadn't been given the proper attention to be improved. Every change had been reactionary to various needs and it had changed hands so many times that any beliefs or design philosophies it had were long lost.


One developer estimated that wrangling the build system alone accounted for 30% of their development time (that's three days of a two week iteration).

When asked how much effort was required to fix it, the answer was "You can't."

Taking Out Your Technical Loan

While I wasn't originally, I'm now a fan of the term "technical debt". It brings with it a handful of related concepts and terminology that make discussions easier. In this case, compound interest, and bankruptcy.

It all starts when you first make a big investment; probably your first release. Fixing up the last minute bugs before release, cutting corners in the name of getting to market, making "just for now" trade offs going against technical soundness. Each of these things add to your technical debt. There's nothing wrong with this, it's expected.

But then your debt starts adding up pretty quickly:
  • Changes in business needs for the system
  • Changes in company direction
  • Changes in team composition, staff sizing
  • Cutting corners on in-code documentation
  • Industry changes
  • Not keeping up with underlying frameworks
  • Not following proper language techniques
  • Quick security patches
  • Scalability needs
  • Blindly meeting code metrics
  • Over reliance on under-skilled developers
  • Not testing on up to date integration points (databases, operating systems, etc)
  • Quick and dirty bug fixes
  • Minor improvements
Any of these things can put you further into debt, and while you're in debt you'll need to pay compound interest on your technical debt. What was one day of work to fix this iteration can become two days of work a few iterations from now and a week of work next release. Leave it longer and suddenly it's a month of work two years from now.

How does this happen?

Lots of ways.

One of the beautiful things about refactoring is the way it takes into consideration the "context" that a developer has gained in immediately working with the code. Their short-term memory is filled with nuances and intents and allows them to glide smoothly through code. They can envision the refactoring and perform it optimally.

As time passes, they lose some of the context. Surrounding code begins to change, bug fixes are introduced, and other developers build on what was written instead of what was intended. More time passes and other code is refactored, team members come and go or the pattern gets copied elsewhere in your code base.

A few iterations later, there's a hierarchy of dependencies built on this code and what was once a simple refactoring has become a costly code spill cleanup effort.

Eventually the frameworks become out of date and coding styles fall out of style. Instead of wanting to clean up the code, new developers will work around it, introduce a functional duplicate, wrap it or simply make their changes as a series of hacks.

If code quality drops too low, some say morale will follow and keeping more than a handful of experienced developers will become a challenge; attracting developers eager to land their first job, negative velocity developers or those that simply don't care.

This affects your ability to pay off your debt.

Ways To Pay

Every organization has different priorities, goals and principles. As such, calculating how or where to pay for your technical debt varies wildly. Of course, the rule of thumb is "Pay off debt that is costing you the most." There are a handful of ways to do this:
  • Research and Analysis: Work to establish what business functionality should be delivered next then prototype any areas that are known hot spots or developers do not have experience with. With the gained experience determine if the code in question is deeply in debt or if it can afford the new feature.
  • Raise Rates: When asked to estimate a new feature in an area that has high technical debt, focus on estimates which include needed code revitalization. Just as it's the business duty to plan features customers want, it's a developer's responsibility manage risk ensuring it's done technically correct.
  • Foreign Exchange: Exchange the technical currency into salaries as an incentive to keep developers. Many larger, non-technical firms follow this route. It's like buying carbon credits; the debt is still there, but developers are rewarded for tolerating it.
  • Creative Accounting: Buffer estimates and deliverables with needed cleanups. Once someone knows an area of code well, perform small improvements under other ledgers.
  • Get a Second Job: One of the most dire methods is to band together and either work nights or through lunchtime to make the needed improvements.
  • Bankruptcy: Sometimes there's honestly no other choice other than to start over.
Back To The Build

Our build system had gone bankrupt. It had gained such complexity that we could no longer service the interest on the debt.

In our case, this opened the door to switching build tools, cleaning up packaging and performing much needed work on the development environment. It's exciting, as it will not only boost the productivity of each developer, but it hopes to also be something that isn't always dreaded; so developers can focus on solving their problems more effectively and not how to make sure everything builds properly.

And, if the payments are made on it on a regular basis, serve our needs well for years to come.




Syndicated 2011-03-30 04:07:00 (Updated 2011-04-07 23:08:43) from Idcmp

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!