Watch Your Amount of Technical Debt

With the ongoing catastrophes in the financial and credit markets of late, this is a perhaps a good time to review the concept/metaphor of technical debt; particularly when it appears increasingly likely that IT budgets going into at least next year will hold steady or, in other situations, get cut back. First, some history on how this concept came about.

Ward Cunningham proposed the concept of technical debt back in 1992. As he described in a paper delivered to OOPSLA that year:

"...We believe this process leads to the most appropriate product in the shortest possible time.  There are pitfalls, however.  Not everyone is comfortable with sudden changes in direction, especially programmers. Here the best antidote is more complete familiarity with the product and its implementation.  Changes will be accepted when they are recognized as do-able.

Another, more serious pitfall is the failure to consolidate.  Although immature code may work fine and be completely acceptable to the customer,  excess quantities will make a program unmasterable, leading to extreme specialization of programmers and finally an inflexible product. Shipping first time code is like going into debt.  A little debt speeds development so long as it is paid back promptly with a rewrite.  Objects make the cost of this transaction tolerable.  The danger occurs when the debt is not repaid.  Every minute spent on not-quite-right code counts as interest on that debt.  Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object- oriented or otherwise."

An additional point-of-view was offered by Martin Fowler in 2003:

"...You have a piece of functionality that you need to add to your system. You see two ways to do it, one is quick to do but is messy - you are sure that it will make further changes harder in the future. The other results in a cleaner design, but will take longer to put in place.

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments."

Given the current economy and its impact - now and on-going, on IT initiatives and projects, it is vital that we keep this concept in mind as we plan, design, and execute them. Martin mentions the 'extra effort' required to refactor because of quick-and-dirty choices made initially. What that also means in some cases is increased costs and time-to-market, and neither of these are very palatable to IT execs and project sponsors who are under increasingly shrill budget and time pressures.

It is also interesting to note that this metaphor applies no matter what software development methodology is employed. If, for example, an agile project requires a number of iterations to refactor codebases without adding additional business functionality, the technical debt incurred in the previous iterations is being addressed and 'paid down' - usually with additional budget and time, but nonetheless paid off.

Is technical debt bad and to be avoided? The answer I usually give is no, as long as its: a) managed properly; and b) generally 'paid down' in a time/cost sensitive manner as projects progress to completion. All projects incur some form of technical debt as they proceed. The real problem with technical debt is when it, like subprime mortgage loans and collateralized debt obligations, becomes toxic - toxic to the business, the development teams, the IT organization, etc. Here are some forms of technical debt that can get progressively toxic over time:

• Code complexity increases to the point that modifications cannot be made without introducing significant defects, and the time/cost to refactor means that the effort is essentially a complete redesign/re-write.
• The code does the 80% of functionality that was 'nice to have,' but non-critical, fairly well and the 20% of critical functionality poorly.
• Architecture and design activities up-front are given short-shrift, and development teams 'go their own way' in developing systems - the amount of technical debt incurred can get particularly gruesome in situations where multiple development teams are engaged to build large systems without enough architectural and design coordination between the teams.
• Situations where business analysts, product owners, sponsors, or major stakeholders will not (or cannot) decide on major functionality issues, leaving the development teams to guess and proceed without clear motives or direction. This happens more often than we care to admit.

There are lots of valid reasons for incurring technical debt, and as long as the debts do not become toxic, and are 'paid down' properly, the projects incurring them should proceed well. However, given the current situation with the economy, and the reaction to it in most of our workplaces, it behooves us as IT and project management professionals to not only keep this concept front-and-center during our projects, but to deal appropriately with it so it doesn't get out of hand and make our organizations look like the AIG or Lehman Brothers of IT.

Comments

[Ted Simpson]

Thank you for this post. Current trends in technical debt do mirror those in financial debt. So many executives seem so willing to risk so much on complexities they don't care to fully understand. Thanks again, this is a very important concept to bring to light.

[Evelyn Guzman]

I like the parallel you described between technical and financial debt. I could not have understood it without your clear explanation. They are alike in ways I never thought of like having to finish off a project using technical shortcuts or something to take advantage of some market opportunity the same way one would go and borrow funds to cover some transaction that is beneficial only at that moment.

Evelyn Guzman
Debt Challenger