Is technical debt management problem more of a culture issue or insight issue

Question

Disclaimer: I don't expect zero tech debt. In this post, technical debt problem refers to severity that has been causing negative impact, say productivity.

Recently I was thinking to build a tool to automatically generate tech debt report from issue tracker - introduction rate vs cleanup rate over the time. Apart from the total, there'll also be numbers broken down by project team and by manager, so that managers could easily get insight on current tech debt level, without delving into issue tracker and details (such tool might already exists, I need to research to avoid reinventing wheel).

Motivation wise, tech debts have been snowballing for years. Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed. I hope the periodic report will help to improve tech debt management issue.

However, on second thought, I wonder will increasing visibility of tech debt level really helps to raise priority. Generally, is tech debt issue an org culture issue or just lack of tool/insight? I supposed there's no universal answer, I wonder which is the more common cause. What's your experience?

--- Update 2/28

Clarification: I believe most management are intelligent enough to realise there's impact, especially after teammates reported pain in terms of project productivity. My gut feeling is that, they don't have a concrete picture about how serious problem is. My idea is to help management to gain clearer picture, via two steps:

Have techdebts logged, and have their impact tracked. (there are challenges, but that's beyond scope of this question)
Have a report for introduction rate vs cleanup rate (there could be further breakdown by high/low impact).

My curiosity comes from, will these efforts help or are they just waste of time, generally speaking (not specific within my org) - hence the question what's your experience. If it's org culture issue, then most likely these efforts won't help much.

Instead of analysing the bug tracker, analyse the source code repository. You can see how complex the software is, how often the complex parts are being edited, and how many and who are across the contents of each file. Place that in some pretty graphics and you can highlight why 5 years ago code change was so much easier than now. — Kain0_0, Feb 26 '20 at 05:10
Good idea. We do have static analyser scanning repo as part of build. Once false positive rate improved to acceptable level, it's worthy to find a way to integrate that result into report. Doubts: (1) The primary concern is about more complex debt such as design-implementation gap/shortcut implementation. To auto detect from repo, don't we have to manually document such debts as TODO and check-in? (2) Do code analysers support custom scripting? We need to be able to correlate offended code with developer and their team/manager, as the same product is co-owned by multiple managers. — shiouming, Feb 26 '20 at 05:40
This seems to assume that the technical debt is known and recorded - but its usually like an iceberg, 90% of it is out of sight and only becomes known when it causes a problem. — afaulconbridge, Feb 26 '20 at 14:41
"tech debt clean up, more often they will be asked to remove those numbers from estimate"
That is a red flag right off the bat. It's a culture issue. It's a trust issue. It's a management and leadership issue. — Brandon, Feb 26 '20 at 18:17
IMO, a software developer owns their estimate. Stick to your estimate. Do things correctly. If someone bullies you into cutting corners and/or lowering your estimate, say no.
The only person you can't say no to is your boss. If you get to a point where your boss is demanding you do bad work, then it's time for a new job. — Brandon, Feb 26 '20 at 18:23
Cleanup rates can be gamed. Google has over 50,000 improperly closed bug reports in Android alone. — StackOverthrow, Feb 26 '20 at 20:41
My rep is too low for an answer but I feel that flaters answer, good as it is, is too verbose to really bring the crux of the matter home: Management loves cowboys The more you prioritize short term gains over long term viability, the more you will be rewarded. That senior dev that writes spaghetti code and dumps the fallout on your desk? He's gonna get a promotion, cause he "understands the buisness". — Douwe, Feb 27 '20 at 10:32
@afaulconbridge: Extending your iceberg example, I don't need to see the other 90% of the iceberg to know it's there, even though I don't know its exact shape. Similarly, you're correct that the majority of the problematic situation isn't known about, but its existence can be inferred based on experience and being aware of the other 10%. Planning wise, management doesn't care about the shape of the iceberg even if they were fully willing to address it. They care about the existence and approximate size. — Flater, Feb 27 '20 at 10:45
Unless legacy software costs you money every year - interest - it's not technical debt, it's just legacy software. The reason to pay tech debt down is to reduce ongoing and future costs, nothing else. Software doesn't rust. — Ben, Feb 28 '20 at 09:10
@afaulconbridge in a lot of cases it is already enough to track the underlying semantic/business logic technical debt, e.g. needing to support some legacy protocol in this and that module, which a) explains why a huge iceberg of technical debt might be in the code and b) can act as the trigger to clean up that part once the requirement is no longer valid. Without such tracking indeed all of that iceberg becomes invisible until one looks at that module and drowns. — Frank Hopkins, Feb 28 '20 at 14:24
The problem with refactoring technical debt is that what qualifies as technical debt is often in the eye of the beholder. One developer sees a perfect class, the other sees a design issue. And refactoring itself means that they functionality that touches that code should be completely re-qaed. A lot of technical debt expeditions end in bugs going into production because not enough testing and qa occurred, — Mark Rogers, Feb 28 '20 at 17:08
@Kain0_0 That's a pretty good idea; Source and revision complexity does sound like it would be a great metric for tech-debt. That's certainly the case in what I've experienced as well, when you're having to make complex revisions in complex files or methods then maybe it's time for a some housekeeping.. — xtratic, Feb 28 '20 at 19:39
@xtratic I can't claim the idea as mine, just can't remember where I got it from. I think it was the Code as a Crime Scene idea I saw a few years ago. But anyway yes, it aligns very well with anecdotal experience. Which leads me to the opinion that all code is Technical Debt. The difference is that some of it charges exorbitant interest and transaction fees, while the rest of it is quite happy to sit there for pennies every few years. — Kain0_0, Mar 01 '20 at 23:55

Flater · Accepted Answer · 2020-02-29T20:29:53.850

101

Anecdotally

I am a consultant developer. I have been hired on several occasions specifically to "fix the development issues". Some customers are aware of issues in their development process, whereas others only see it as a bunch of bugs that need to be fixed without looking at the cause of the bugs (i.e. bad coding practices).

In my experience, one company that asked for help in fixing the development process was actually interested in taking steps to improve the development process. In other companies, their interest existed up until it required action from their side (e.g. reprimanding a developer who actively rolls back refactoring or improvements, or actually giving me access to the tools needed to set up a CI/CD pipeline).

Based on my experience, bad practice starts off as a developer deficiency. Not a willful one, but rather a matter of either inexperience or general corner-cutting attitude. Whatever the cause, these developers will show quick results due to not taking the time for due diligence such as testing, reviewing or refactoring.

Management will notice those quick results, and will over time come to expect this efficiency. They don't handle the fallout from bad practice (i.e. bugs) directly, but they do benefit from the shorter development times.

At this point, it becomes a feedback loop. Management communicates expected (quick) deadline. Developers are forced to cut corners to achieve it. The codebase degrades. The initial quick release turns into a maintenance cycle of unclear and erratic bugs, regressions, and general lack of readability. In order to cope, while keeping up with the continuing demand for quick results, developers are forced to cut corners in their bugfixes.

The cycle continues, while the quality and performance of the codebase is eroded, and also the good practice skills of developers erode and start being regarded as "needlessly" time consuming. If some developers stick to good practice and others don't, management will judge them based on how quickly they deliver - without observing the bugs or the causes of the bugs.
The good practice developers are deincentivized, the bad practice developers are incentivized. Over time, due to positive/negative feedback from management, the bad practice developers take on a more leading role than the good practice developers, and the bad practice becomes the law of the land.

Speaking from the experience of a company whose main workforce was external consultants, the good practice devs simply leave or become disenfranchised bad practice devs. The (initial) bad practice devs stick around. This perpetuates the imbalance of the bad practice devs having seniority over the good practice devs.

At this point, bad practice has become an endemic company culture. It is reinforced from all sides (including the sales department in case of dev companies), and any good practice suggestion that pops up is often drowned out by the popular support for bad practice, combined with management's intolerance for longer deadlines.

This devolution is something I've observed with at least three different companies. The same events and general work climate pervaded through all three companies.

The monkeys and the ladder

Whenever I talk about detrimental company culture, which often manifests as a "this is how we've always done it" attitude, I am reminded of the parable of the monkeys and the ladder.

Suppose I had turned off the shower around the time of picture 4. The monkeys could have gone up that ladder without any repercussions, but their "company culture" prevented it from happening, based on what is now an outdated idea (since the shower is no longer active).

This parable touches exactly on the erosion of good practice that takes place. Popular but misguided support for bad practice inhibits anyone who tries to make a change for the better by introducing good practice.

The issue isn't with social checks and balances. The same principle is used in other companies to keep up the good practice and squash any bad practice suggestions.

The issue is with the blind acceptance of "things are done this way" without ever being able to re-evaluate. When it reaches that stage, the behavior is a company culture.

Answering your questions

Generally, is tech debt issue an org culture issue or just lack of tool/insight?

It depends what stage of the process you are on. In the beginning, it's a lack of insight and/or tooling. But when combined with management that looks only at results and not ongoing issues and thus wrongly (possibly unknowingly) incentivizes the bad practice, it becomes a feedback loop and over time turns into company culture.

edited Feb 29 '20 at 20:29

answered Feb 26 '20 at 12:10

Flater

49,580

"Not a willful one, but rather a matter of either inexperience or general corner-cutting attitude" - isn't corner-cutting always willfully done? – iruvar Feb 26 '20 at 17:44
18

Ironically to the "blind acceptance" moral, the five monkeys experiment is completely made up – BlueRaja - Danny Pflughoeft Feb 26 '20 at 18:38
1

I'm not sure whether the picture of incompetent developers and ignorant managers rings true in my experience. Typically, managers and directors are not ignorant but positively short-termist - their whole business may be based on undercutting the market by doing things badly and following commercially unsustainable practices. When they hire a consultant to fix things, they haven't seen the light, they're pursuing what seems to be another quick short-term fix. If the consultant advises them to mend their short-termist ways and buckle down for a long slog, then his feet won't touch the floor. – Steve Feb 26 '20 at 19:21
@Steve Your experience hints at the fact that technical debt isn't always bad. Sometimes it makes sense to assume technical debt in order to be first to market. Bad technical debt is accumulated unintentionally, or, as seems to be your experience, intentionally but with no intention of ever paying it. – StackOverthrow Feb 26 '20 at 20:46
The same cycle can start as a management deficiency: lowball salaries attract those inexperienced or corner-cutting developers, and interviewers don't know how to evaluate them. – StackOverthrow Feb 26 '20 at 20:51
1

@user560822, my point is that often not only is it intentional that debt has been accumulated thus far, but so also to accumulate it further. The business that brings in a consultant or a new hire, is not necessarily doing so because they've acknowledged any fundamental error which requires a fresh approach or perspective, but as a way of continuing their existing ways by adding more minions. The chemical company who profit by dumping waste, say, do not hire more diggers for those diggers to tell them to stop dumping, they hire them to better ensure the dumping continues. – Steve Feb 26 '20 at 22:12
What is the end game? Have you seen it? A rewrite and then the cycle starts over again (after a lost 3 years where the old system still had be maintained in parrallel)? – Peter Mortensen Feb 27 '20 at 02:14
3

Well said. The short explanation is that fixing technical debt (and thus preventing future bugs) has no business value while fixing immediate bugs has. So basically improving a software product is destroying business. Also see Iron Gremlin's excellent answer. We should never talk about TD in public, it is just awkward to outsiders and a very hard sale. Labeling an issue as TD in Jira is the most certain way to send it straight to the bottom of the backlog. Split it up and distribute it among popular issues instead and claim the time you need. – Martin Maat Feb 27 '20 at 07:02
@Peter, rewrites usually don't solve the problem, as it's the same developers following the same bad practices, with a maze of poorly understood features. Either they gradually wise up with experience, or the company becomes uncompetitive. – Robin Bennett Feb 27 '20 at 10:02
@iruvar: By "willful deficiency", I mean someone who knowingly delivers a bad result. Corner-cutting attitude is generally willful behavior, but doesn't inherently entail knowing that what you're doing is bad, e.g. someone who genuinely favors speed over quality. One example would be a developer whose first job experienced was in a bad practice company culture, who now applies the same attitude at a new company, but doesn't know it's a detrimental approach to development. – Flater Feb 27 '20 at 10:25
@Steve: Following your experience, which definitely exists as well, it's not unreasonable to think that such a manager would play the role of someone willing to work at fixing things just to incentivize the consultant to put in the effort to fix things. At that point, as a consultant, it is impossible to conclusively say whether any given customer is genuinely willing (up until the point they're not) or simply claiming to be willing; which means that it's hard to separate your case from mine, from the POV of the consultant. – Flater Feb 27 '20 at 10:29
@PeterMortensen: The only sustainable solution I've found is to get out. I am unsure if there is a way to stay in it and keep your own career quality/sanity; and I'm not willing to risk it to try. This may be a biased answer that can only be given by someone who hasn't stayed in it indefinitely; but it's possible that those who do simply give up on good practice and instead play to the incentivization game that their (indefinite) employer has set up for them - for as long as they're able to be a company with employees. – Flater Feb 27 '20 at 10:33
1

Related: https://skeptics.stackexchange.com/q/6828/2974. – TRiG Feb 27 '20 at 10:33
7

There is one insight missing from your answer: Corner-cutting is not always wrong! Sometimes high code quality (== low long term maintenance cost) is important. Sometimes quick delivery (low cost and short time to market) is a higher priority. Knowing which and communicating this to development is the managements responsibility. – Guran Feb 27 '20 at 11:48
@Guran: Yes and no. I agree with you in some parts. I cut corners all the time when banging out a small console app for my own shortlived purpose. However, corner cutting inherently leads to technical debt, as technical debt is effectively defined as "all the corners we've cut". Therefore, in a situation where you observe technical debt and label it as bad (which the question does), you are inherently in a situation where corner-cutting is wrong. – Flater Feb 27 '20 at 11:55
1

@Flater That is true. OTOH, I've seen plenty of dev teams that cannot distinguish TD that actually hurts business from TD that merely is annoying and hurts our professional pride. The reason is always poor communication between dev and mgmnt. – Guran Feb 27 '20 at 12:15
I understand the need of having clean code. But there has to be a balance between trying to clear the TD and delivering features to the business. Eg In my department, I have seen people writing amazingly clean code with close to zero TD but we take endless time, we dont deliver any new features. We work on a really small refactoring for a couple of weeks(that too in pairing mode). Now imaging the time and resource being wasted from business point of view. So the main focus should be on delivering features and side by side, we should keep working on TD. – shoan Feb 28 '20 at 12:55
3

@shoan: As with all things, balances indeed need to be struck. However, it is often forgotten (in bad practice environments) that clean coding speeds up the delivery process over the long-term. The usual flawed reasoning pro-bad-practice (or should I say anti-more-good-practice) trades away long term benefits (maintainability and future efficiency) for short term benefits (getting to the next deadline). There are certainly cases of excessive good practice, but anecdotally I see many more cases of reasonable good practice unfairly being painted as if it were excessive. – Flater Feb 28 '20 at 13:04
2

Just pitching my 2¢ to point out I agree with this answer 100%, and that it isn't specific to software engineering. I have seen this in busy Accident & Emergency departments in hospital contexts, for instance, where there is a pressure for each doctor to "see as many patients as possible". If one has seen 10 patients in their shift, and another 3, the second doctor is criticized, while the first one praised, even if the latter only had time to see 3 because they were dealing with 'technical debt' from the first person cutting corners and putting the patient at risk. – Tasos Papastylianou Feb 28 '20 at 14:01
1

@TasosPapastylianou: I think we can boil down the underlying issue (across professional fields) to there being a biased observation (e.g. mgmt sees benefits but not drawbacks), which leads to a misguided interpretation of what actually contributes positively, which in turn leads to a misguided incentivization structure towards employees, which in turn perpetuates the flaws that were misguidedly observed as being positive for the business. – Flater Feb 28 '20 at 14:10
@BlueRaja - Danny Pflughoeft More ironically your avatar is a monkey so I am not so sure about the reliability of your claim. I want to believe that story dammit! – Martin Maat Feb 28 '20 at 18:59
Very good reply! You really got the point! – bernie Mar 04 '20 at 08:55
This doesn't correspond to my experience as a consultant. It is sometimes true, probably more so in larger organizations. "Technical debt" is a poor term, as it implies the inevitability of complexity. Looking only at the code base, complexity is the true problem, although this can take many forms. The fundamental issue is a lack of good mentoring, and a proliferation of poor mentoring. Many teachers/mentors promote cargo cult or cookie cutter design approaches, and finding one focusing on simplicity is difficult. The window of opportunity to teach code reduction habits is short. – Frank Hileman Mar 25 '20 at 17:40
@FrankHileman Technical debt does not automatically imply that "we must make it more complex to improve it". Technical debt also includes refactoring which lowers complexity but takes time, time which is not available (or invested) at that point. It's also not just a cause of poor mentoring - some technical debt is knowingly taken on for ulterior motives (e.g. reaching an important deadline, dealing with unexpected developer unavailability, ...). Yes, bad mentors exist, and yes, some debt is created through slapdash simplistic code; but that is a subset of what technical debt can entail. – Flater Mar 25 '20 at 23:17
@Flater I don't think I understand you. Refactoring increases the technical debt? When examining the code externally, (as a consultant) outside of any other knowledge, the only obvious "debt" is complexity. If complexity is reduced, there is less work to be done in the future -- unless the complexity was essential, and in this case, why would it be considered "debt"? You might consider "bugs" to be "debt", but the user would simply call those "errors". I would not call a bug backlog, "technical debt". – Frank Hileman Mar 25 '20 at 23:42
@FrankHileman: Technical debt comes both in the shape of overengineering and underengineering. You seem to equate "complexity" with "bad coding practices" (which generally comes in the form of needless complexity, but not exclusively so), but inexperienced developers label good practices (that they do not yet understand fully) as overly complex - which makes for a very ambiguous definition of "complexity". A bug backlog is part of (or caused by) the technical debt when the bugs are a consequence of the over/underengineered code that is in need of refactoring. – Flater Mar 25 '20 at 23:53
@FrankHileman: As a simple example, from a junior developer's perspective, it makes sense to consider implementing SOLID a matter of adding complexity, because abstracted logic is more difficult to grasp than concrete logic. That doesn't mean that it's a bad thing to do, but it is complex hence why an innate understanding of clean coding practices is the main distinguishing feature between junior and senior developers. Understanding that clean coding is a necessary complexity (compared to a simpler bad practice codebase) is something that comes with advanced experience. – Flater Mar 25 '20 at 23:57

Guran · Answer 2 · 2020-02-28T09:38:51.930

23

Usually, it is neither. Usually it is mainly a problem of communication.

What many fail to realize is that technical debt is not a big problem for a company, precicely like financial debt is not.

Interest, on the other hand, is. It would be irresponsible to make unneccessary installments on a loan with very low or zero interest.

When you talk about reducing technical debt, what you are actually asking to do is installments och that debt. Since management is unaware of the interest (ie cost) of that debt, this is not viewed as important.

What you should do is to highlight, not the work required to reduce technical debt, but the extra work required to deliver new features (anr/or fix bugs) due to this debt.

So (to take some easy examples):

Do not ask permission to set up a CI/CR pipeline. Do specify that every release takes an extra half hour per environment. Do not ask for automate testing. Do specify that you use two days of manual testing for things that could have been automated. Do not ask for time to refactor. Do specify those extra hours you spent searching through poorly structured code.

This way you raise issues that actually matter to management. And if management is smart they can even decide if it is actually worth fixing the problems or not. (Why fix tech debt in a product that won't be supported next year)

edited Feb 28 '20 at 09:38

answered Feb 27 '20 at 12:10

Guran

545

2

agree. 99% of "tech debt" is just "code not following the latest trend" You have to focus on idenitfing and solving actual problems – Ewan Feb 27 '20 at 12:15
@Guran I like the do, do not advice you wrote. I have seen that pattern and observed most of the time, folks are hesitant to step up to the plate. If you have any insight on the nature of that hesitation, I would like to see you add more to your answer. – learning2learn Feb 27 '20 at 16:40
2

@learning2learn Sorry, I don’t really understand what you’re after. Can you give an example? – Guran Feb 27 '20 at 19:05
@Guran my premise is that TD can be a reason project effort is underestimated, and if you agree, do have insight why people may not speak up about it? – learning2learn Feb 27 '20 at 20:48
THIS! Unless legacy software costs you money every year - interest - it's not technical debt, it's just legacy software. The reason to pay tech debt down is to reduce ongoing costs, nothing else. – Ben Feb 28 '20 at 09:08
2

@learning2learn In my experience, TD may be a cause of underestimated effort, but rarely the no 1 factor. (As long as we're talking about TD in our own code and not external dependencies) Developers usually have a pretty good grasp about how easy/painful it is to add functionality in their own codebase. As for why people don't speak up... well "we have a TD problem" easily reads as "we wrote crap code" – Guran Feb 28 '20 at 09:36
@Guran If I understand your experience correctly, management doesn't have the insight (cost/impact of TD), which is needed in order to do prioritisation effectively. In your case, this is caused by miscommunication between engineers and management - engineers did not provide the missing info, and management did not ask the info that they need but missing. Right? – shiouming Mar 01 '20 at 14:24
@shiouming Close enough. Remember though that communication goes in two directions. Good management doesn’t just tell dev what to build but also why. – Guran Mar 01 '20 at 15:11
@Guran I recall one of my past experience few years ago, I requested teammates assistance to log down TD impact that they experiencing. However, only minority were willing to do so (I worked in a large product team, >50 developers), as many other believed that management won't take it seriously even if they log them down. Have you encounter such trust issue before? Do you have any suggestion on, as a developer, how to resolve trust issue between teammates (developers and management)? Thanks. – shiouming Mar 02 '20 at 01:48
1

@shiouming I wish I had. Unfortunately all sane methods of improving code quality requires trust in the first place. Due to this, my first priority in any new environment is to establish trust. (By delivering features and being transparent) If trust cannot be established, everything spirals downwards. The subject of reestablishing trust is definitely a good subject for a separate SE question. (Bet there are several already) – Guran Mar 02 '20 at 15:19
'installments och that debt', what does 'och' mean? – Motomotes Mar 09 '20 at 14:56
@Motes it means autocorrect in the wrong language... Should have been “on” – Guran Mar 09 '20 at 16:04

score 9 · Answer 3 · answered Feb 26 '20 at 04:53

9

We already had such a report available in our issue tracker (Jira). When the introduction vs cleanup graph started getting out of hand, management indeed devoted more resources to fixing it. I think visibility definitely makes a difference in prioritization.

The main problem is, the visibility is all about the time between a defect being recorded and it being fixed. This resulted in more time being devoted to fixing recorded bugs. This resulted in fewer people being available to create new features. Developers feel stretched thin and rushed so they are skipping things that don't show up directly in reports, like refactoring and automated tests.

We're now in this weird situation where we are paying down debt faster, but also generating it faster. That goes to your culture question. Having priority to fix highly visible bugs is very different than having a culture of building quality in to begin with. The latter is much more difficult to cultivate because it is more difficult to measure and feels like more up front effort.

answered Feb 26 '20 at 04:53

Karl Bielefeldt

147,435

Hmm..., by any chance, are trivial/low impact debts given more attention than needed in your case? On the other hand, I believe the outcome your team's experiencing is definitely not unique, the larger the resource gap, the more serious it will be. It seems that we're just dealing with another form of resource issue. – shiouming Feb 26 '20 at 06:16
2

How do you track technical debt in jira? I couldn't find anything in jira related to technical debt – BЈовић Feb 26 '20 at 12:45
1

@BЈовић when you see concrete examples, filing a "fix-it" ticket with a known tag is usually enough to let people track stuff that needs fixed – Morgen Feb 26 '20 at 17:42
1

If your paying it off faster but generating it faster doesn't that mean you're just moving it around? – Bloke Down The Pub Feb 26 '20 at 19:29
Basically yes. Moving it from older code to newer code. – Karl Bielefeldt Feb 26 '20 at 22:24

Martin K · Answer 4 · 2020-02-26T16:23:14.097

7

The SCRUM canonical answer is that the team owns the tech debt. The team builds the sprint backlog. The team thus controls the rate tech debt is reduced.

In practice, angry managers will appear if you actually do this and escort you out of the building :)

More seriously, it's important to have good communication with the PO about what tech debt is causing issues in production and which tech debt causes current feature implementation to be slow. That will help you drive resolution of critical tech debt without alienating the PO and have one strategy for your deliverables.

edited Feb 26 '20 at 16:23

answered Feb 26 '20 at 10:26

Martin K

2,897

"which tech debt causes current feature implementation to be slow" That's a good point. – shiouming Feb 27 '20 at 05:39

score 4 · Answer 5 · answered Feb 26 '20 at 17:17

Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed

Why does this happen?

Ask your management. Make sure they're aware of this phenomenon, and get their justifications for it.

Working on tech debt needs to be 'sold' as a feature. Your management needs to be able to apply a cost/benefit analysis to the effort necessary to resolve the debt/the value of resolving the debt.

That having been said -

The development team owns the code. If you need to refactor before you introduce X feature because the team, as a whole, understands that getting further out on that limb endangers the future of the project, then include that in your estimate, and tell them that's what needs to happen.

They don't get to tell you what needs to happen to implement a feature - They can't. It's not their job or their skillset, and they don't have enough knowledge to make a judgement anyway, and they know it. If you tell them it'll take X story points, and the team has consensus around that figure, they'll accept it on fiat.

Have standards, stick to those standards, and have faith in the significance of your own standards. This is part of your job, because the people feeding you features have no context to set those standards for you.

If you can't get consensus on your team regarding what refactor work needs to get done for a given feature (IE, whether or not it's up to aformentioned standards), that's either a cultural problem that can be addressed internally, or an indicator that this tech debt isn't actually accruing in areas that are as critical as they seem from your perspective - And you'll need to pursue that conversation at length with your team to find out which one it really is, and be willing to understand that it's probably at least a little of both.

"get their justifications for it" Most common reason given, either "Project deadline is fixed, we don't have more manpower to be assign to the project" or "We have many business projects to be deliver. We'll come back to work on tech debts in future whenever we have time" (we rarely have free time, of course). — shiouming, Feb 27 '20 at 05:58
@shiouming - That means you're failing to sell the importance of resolving technical debt to your management. They need to understand, clearly, what it means to your velocity in adding features to do work involving fixing things. Focus on communicating that doing X work will enable all future work involving Y topic to go faster. If they're still making the call that it's not worth it to do the tech debt, that's their job, and then it's just a question about whether or not you trust their judgement enough to keep working there. — Iron Gremlin, Feb 27 '20 at 20:16
I believe most management are intelligent enough to realise there's impact, especially after teammates reported pain in terms of project productivity. My guts feeling is that, they don't have a concrete picture about how serious problem is. My idea is to help management to gain clearer picture, via two steps: (1) have techdebts logged, and have their impact tracked (2) have a report for introduction vs cleanup rate. My doubt was that, will these efforts solve the issue or just a waste of time, generally speaking (not specific within my org). — shiouming, Feb 28 '20 at 02:16
@shiouming I've seen such an effort succeed (for certain values) multiple times. Standardizing how technical debt is communicated, and making sure it gets added to the backlog as user stories, goes a really long way towards helping management get a handle on proper prioritization. I would not consider the first step on your two step plan to be a "waste of time" on any project, rather, I would consider that to be best practice. The second step may or may not be of value, depending on org and politics. — Iron Gremlin, Mar 02 '20 at 19:19

score 2 · Answer 6 · answered Feb 28 '20 at 18:23

XP solved this problem a couple of decades ago in a very simple way.

Customers, or product owners, or whatever you care to call them, cannot ever declare how long a story will take. They can only put stories in the order they'd like them to be done and, if they don't like certain estimates, negotiate with the developers to change stories to be cheaper.

The development team (which cannot change story priority, ever) is responsible for maintaining an appropriate level of technical debt, determining when it can be increased for business purposes and under what schedule it must be paid back, and so on. The time and effort necessary to handle this at any particular point is never shown to the customer, but merely built into the current estimates for stories.

In particular, you never have a story about refactoring or reducing technical debt or anything like that because the customer has full control over the scheduling of the stories, up to and including the ability to say, "I want you never to do that one." If you are telling the customer to make decisions about technical debt, you are abdicating your responsibility as a developer to ensure that the project can move forward at a predictable rate. (Do you seriously think that the customer can be better, or even as good, as you at understanding the future effects of technical debt in the codebase?)

So split your management of projects into techinical management (the "developer team," which cannot set story priorities) and product management (the product owner or "customer" in XP lingo) who cannot make technical decisions, including decisions about how much technical debt one should hold or how much time and effort one should spend at the moment paying back existing technical debt.

If for cultural or other reasons you do need to have someone making decisions about both of these, at least try to have them differentate these two roles and keep them independent, i.e., they're wearing the "developer hat" or "product owner hat," but never both at the same time. (This worked well for me when running my own company.)

score 1 · Answer 7 · answered Feb 26 '20 at 13:14

I tend to be in favor of tracking technical debt in the issue tracking tool and tagging it appropriately. In Jira instances, that often means an issue type. In Github, a label. Other tools may have other methods. However, doing so requires awareness of the technical debt to begin with. I'd be concerned that any reports don't give an accurate picture of the state of technical debt in a product, but rather the state of known technical debt in the product. Encouraging people to evaluate and track technical debt in the issue tracking would be a process, and perhaps a cultural, change for the organization.

It's going to depend on how your organization views technical debt. Before working on a given module or component, the developer can query the issue tracker for known technical debt and this can help guide estimates. Knowing known deficiencies in test coverage or tight coupling or other things that are now seen as poor design decisions can give the necessary insights to help guide estimation and planning activities. However, the product and project managers can always push back on these estimates with deadlines and committed dates. It's a balancing act.

bob · Answer 8 · 2020-02-26T19:53:26.823

It sounds like a culture issue

Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed.

To me this screams culture issue. It seems like management is aware of the problem (maybe not fully, but at least somewhat), and doesn't want to hear it. That something is more important to them than code quality right now. How is management incentivized? What counts as success for them? Stable, maintainable code? Or splashy new features, time to market, etc.? It sounds like it's the latter. In which case telling them the code stinks isn't going to help anything, because quality isn't how they define success. To them, it doesn't stink. So ultimately you need to first learn what your corporate dev culture is, before you can decide whether and if so how to tackle this problem.

"How is management incentivized? What counts as success for them? Stable, maintainable code? Or splashy new features, time to market, etc.?" Time to market is definitely part of management's KPI. I'm not sure if maintainability is included. — shiouming, Feb 27 '20 at 06:02
It's also possible for an org that doesn't have the minimum resources required to function properly. — shiouming, Feb 28 '20 at 06:34

Is technical debt management problem more of a culture issue or insight issue

8 Answers8

Anecdotally

The monkeys and the ladder

Answering your questions

It sounds like a culture issue