By Jason Roys
On January 11, 2023, air travelers received a nasty shock. The Federal Aviation Administration (FAA) halted flights nationwide for several hours so it could reboot a computer system that is critical to air safety. The FAA had a serious technical debt problem.
NOTAM (for “Notice to Air Missions”) delivers real-time data about potential air hazards, from bird flocks and weather systems to unauthorized aircraft. The system has been in use for decades, but this was the first time anyone could remember such a widespread failure. The FAA said the cause was a contractor who deleted files while working to synchronize the primary and backup databases.
The term “technical debt” entered dinner conversations across the country. Many wondered whether the FAA done enough in the way of IT modernization to identify and resolve issues with this legacy system. The topic is complex, and it has major implications for the U.S. government, federal agencies and all our critical infrastructure, including public utilities and financial systems.
Technical debt, also called tech debt or code debt, is a cost of doing business. Like financial debt, it permits an agency or a company to accomplish something in the here and now and put off paying for it. You might call it a “code now, pay later” system. But, like financial debt, it can overburden the balance sheet when the cost of paying it back accumulates too much “interest.” At a point, IT teams spend most of their time and budget on fixing legacy code and hardware rather than on software development and innovation. Equally important, or arguably more so, in national security situations, tech debt can literally prevent new code from being deployed.
In this article, we will look at what causes technology debt, discuss ways to recognize the symptoms, provide strategies for addressing technical debt and show how a technology consultancy like SDV INTERNATIONAL can help revive operations.
WHAT IS TECH DEBT?
People identify technical debt in several ways. In a 2020 article, McKinsey & Company referred to tech debt as the “off-balance-sheet accumulation of all the technology work a company needs to do in the future.”
Some technical debt is unavoidable because of time constraints, unexpected conditions, and evolving requirements. Such factors cause software development teams to take shortcuts, such as forgoing documentation and skipping test cases. Technical debt, therefore, also represents the cost difference between the resources and time it takes to develop a perfect software feature and the effort required to deliver it in a timelier manner. The agency ends up paying interest in the form of increased operation and maintenance costs.
Just like monetary debt, not all technical debt is bad. Just as a company takes out a loan to grow its production capabilities, tech debt is often used as a tool for delivering a piece of software or functionality right away. That is judged to be worth the cost of fixing or improving things later. But if the ‘later’ software development never comes, costs and problems accrue.
Legacy technology is one of the types of technical debt prevalent in federal agencies. Risk-averse and budget-challenged agencies keep their legacy systems as long as they can, deploying patches and manual processes to keep things running. In 2015, 75% of the government’s IT budget was earmarked for operating and maintaining legacy systems. Only 25% went to new technologies. Today’s estimates for managing technical debt are not much different.
This approach increases vulnerability to cyberattacks, as old systems have trouble keeping up with new threats. The government is paying attention. For example, the National Defense Authorization Act of 2022 authorized the U.S. Department of Defense to study tech debt and evaluate its risks.
HOW TO RECOGNIZE AND MEASURE TECHNICAL DEBT
First, we need to distinguish between intentional and unintentional technical debt. Some may suggest that intentional technical debt is used as a strategic tool.
An organization takes on intentional technical debt when it makes a conscious decision to optimize for the present, to perhaps ensure customer satisfaction in the near term, rather than for the future. There are both short-term and long-term variations. In either case, this type of ‘loan’ implies the organization has a plan for paying back or reducing technical debt.
Unintentional technical debt may be described as the non-strategic result of doing a poor job of managing architecture, in other words, the structure of a software system and the discipline of creating such structures and systems.
Sometimes software developers are not aware of tech debt. The following are some common symptoms that indicate tech debt issues:
Most of the IT budget is going to operations and maintenance.
An organization (its projects) is increasingly slow to produce functionality.
Developers have the stress of dealing with last-minute, unforeseen problems.
Software uses out-of-date libraries or ones that no longer are supported.
Solutions become increasingly buggy and new bugfixes create new bugs.
Automatic testing doesn’t cover at least 90% of code.
Production environment is unstable.
One key indicator: If after a couple of minutes of reading code, the development team still can’t fully explain what it does, chances are the organization is accumulating technical debt.
Before an enterprise can address technical debt, it has to measure it. Here are some common metrics:
Defect ratios. If new bugs begin to outnumber the fixes, technical debt is growing.
Coding quality. Development teams that use coding standards and strive for quality code can mitigate technical debt. Over time, increases in manual labor (cost) will be required.
Completion time. If engineers and developers spend a lot of time on what are traditionally easy tasks, it can be a sign of code or infrastructure trouble.
Rework. If engineers are reworking infrastructure or code in production, they are likely dealing with technical debt.
Technical debt ratio (TDR). This metric compares what it costs to fix the problems against the total cost to build the project. The equation for TDR is: (remediation cost / development cost) × 100. How much technical debt is too much? Many well-run enterprises set a TDR target of no more than 5%.
PREVENTING AND MANAGING TECHNICAL DEBT
As we’ve noted, technical debt is sometimes a strategic decision made to meet particular business needs. That said, an organization can use these best practices to prevent unwittingly accumulating tech debt or compounding existing debt.
INFRASTRUCTURE
Infrastructure is critical for software development and code deployment because it provides a consistent and reliable environment for software developers to create and deploy code. In this case, infrastructure refers to the resources that are needed to support software development and code deployment. It includes, for example, services such as cloud computing and storage, databases, and security tools. Additionally, infrastructure can include practices such as automated testing and continuous integration, which are essential for ensuring the quality and reliability of code.
Without good infrastructure in place, there would be no effective way to test and deploy code reliably, leading to errors and inconsistencies in the final product. Low debt infrastructure also allows developers to quickly deploy and scale their applications, as well as access resources like databases, cloud storage, and other services. Finally, a strong infrastructure ensures that code is secure, reliable, and compliant with industry standards and regulations.
DON’T STINT ON IT TALENT
Hiring the right people to develop complex software is the first step toward avoiding tech debt. IT leaders must give developers the right resources (including time) and infrastructure. Overworking developers puts an organization on the road to technical debt. An IT consultancy like SDV INTERNATIONAL can help an agency or company get a team up and running quickly and share the workload.
USE PROJECT MANAGEMENT AND QUALITY MANAGEMENT SYSTEMS
Project management lets an organization plan out the entire development, and a quality management system (QMS) is an enterprise-wide system of documents, policies, procedures and controls that will create and deliver high-quality products or services to customers every time.
TEST, TEST, TEST
Automated testing is efficient in terms of time and effective in terms of outcome, ensuring bug- and error-free code. Manual testing sprints have their place, but they are not a substitute for automated testing.
OBSERVE CODING STANDARDS
Substandard code leads almost inevitably to technical debt. Yes, it can save time, but an organization will pay for it on the back end. Embrace more efficient code structures.
TRY AGILE PRACTICES
In the traditional waterfall model of software development, each phase of development derives from the previous phase and cannot proceed without it. Agile software development is based on iteration, where requirements and solutions evolve through collaboration among self-organizing cross-functional teams. Some agile frameworks include Extreme Programming, Scrum and Feature-Driven Development.
TRACK TECHNICAL DEBT
Measure a product or system’s technical debt to provide early warning signs of potential problems. Examples of key indicators are code quality, code coverage, velocity rates, defect rates, defect resolution rates and TDR. To manage technical debt, you must track it.
USE A LOW-CODE PLATFORM
The laborious nature of coding is giving way to low-code systems, a more intuitive drag-and-drop interface that speeds software development. A low-code development platform or LCDP allows teams with strong senior oversight to build complex software quickly. SDV INTERNATIONAL utilizes Artificial Intelligence (AI) and Machine Learning (ML) to significantly accelerate coding in environments.
REVERSING OR REDUCING TECHNICAL DEBT
Many of the best practices for reversing or reducing technical debt are the same as the ones for preventing it in the first place. However, organizations also need to put long-range planning ahead of a mindset that prioritizes the timing of project delivery.
Federal agencies are starting to realize this. An enterprise as vast as the federal government has piled up enormous technical debt. While business needs and technology have evolved, many agency leaders have found themselves stuck with legacy systems that are vulnerable to cyberattacks and require a good deal of maintenance. They are forced to focus their resources there rather than on digital transformation.
Many agencies have entrusted SDV INTERNATIONAL to help them reduce technical debt, including the Department of Defense, the Department of Veterans Administration, the Federal Aviation Administration and many others. Data integration, data security, procurement, research, and health information management are among the business processes SDV INTERNATIONAL has tackled for these agencies. We often implement cloud-based solutions that don’t rely on legacy technology and legacy coding.
SDV INTERNATIONAL will provide a complimentary white paper on this topic with an actionable technical approach to qualified organizations that support the U.S. Government.