Technical Debt Interest Assessment: from Issues ... - ACM Digital Library

Technical Debt Interest Assessment: from Issues to Project Antonio Martini

Simon Vajda

Rajesh Vasa

Chalmers University of Technology Gothenburg, Sweden [email protected]

Deakin Software and Technology Innovation Lab Melbourne, Australia

Deakin Software and Technology Innovation Lab Melbourne, Australia

Allan Jones

Mohamed Abdelrazek

John Grundy

Deakin Software and Technology Innovation Lab Melbourne, Australia

Deakin Software and Technology Innovation Lab Melbourne, Australia

Deakin Software and Technology Innovation Lab Melbourne, Australia

Jan Bosch Chalmers University of Technology Gothenburg, Sweden [email protected]

ABSTRACT

1

The interest of Technical Debt (TD) is difficult to calculate, especially on a project level. Current approaches are based on fine-grain issue assessment, but there is no evidence about how TD is assessed on a project level. A few tools use an aggregation function that sum the TD issues on a project level. We conducted a multiple case-study on four different projects. We asked the project teams to assess the TD both on an issue level and on a project level. We also asked the product manager and a senior developer to assess the TD on a project level. We found that the function mapping the interest of TD to a project overall is not the sum of issue-level TD. We report the quantitative results of the performed experiment and we also developed a qualitative explanation of the results based on interviews with the development team. This paper represents a first step towards assessing the interest of TD at a project level.

Technical Debt is regarded as sub-optimal solutions that have benefits in the short term but cause extra negative impact in the longterm [2]. The long-term negative impacts are referred to as the interest paid on the debt [1]. So long as the debt is not repaid, the organization pays a current interest and might incur into a future interest. Technical Debt can be repaid with a refactoring of the system to remove the suboptimal solution: this would stop or prevent the organization to pay a costly interest. However, the decision of repaying the debt depends on the knowledge related to the interest [7, 10]: if the interest is (or is going to be) high, the debt is worth being paid. On the contrary, if the interest is not enough to justify the cost of refactoring, there is no reason to "waste" resources to refactor the system. The interest of a technical debt is, however, quite difficult to assess [6, 12]. Moreover, current literature is specifically focused on assessing TD items [4, 5, 7, 11]. Each item is assessed separately: such an approach makes sense when a decision is needed whether to refactor a single suboptimal solution or not. However, it is also important to understand the overall impact of TD on the whole project. This would enable strategic decisions with respect to the project goals. For example, if the accumulated TD (and its interest) is critically hindering the goals of the project, in which case the organization might decide that the resources to repay the debt are worth being spent on a refactoring. In general, although it is a good practice to "divide and conquer" for analysis purposes, the final decisions are taken on a project or product level, evaluating the information holistically. The aggregation of the TD interest on a project level has not been studied so far. One approach is based on ranking the source code based on different code metrics and code changes [9]. However, the study is based on source code TD, and code changes are not necessarily a good proxy for the interest, as shown in [7]. Tools such as SonarQube represent the debt related to a project as a number of days. However, such number is related to the sum of the cost of refactoring (or else the principal) and not to the payment of the current or future interest.

CCS CONCEPTS • Software and its engineering → Risk management;

KEYWORDS Technical Debt; Tracking; Interest; Case Study; Project Management ACM Reference format: Antonio Martini, Simon Vajda, Rajesh Vasa, Allan Jones, Mohamed Abdelrazek, John Grundy, and Jan Bosch. 2017. Technical Debt Interest Assessment: from Issues to Project. In Proceedings of XP ’17 Workshops, Cologne, Germany, May 22–26, 2017, 6 pages. https://doi.org/10.1145/3120459.3120469 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. XP ’17 Workshops, May 22–26, 2017, Cologne, Germany © 2017 Association for Computing Machinery. ACM ISBN 978-1-4503-5264-2/17/05. . . $15.00 https://doi.org/10.1145/3120459.3120469

INTRODUCTION

XP ’17 Workshops, May 22–26, Antonio 2017, Cologne, Martini, Simon Germany Vajda, Rajesh Vasa, Allan Jones, Mohamed Abdelrazek, John Grundy, and Jan Bosch In this study, we aim at understanding how the interest, related to single TD items, is aggregated to an overall impact of TD on a project. Our research question is therefore: • RQ: How is the impact (interest) of TD aggregated from issues to project? We have conducted an exploratory, multiple case-study involving developers and a product manager responsible for 4 software projects. We have obtained the estimated interest for single issues and the overall estimated impact of TD on the project. The analysis of the results shows that the aggregation of the interest of single items is related to the overall impact of the technical debt on the project. However, the function that aggregates the single issues to the project level is not a sum, contrarily to what might be expected from the heuristics used in tools such as SonarQube. Furthermore, when considering the interest of TD on a project level, there are other factors that need to be considered for the calculation of the interest. We report the results of the quantitative investigation involving the estimated interest of 4 projects, and we provide a preliminary qualitative explanation of the mapping function that takes as input the interest assessed for a set of Technical Debt issues and returns an overall project impact indicator as output.

2 BACKGROUND 2.1 Analyzed Projects In this study we have assessed four different projects, developed and maintained by four different teams with some level of overlap i.e. some team members are common across these projects. Here is a brief on each of these projects: • Wicketkeeper is cloud-based software that classifies email as solicited or unsolicited using neural networks that are periodically retrained. (1) Issue - The functionality that transforms emails into sequences that can then be fed into the neural networks lacks unit tests. Cause - Inconsistent, undocumented or unexpected behaviour of the third-party text parsing libraries involved which would have been extremely time consuming to capture in tests. Reason - Development involved frequent training runs that exposed the functionality to millions of varied inputs in a short amount of time (within a development environment), compensating for the lack of tests. (2) Issue - Thorough integration testing of the communication between components of the system was absent.Cause - The dynamic nature of the implementation language and frequent changes to architecture, and thus component communication, created a perception that integration tests would provide little immediate value.Reason - Postponing the creation of integration tests enabled rapid experimentation with respect to the system’s archictecture and the data pipeline. • Virgil is a platform for distributed computing that allows for the distribution of blackbox, resource-intensive tasks

using a simple job description language and powerful underlying cluster and resource management and monitoring tools (1) Issue - Selecting Apache Aurora to handle scheduling operations on the cluster so those operations didn’t have to be implemented. Aurora has a number of reliability/rescheduling features that can be taken advantage of and was an ideal fit at the time Cause Apache Aurora doesn’t provide enough fine control over scheduling decisions, which will become a problem over time. For example the user will expects jobs to completed in order of submission, but this is not the default behaviour Reason - A workaround abusing the priority functionality had to be implemented, which introduced more upkeep, maintainence and fragility to the system. Only a small part of the API (submit job, request job status, kill job) is used, therefore it would be logical to swap out and replace. Unfortunatley that would be a lengthy undertaking to write a replacement. Ultimately the overhead of maintaining a workaround will only increase over time, therefore a replacement will need to be written eventually. At this point there is no other existing OSS replacement that fits our requirments. • Czar is a platform and set of tools for unifying project management and productivity applications used in software projects and encouraging consistent and disciplined software engineering practices. (1) Issue - Deployments are currently manual. Cause - There are no database migrations. Reason - The process is also manual. • Serenity is a platform that enables the construction of artificial intelligence and Internet of Things applications, with a focus and application in smart homes and environments. (1) Issue - Cron-based scheduling Cause - Cron expressions are strings, that appear fairly arcane and complex. It is fairly easy to make mistakes and errors without noticing. Reason - Any service that needs scheduling therefore has cron expression strings. (2) Issue - Javascript as a technical debt thought experiment.Cause - Javascript is a non-statically-analysed language. This means that there are little frustrations that need to be overcome. As the main language of the platform, these little frustrations are very common and occur frequently. Typos become bugs and it is easy to mismanage consistency of interfaces, etc. Reason - Increased adoption of Flow for static type checking which assists with these frustrations. The use of Javascript is still thought of as technical debt to a minor degree. (3) Issue - Diagnostics monitoring server. Cause - The original version deployed only tracked head state and could only provide dumps of each table. A recent change has been to keep historical data and add a number of end points, to assist with more useful queries in the future. Reason - A full understanding of how the

Technical Debt Interest Assessment: from Issues to Project server would be used was not available at the time of initial implementation. A general idea of what information was needed was collected and stored. This was then adjusted based on what was found to be most useful. (4) Issue - AI ensembler. Cause - Starting work on the service. Reason - This issue may cause technical debt to be acquired due to an undecided approach.

2.2

Data Collection

We started the study with a workshop with the participants, to brief them about the definitions of Technical Debt (e.g. using the definition in [2]), followed by concrete examples. This helped aligning the understanding of TD among the participants and reducing possible construct validity threats. We also set a specific terminology that was more familiar to the developers: for example, instead of "interest" we used "Nagative Impact" (NI). From the four cases described in the previous section, we involved seven people: six developers (with different level of expertise) and a product manager, with knowledge about all these projects. We had separate interviews with all the participants. For the Serenity project, we interviewed more than one developer, as the project was significantly bigger than the other ones. For the other three projects, each developer felt confident in identifying and assessing all the TD items present in the system. As a first step, we interviewed the product manager: we asked him to give a list of Technical Debt issues present in these projects, and then we asked him to give an Overall Impact (OI) value of the TD on the projects. The impact ranged from 1 to 10, where 1 was interpreted as "very low or no impact on the project", and 10 was interpreted as "very high impact on the project". As a second step, we asked project developers to assess the Negative Impact (NI) of the most important issues in their (part of the) project. To obtain this value, we used a method recently developed and evaluated in practice [7]. To simplify the method, the authors developed a tool named AnaConDebt [8]. Although the focus of this paper is not on such method, we recall here the procedure to assess the interest of TD. We collected, from the developers, the assessment of seven factors and their estimated growth over time, according to [7, 8]. Then, we asked the developers to give an estimate of the Negative Impact of each TD issue from 1 to 10, where 1 corresponded to "no or very low NI" and 10 to "very high NI". The seven factors taken in consideration are [8]: • Reduced Development Speed If the speed is reduced, the interest of the TD item is affecting the evolvability and the maintainability of the system • Bugs related to the TD item If many bugs are related to the area where the TD is located and the developers suspect a connection between the internal quality issue and the bugs, this greatly affects the maintainability of the system and the time that needs to be spent on fixing bugs instead of developing new features. • Other qualities compromised There are other qualities that can be affected by the TD besides maintainability and

XP ’17 Workshops, May 22–26, 2017, Cologne, Germany evolvability, as shown in [7]. Here we used the ones suggested in the ISO standard [3] but excluding maintainability and evolvability (as they are already covered by the previous factors) • Other extra-costs As shown in [7], there might be other context-dependent extra-costs to be considered when assessing the interest. It is therefore important to give the possibility of an extra factor to the user. • Frequency of the issue The severity of the interest is related to how frequently the negative impact occurs. • Spread in the system The larger the portion of the system affected by the TD, the more ripple effects the interest might have on the organization and on the newly added code. • Users affected Although the TD might not be located in a large part of the system, such part might be accessed by many users. In this case, the extra-cost of the interest would be multiplied by all the stakeholders suffering its payment. Finally, the tool gives a suggested impact value from 1 to 10. The developers were left free to adjust their estimates by taking in consideration the suggested impact or not. The method and tool AnaConDebt were previously evaluated with other organizations, where they showed to be helpful and precision in estimating the interest of TD items [7, 8]. From this data collection, we obtained a list of the most important issues for each project together with their NI, reported in table 1. This procedure was done to apply the current published best practices related to the assessment of the interest of Technical Debt. After the assessment of the single items, we asked the developers to give an overall impact value (OI) of the Technical Debt on the project, in a similar way as we asked the product manager. This was done to compare the different views between the product manager and the developers on a project level. After the assessment, we asked the developers to explain the relationship between the NI reported for the single issues and the estimated OI on a project level. As last step, we showed the results (see the Data Analysis section) to the product manager, to understand the results and to give him the possibility, if needed, to adjust his assessment. Also, in the last step, we asked the product manager to mention what factors he was using when assessing the impact of TD on a project level. We also asked him to weight the factors by importance. In summary, for each project we obtained: • a list of TD issues with associated NI • an OI assessment given by the developers • an OI assessment given by the product manager and in some cases by the architect • a qualitative explanation of how the OI was estimated using the NI

3 DATA ANALYSIS AND RESULTS 3.1 Quantitative results We analyzed both the quantitative data and the qualitative explanation of the participants.

XP ’17 Workshops, May 22–26, Antonio 2017, Cologne, Martini, Simon Germany Vajda, Rajesh Vasa, Allan Jones, Mohamed Abdelrazek, John Grundy, and Jan Bosch Table 1: Issues assessment Issue/Project Wicketkeeper-1 Wicketkeeper-2 Wicketkeeper Wicketkeeper Wicketkeeper Virgil-1 Virgil-2 Virgil Virgil Czar-1 Czar-2 Czar-3 Czar Czar Serenity-1 Serenity-2 Serenity-3 Serenity-4 Serenity-5 Serenity Serenity Serenity Serenity

Informant Dev Dev Dev Arch PM Dev Dev Dev PM Dev Dev Dev Dev PM Dev-1 Dev-1 Dev-1 Dev-2 Dev-2 Dev-1 Dev-2 Arch PM

NI or OI NI: 4.5 NI: 3.6 OI: 3.7 OI: 4.2 OI: 4.5 NI: 4.6 NI: 4.3 OI: 3 OI: 3 NI: 8 NI: 7.1 NI: 6.2 OI: 4 OI: 3 NI: 7 NI: 6.2 NI: 5.25 NI: 5.85 NI: 3.1 OI: 3 OI: NP OI: NP OI: 4

As for the quantitative analysis, we obtained the data visible in table 1. For each project, for each issue, one or more developers reported an NI assessment. For example, the first row in table 1 shows the NI assessment given by the developer interviewed for the WicketKeeper project, in particular for Issue 1. The second row is related to the assessment of the same developer for Issue 2 related to the same project. The third row shows the OI given by the developer. The fourth and fifth rows show the OI given respectively by the architect (an experienced developer) and the Product Manager for the project. For WicketKeeper, Virgil and Czar we interviewed one developer, as the projects were rather small. We interviewed two developers from Serenity, as the project was significantly larger than the other three. However, when asking for the OI to some of the developers, they felt that they did not have enough information to give a reliable answer ("OI: NP" in the rightmost column). Once we obtained the data for issues and projects, we analyzed the aggregated values and we compared them across roles, in order to triangulate the answers and to find possible patterns and relationships among the results. The main results are visible in 1, and below we explain how we aggregated the values and which variables we analyzed to obtain our main findings. We calculated the sum and the average of the NI of the issues assessed by the developers for each project. Then we compared it with the OI given by the developer, with the OI given by the product manager and with their average. This was done to understand the relationship between the NI of the issues and the OI given by developers, architects and product manager.

WicketKeeper 4,05 avg of NI issues

0,1

0,5

diff between avg NI and avg OI

diff between OI

Virgil 4,475 avg of NI issues

1,475

0

diff between avg NI and avg OI

diff between OI

Czar 7,1 avg of NI issues

3,6

1

diff between avg NI and avg OI

diff between OI

Serenity 5,47 avg of NI issues

0,47

2

diff between avg NI and avg OI

diff between OI

Figure 1: Graphs summarizing the analyzed data from NI and OI

We immediately found that the sum of the NI:s was much bigger than the OI:s. This is an important finding, as it shows that the NI of TD is not used by the stakeholders in an additive fashion to obtain the OI. We noticed that the average was a closer value to the OI, and we took a closer look at the difference between the average of NI and the average of the OI. These data are visible in 1. We also analyzed the difference between the OI given by the Product Manager and the OI given by the developers, in order to triangulate the data and to understand if the assessment was consistent. With the help of the qualitative data, we also found how different roles perceive TD differently and how they adjust their perception with the help of other participants’ input. The graphs in 1 show the comparison of the main variables that were observed. We omitted the sum of the NIs, as it diverged too much with respect to the other variables to be useful for a visual analysis. Below we explained the main findings related to this analysis on the aggregated values: • On the left part of the graph, we can observe the average for the NI of the project issues

Technical Debt Interest Assessment: from Issues to Project • The middle column shows the main difference between the average NI and the average OI. From this datum, we can observe if OI can be considered a function of the NI or if the OI is completely unrelated from the NI. In all cases, we can observe a certain difference for most of the projects, although the difference is not high. This would support the hypothesis that the OI is a function of the NI of the issues. However, such function cannot be considered to be a sum or a simple average, as these two hypotheses are not supported by the results, as explained earlier. • On the right column, we can observe the difference between the OI from the product manager and the OI provided by the developers. This would show if the OI is aligned or if the local view of the developer is different from the overall view from the product manager. The values are quite consistent here, apart from the last (and biggest) project. This would support the hypothesis that for small project the perception of the impact of TD is aligned between developers and product manager. However, as the project grows, the perception seems to diverge. An important result comes from the follow-up discussion with the product manager observing the results. He decided to adjust his OI:s towards the values given by the developers (not replacing them though). However, in some cases, the PM decided to keep his values, for reasons related to the specific team. This shows how understanding TD negative impact (and therefore its interest) seems to be a collaborative and iterative task.

3.2

Qualitative results

Observing the difference in between the OI:s and between OI:s and NI:s, it appears that, although the OI depends on the NI:s, there are other factors weighted in the estimation of the impact of TD on the overall project. After the quantitative assessment of the NI and the OI, we asked the developers and the product manager to explain what other factors were used to reach the OI value. After the analysis and synthesis of such factors, we then asked the product manager to rank them as more or less important when weighting them to obtain the OI for the project. The factors are reported in table 2. In such table, we report also the importance given to the factors by the product manager: the factors marked with "1" are factors that are regarded as more important, while the ones marked with "2" are regarded as secondary factors.

4

DISCUSSION

The study aimed at understanding how to aggregate the negative impact from single TD issues to the project level. In other words, given a list of TD items with an assessed interest, how can we calculate an estimation of the overall impact on the project level? In this study we have observed how the developers and the product owner of 4 projects were using the information given by the Technical Debt items to assess a negative impact of the whole project. We have collected quantitative data from the participants regarding the negative impact assessed for the major TD items in the projects. We have collected similar quantitative data related to

XP ’17 Workshops, May 22–26, 2017, Cologne, Germany Table 2: Factors weighted to obtain the overall TD impact on the project Factor Negative Impact (NI): this is the negative impact that is calculated by taking in consideration the NI of the issues. It is also important to understand what NI this issue has on the client. Portion of code: the bigger the portion of the code affected by the NI, the higher the impact that the TD has on the project. In other words, if a large portion of the code does not contain TD, then the TD is limited Project size and complexity: the bigger and the more complex the project is, the more TD is going to have an impact, both for refactoring and as ripple effects of its NI Project size and complexity: the bigger and the more complex the project is, the more TD is going to have an impact, both for refactoring and as ripple effects of its NI Roadmap: it is important to understand if the TD is going to affect many releases for the client or if the life-cycle of the product is limited Positive impact: TD is supposed to have positive benefits. These need to be weighted in when estimating the OI. For example, if TD has a NI related to productivity but at the same time it gives a boost in productivity on the other hand (for example for a quick delivery), the two impacts need to be balanced. However, according to the product manager, the positive impact is secondary with respect to the negative impact TD alternative: the impact of TD is calculated as lower if the team knows about it and there is a solution to repay it when needed. TD that cannot be refactored is considered as having more impact on the project Cultural attitude of the team towards TD: the product manager would consider higher the impact of the same TD if the attitude of the team towards refactoring it is negative. In other words, if the team is not keen on repaying TD, the impact can be considered greater

Weight 1

1

1

1

1

2

2

2

the overall impact of the TD on the whole project. We have compared these values among each other and among different roles, to analyze different perspectives and their final results. The main considerations that we can formulate from these observations are the following: • The Overall Impact of TD on the project is not calculated, by the developers and product owner, as a sum of the single items’ Negative Impact • The TD items’ Negative Impact is related to the Overall Impact on the project. The relationship between the two seems to be related to the average of the TD items’ Negative Impact. • The Overall Impact of the Technical Debt on the project is intuitively estimated by the developers and product owner in a similar way, although both the roles have different information on the project. For this reason, the best way

XP ’17 Workshops, May 22–26, Antonio 2017, Cologne, Martini, Simon Germany Vajda, Rajesh Vasa, Allan Jones, Mohamed Abdelrazek, John Grundy, and Jan Bosch to assess TD impact on a project level is to combine and iterate the two perspectives. • In addition to the Negative Impact of the TD items, there are other factors to be considered when calculating the impact on the project level. The formula could be represented, in an attempt to formalize the estimation, as follow: N I P = {ni 1 , ni 2 , . . . , ni n }

(1)

ni x

(2)

where is the Negative Impact of the item x in project P. The equation for the Overall Impact of the project P is therefore: OI P = k(f (N I P ), Other Factors ) | {z }

(3)

factors in Table 2

where f() is the average of the factors, while k, although in need for further investigation, might be represented by the weighted average of the factors in Table 2. Further investigation is needed to understand how to more precisely calculate this formula. However, we propose a first high-level understanding of how the negative impact of TD is estimated for a project. The limitations of this study are mainly related to the studied context. Most of the analyzed projects are rather small, while the big one presented some challenges for the estimation, for the stakeholders, in estimating the total impact on the project. We are currently conducting other studies with other organizations, involving larger projects.

5

CONCLUSIONS

Quantifying the interest of TD is a difficult task, especially at a project level. Current best practices are focused on assessing single TD items or by summing the cost of refactoring several issues. However, these do not help assessing the impact of Technical Debt on the project. We performed an exploratory study in 4 projects to understand how developers and managers assess TD interest on a project level, given their assessment of the single items. We found that the engineers and the product manager estimate the impact of TD on the project by using the information given by the single items, but not as a sum. Instead, they intuitively calculate a sort of average of the items, and they weight them according to other factors, such as the portion of the code affected by the Technical Debt, the project size, the Roadmap, the positive impact of the TD, the existence of an alternative and the cultural attitude of the team. We capture this relationship in a preliminary formalization of the problem, which gives a first basis for developing future studies and to develop methods to assist the stakeholders to estimate their Technical Debt.

6

ACKNOWLEDGMENTS

We would like to thank the Software Center at Chalmers University and the DSTIL Lab at Deakin University for making this research possible.

REFERENCES [1] Areti Ampatzoglou, Apostolos Ampatzoglou, Alexander Chatzigeorgiou, and Paris Avgeriou. 2015. The financial aspect of managing technical debt: A systematic literature review. Information and Software Technology 64 (Aug. 2015), 52–73. [2] P. Avgeriou, P Kruchten, I. Ozkaya, and C. Seaman. Managing Technical Debt in Software Engineering (Dagstuhl Seminar 16162). [3] ISO International Organization for Standardization. 2015. System and software quality models. (2015). http://www.iso.org/iso/catalogue_detail.htm?csnumber= 52075 [4] Yuepu Guo and Carolyn Seaman. 2011. A Portfolio Approach to Technical Debt Management. In Proceedings of the 2Nd Workshop on Managing Technical Debt. ACM, New York, NY, USA, 31–34. [5] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio QB Da Silva, AndrÃľ Luis M. Santos, and Clauirton Siebra. 2011. Tracking technical debt: An exploratory case study. In Software Maintenance (ICSM), 2011 27th IEEE International Conference on. 528–531. [6] A. Martini, T. Besker, and J. Bosch. 2016. The Introduction of Technical Debt Tracking in Large Companies. In accepted at APSEC 2016. Hamilton, New Zealand. [7] Antonio Martini and Jan Bosch. 2016. An Empirically Developed Method to Aid Decisions on Architectural Technical Debt Refactoring: AnaConDebt. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE ’16). ACM, New York, NY, USA, 31–40. https://doi.org/10.1145/2889160.2889224 [8] A. Martini and J. Bosch. 2017. The Magnificent Seven: Towards a Systematic Estimation of the Technical Debt Interest. In submitted at MTD 2017. [9] Ariadi Nugroho, Joost Visser, and Tobias Kuipers. 2011. An empirical model of technical debt and interest. In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD ’11). ACM, New York, NY, USA, 1–8. https://doi.org/10. 1145/1985362.1985364 [10] Klaus Schmid. 2013. A formal approach to technical debt decision making. In Proceedings of the 9th international ACM Sigsoft conference on Quality of software architectures. ACM, 153–162. http://dl.acm.org/citation.cfm?id=2465492 [11] C. Seaman, Yuepu Guo, Nico Zazworka, F. Shull, C. Izurieta, Yuanfang Cai, and A. Vetro. 2012. Using technical debt data in decision making: Potential decision approaches. In 2012 Third International Workshop on Managing Technical Debt (MTD). 45–48. https://doi.org/10.1109/MTD.2012.6225999 [12] Jesse Yli-Huumo, Andrey Maglyas, and Kari Smolander. 2016. How do software development teams manage technical debt? An empirical study. Journal of Systems and Software 120 (Oct. 2016), 195–218. https://doi.org/10.1016/j.jss.2016. 05.018