using non-invasive measurement techniques in agile software ...

USING NON-INVASIVE MEASUREMENT TECHNIQUES IN AGILE SOFTWARE DEVELOPMENT: A SWOT ANALYSIS Andrea Janes, Barbara Russo, Giancarlo Succi Faculty of Computer Science, Free University of Bolzano-Bozen, Dominikanerplatz 3 Piazza Domenicani, I - 39100 Bolzano {ajanes, brusso, gsucci}@unibz.it

ABSTRACT Measurement in software production is essential for understanding, controlling, and improving the development process. Past research has emphasized the importance of a disciplined data collection process as a prerequisite for a sound, solid, and useful analysis. This article presents motivations and impediments of non-invasive measurement techniques, also using the theory of self-determination. Special attention is given to the consequences of noninvasive measurement on measured individuals in terms of participation to the measurement process and of the overall the work motivation. 1. INTRODUCTION To measure means to make a comparison to a standard. It makes measured things comparable in regard to the defined standard and also comparable among themselves. Measurement in Software Engineering helps (as in every other science) to make phenomena more visible, understandable and therefore controllable. Software metrics play a crucial role for cost and effort estimation of projects (e.g. Barry Boehm’s COCOMO model), for deriving productivity models and measures to evaluate the performance of ongoing processes during different development phases [Fenton and Pfleeger, 1997]. Software metrics proponents claim that using measurement techniques as a feedback mechanism help to change existing processes into more disciplined and therefore manageable ones. The other side of the medal is that calculating software metrics or recording performed steps, activities and decisions is a time consuming and therefore costly activity. Manual data collection depends on the discipline of people collecting the data. As such, it is error-prone. Additionally, an accurate manual data collection requires effort and such effort may impact the productivity of the individuals being measured. Wrongly collected or omitted data can alter significantly the overall result of the analysis and in this way render the manual data collection useless. Non-invasive measurement techniques have the potentials of overcoming the limitations of manual data collection. Their aim is to collect data and to deduce measures from such data with the minimal possible user intervention. Since the primary objective of agile methods is to provide value rapidly, the use of metrics to increase software quality and to monitor the project advancement is not

very common. Non-invasive measurement techniques as a technique that does not require a direct effort from the developers can improve agile methods considerably. In agile development, the focus lies more in rapid value creation, that means focusing only on the activities that provide value to the customer now and not on activities that perhaps might help in the future. Now, non-invasive measurement tries automate the measurement steps performed by hand before. This is done using a set of techniques: • by writing plug-ins for specific applications that log what the user is currently doing, • by writing plug-ins for the operating system that analyze what application is currently used or what files are currently changed, • by analyzing the output of a user, the produced artifacts and by analyzing this artifacts, • by combining everything in order to infer the information that previously was collected manually. To better analyze the concept of non-invasive metrics collection we want to perform an analysis that is quite common in economics: a SWOT (Strength, Weaknesses, Opportunities and Threats) analysis of non-invasive measurement techniques. 2. STRENGTHS The improvement of the software development process relies on metrics to understand the underlying development process. An example of an approach for improving software quality is the “Personal Software Process” (PSP) [Humphrey, 2000]. The PSP has a focus on improving the activities directly performed by the single developers. Because every single developer is considered a different entity, it is proposed that developers must “plan their work and they must base their plans on their own personal data”. The PSP wants that engineers feel responsible for the quality of their work; developer should focus on personal improvement using metrics to monitor their own work and to help them to make better plans. To accomplish this, time, size, and quality measures are collected. The adoption of PSP shows that software engineers improve in estimating accuracy and early defect removal. Unfortunately, the use of PSP is not completely hassle free: the accurate manual data collection produces considerable overhead as the single forms must be filled out and a continuous “context switch” between product development and process recording is needed [Johnson et al., 2003]. Frequent “context switches” mean that the developer has to change very often on what he is currently focusing his attention. This required behavior is not only experienced as an annoying activity but could also be the cause of errors done because the engineer was not enough concentrated on the specific topic currently developed or is seen as “too intrusive for many users who desire long periods of uninterrupted focus for efficient and effective development” [Johnson et al., 2003]. Especially in an agile development, where the focus lies more in rapid value creation employees consider the creation of reports and the tracking of their development process as a wasting of time. Additionally, the manual data collection (if not carried out with the necessary motivation) is error prone so that the collected data is not always good enough to perform serious data analysis.

To overcome the shortcomings of the manual software metrics collection and analysis different ideas for automatic, non-invasive software metrics collection have been proposed, e.g. Hackystat [Johnson et al., 2003] or PROM [Sillitti et al., 2003]. These new proposals reduce the needed overhead for developers trying to adopt PSP using some kind of plug-ins attached to the single development editors. Plug-ins collect the required metrics and report them to the central server where the data is stored and analyzed. This approach has several advantages not only when used in connection with PSP: completely automatic data collection routines allow creating a consistent, structured measurement environment. Data can be collected in an unbiased, uniform way. This increases the consistency of the collected data and raises the confidence in using the collected data. Furthermore, a completely automatic and non-invasive metrics collection requires less training and less effort for the implementation of a methodology based on software metrics such as PSP. If less training is needed, it is possible to involve more people in shorter time and therefore to perform a broader data analysis to produce more reliable results. In summary, as strengths for using non-invasive measurement techniques we identified • the user is not interrupted in his work • the user does not loose time to collect metrics • the collected data is more reliable 3. WEAKNESSES Gopal et al. analyzed determinants of success of measurement programs in software development [Gopal et al., 2002] considering two variables: “use of metrics in decision making” and “increase in organizational performance”. Influencing factors are grouped into two categories: the organizational and technical context. The organizational context analyzes the commitment of the organization to implement the metrics program, the technical context the usability and perceived usefulness of the collected data. One of the conclusions of their survey is that the variables “use of metrics in decision making” and “increase in organizational performance” have a cyclic relationship. An increased use of metrics information in decision-making leads to higher organizational performance. Furthermore, an increase in organizational performance is associated with an increase in the use of metrics information in decision-making [Gopal et al., 2002]. The authors state that the “frequency with which actual metrics are collected”, “the presence of a systematic process of collecting metrics across the organization” and the “kind of metrics collected” are significant influencing factors for the use in decision-making. The mentioned results speak in favor of non-invasive metrics collection since this technique assists in the creation of a systematic, uniform, unbiased process of collecting metrics across the organization. Regarding factors that influence the “increase in organizational performance” after the introduction of measurement programs the “alignment of personnel in setting metrics goals”, “management support”, and the “institutional belief” factor have a significant impact.

According to these results, it is important for a successful introduction of a measurement program to involve employees in the performed activities, not only management but also “to increase the involvement of even non-measurement related people in setting the goals of a metrics program [Gopal et al., 2002].” The involvement of people becomes even more important in the implementation of non-invasive measurement techniques. If such tools are introduced without or not enough involvement of the participating users the automatic metric collection software can be perceived as a kind of “spyware” since it is not anymore visible what data is collected and at what moment the data collection process is active or not. This can create a counter reaction in a way that people do not want anymore to participate because they experience the measurement as a loss of control. In [Jöns and Schultheis, 2002] three forms of control are described: • Objective or manifest control: describes the real influence capabilities and is observable by others. • Subjective, perceived or cognitive control: describes the subjective view of one’s own current or future influence capabilities. It allows experiencing a high sense of control without the need to exercise it or that others observe it. • Illusionary control: is present if the experienced, subjective influence capabilities are perceived as higher as the real ones are. The initial resistance to use non-invasive measurement tools could be explained with the loss of subjective control. The measurement itself is interpreted as interference in the autonomy of the single employee and therefore seen as unfair. In order to analyze why employees in certain situations refuse to use noninvasive measurement and to identify possible solutions, we analyzed non-invasive measurement under the perspective of Self-determination theory (SDT). Self-determination theory is a motivational theory that tries to explain how people can be motivated or why people get demotivated. Motivation means “the psychological feature that arouses an organism to action toward a desired goal1” and in that sense, we want to focus on the motivation of developer to use non-invasive measurement techniques to improve their development process. SDT states that “to be self-determining means to experience a sense of choice in intitiating and regulating one’s own actions. [Deci et al., 1989]” Considering selfdetermination in a work organization where it is important to manage people, Deci et al. classify any “input affecting the initiation and regulation of intentional behaviour” as informational or controlling: “experiencing an input as information fosters selfdetermination, whereas experiencing it as controlling diminishes self-determination. [Deci et al., 2001]” From this perspective the initial PSP, focusing on the improvement of the single developer, giving feedback about his performance seems to be a more informational process so that the perceived self determination is enforced. Non-invasive measurement tools can be easily used in a controlling way and is therefore perhaps perceived as controlling and in consequence, not easily accepted by employees. Following self-determination theory, we can try to classify non-invasive measurement techniques in two groups: tools that collect data with the aim to give informational feedback and tools that collect data for controlling purposes.

1

http://www.hyperdictionary.com

Trying to monitor ongoing processes with the aim to identify problems in the process and to come up with possible solutions before the problem becomes critical is a legitimate goal for the management of a company. Monitoring employees to verify that the performed work meets the expectations is a known issue in the management literature and known as “principal-agent theory”: „Information enters the picture because the principal needs to know if the agents are carrying out the desired actions or at least are delivering the desired outcomes or results [Ellerman, 2000].” Monitoring employees is an extrinsic form of motivation, employees are induced to behave according to the expected behavior and are observed for this reason. This type of monitoring is a form of external intervention intended to change a person’s behavior and therefore poses a thread to autonomy: “… Surveillance displaces autonomy, mistrust undermines self-regard, absence of support and help minimizes achievement, likelihood of punishment for noncompliance reduces risk-taking and innovation, rigidity of standards and administrative procedures precludes the individual's use of his own know-how. (McGregor 1967, cited in [Ellerman, 2000])” The final suggestion is to focus on informational feedback that empowers people: “If such summary data indicate to the manager that something is wrong within the organizational unit for which he is responsible, he will turn not to staff [i.e., not to the accounting or finance staff providing the data], but to his subordinates for help in analyzing the problem and correcting it. He will not assign staff "policemen" the task of locating the "culprit." If his subordinates have data for controlling their own jobs, the likelihood is that they will already have spotted and either corrected the difficulty themselves or sought help in doing so (McGregor 1960, cited in [Ellerman, 2000]).“ To conclude this discussion, we can say that the acceptance of non-invasive techniques is highly dependent on how it is introduced, for what purpose the data is used and how much employees thrust management to use the data in a “noncontrolling” way. Coming back to the PSP example, we identified another weakness: the process of data gathering completely changes compared to manual data collection. In processes where the data collection itself is important because of the resulting learning effects, this can be a problem. Part of the personal software process is that the developer him or herself has to write down how he or she develops, which mistakes are made and how the development proceeds. If this is done automatically, the developer does not know this anymore, but has to analyze and understand the collected data ex-post. This again consumes time so that probably during times where many other (higher priority) tasks are due, the review of the collected data will not be done anymore. If non-invasive techniques help to save more time even considering this disadvantage will be a question addressed by empirical studies in the future work. In summary, as weaknesses for using non-invasive measurement techniques we identified • the psychological impediments in implementing non-invasive measurement techniques, • possible loosing of desired effects in processes where the data gathering process itself has a certain purpose such as rising learning effects.

4. OPPORTUNITIES Once in place, automatic metrics collection is also useful to perform internal and external benchmarking. Benchmarking means to find ways to improve the current performance by comparing the own performance with better examples [Steinbuch, 1998]. Benchmarking is can be categorized as internal and external benchmarking, internal benchmarking means comparing the internal performance measurements with past internal measurements and analyzing the change; external benchmarking means to compare the internal performance measurements with external entities. External benchmarking is not always possible, since companies usually do not disclose their metrics data, as it could be possible to deduce their software development process, i.e. their strategic advantage over competitors. It is possible to compare the internal metrics with “ideal values” published in literature or with published data like the source code of open source projects. Even if internal benchmarking is often criticized as being useless (“it means to compare the miserableness of the past with the current miserableness” [Steinbuch, 1998]), we think that is essential to measure changes in process or product metrics, to verify the effects implemented changes or to analyze the roots of changes in performance or quality. Another opportunity of using non-invasive data collection is the support for management that this technique can give in the field of agile methods. Collected process data allows management to increase the so called “line of sight” of their projects in the sense that, performed activities, time spent in the different project phases, the costs of different features or parts of projects can be calculated and analyzed. This is useful for every software development methodology but in particular for agile development techniques such as Extreme Programming (XP). XP does not particularly address management, in its recommendations [Abrahamsson et al., 2002]. The adoption of XP suffers from this circumstance in the sense that managers have the feeling of somehow loosing the control of their projects. Non-invasive data collection techniques can help to fill missing parts in the methodology and therefore raise the acceptance and suitability of agile methods. Considering the CMM viewpoint of “Quantitative process management”2, “Software quality management”3 and “Process change management”4, XP does not address these aspects and non-invasive metrics collection is a way to implement them in an agile way. The whole idea of non-invasive measurement has different effects, depending on the size of the company: • For larger companies, it helps to lower the costs of a metrics collection system that is already in place. • For small- and medium sized companies, this approach can help to implement a metrics collection system, since probably before it was too costly implement a manual data collection and therefore it was not possible to

2

“Quantitatively control the performance of the software project’s process. Software process performance represents the actual results achieved from following a software process.” [15] 3 “Quantify the quality of the project’s software product and achieve specific quality goals” [15] 4 “Continually improve the organization’s software processes with the goal of improving software quality, increasing productivity, and decreasing the product-development cycle time” [15]

In general, the increased precision of the data allows to perform types of analyses that could not be performed before. In summary, as opportunities for using non-invasive measurement techniques we identified • the possibility to perform internal and external benchmarking, • to support management in order to allow management to better understand the ongoing work and to make decisions based on real, empirically collected data, • to enable also smaller companies to use software metrics in order to improve their software development process, and • to increase the acceptance of agile methodologies allowing management to oversee the progress of the project. 5. THREATS A threat to the concept of non-invasive measurement - in the sense that it threatens the successful and useful implementation - is a misuse of the collected data. As already mentioned in the “Weakness” section, if the data is used in a controlling way this will probably decrease the motivation. The “Equity Theory of Motivation” (J.S. Adams) tries to explain which factors influence workers’ motivation or how it changes over time. It considers the inputs and outputs that the single employer invests or gets from the relationship with the company. Adams states that employees need to feel that there is a fair balance between inputs and outputs. The weight that is given to both factors, inputs and outputs, is subjective, it depends on the single employee. Inputs are typically effort, commitment, skill, flexibility, tolerance, etc., outputs are typically financial rewards, benefits, responsibility, sense of achievement and advancement, etc. Considering this theory, the misuse of non-invasive measurement techniques can have different outcomes: • Output are higher or equal than inputs: if, e.g. the payment is perceived as high enough or the working conditions explain the need to control what people are doing, so that a controlling mechanism is justified and perceived as fair the overall working motivation will not change. • Inputs are higher that outputs: if the working situation is already critical, in the sense that employees do not trust management, than the introduction of a noninvasive measurement system will probably worsen the situation. It depends also on the situation, how employees interpret the controlling use of non-invasive techniques. 6. CONCLUSION We tried to analyze non-invasive measurement techniques from four viewpoints: Strengths, Weaknesses, Opportunities, and Threats. We identified several strengths that speak in favor for non-invasive measurement and several weaknesses that are inherent to the approach itself. The threats that could lead to demotivation of employees because of the excessive control that is exercised over them have to be considered and the

introduction and use of non-invasive measurement techniques has to be adapted accordingly. We think that the opportunities are promising, and will be a considerable improvement, especially for agile environments and small and medium sized companies. 7. REFERENCES Deci E.L., Connell J., Ryan R.M., Self-Determination in a Work Organization. Journal of Applied Psychology, Vol. 74, No. 4, pp. 580-590, 1989. Deci E.L., Ryan R.M., Gagné M., Leone D., Usunov J., Kornazheva B. Need satisfaction, motivation, and well-being in the work organizations of a former Eastern Bloc country. Personality and Social Psychology Bulletin, Vol. 27, No. 8, pp. 930-942, 2001. Ellermann D., The Theory Y Case Against Panopticism: Toward a Critique of the Economic Theory of Agency. Transparency and the global Political Economy Workshop, University of Warwick, UK, February 2000. Fenton N.E., Pfleeger S.L., Software Metrics - A Rigorous & Practical Approach, PWS Publishing Company, Boston, 1997. Gopal A., Krishnan M.S., Mukhopadhyay T., Goldenson D.R., Measurement Programs in Software Development. IEEE Transactions on Software Engineering, Vol. 28, No. 9, pp. 863-875, 2002. Abrahamsson P., Salo O., Ronkainen J., Warsta J., Agile software develpment methods. Review and analysis, VTT Publications, Oulu, 2002. Humphrey W.S, The Personal Software Process - Technical Report. Retrieved June 11, 2004, from Carnegie Mellon University, Software Engineering Institute Web site: http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr022.pdf, 2000 Jöns I., Schultheis D, Kontrolle als Prädiktor für das Erleben von Fusionsprozessen. Mannheimer Beiträge zur Wirtschafts- und Organisationspsychologie, Vol. 2, pp. 3141, 2002. Johnson P.M., Kou H., Agustin J.M., Chan C., Moore C.A., Miglani J., Zhen S., Doane W.E., Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined. Proceedings of the 2003 International Conference on Software Engineering. Portland, Oregon, 2003 Steinbuch P.A. (Ed), Prozessorganisation - Business reengineering - Beispiel R3, Friedrich Kiehl Verlag, Ludwigshafen (Rhein), 1998. Sillitti A., Janes A., Succi G., Vernazza T., Collecting, Integrating and Analyzing Software Metrics and Personal Software Process Data. Proceedings EUROMICRO 2003, pp. 336-342, 2003.