Maximizing Return on Investment (ROI) for

Maximizing Return on Investment (ROI) for Requirements Process Improvement Aminah Zawedde1 and Ddembe Williams2 1

School of Computing and Informatics Technology, Makerere University, Uganda. 2 KCA University, Kenya. [email protected], [email protected]

Abstract. Process improvement like any other investment is carried through a decision making process based on the calculation of a healthy return on investment (ROI) which is a critical part of the business case. Methods and techniques currently used by process improvement practitioners for Requirements Process Improvement (RPI) for advising their clients on ROI worth investing for PI, however fail to provide a fair ROI worthy of investment because many of them lack a systems approach to RPI. This has been attributed to failure to provide RPI teams with the required understanding of the impact of the various factors affecting ROI on each other resulting either in over or under estimates of ROI. In this paper we present a case for holistic analysis (emphasizing the importance of analyzing the whole and the interdependence of its parts) of the key process factors for process improvement so as to result into realistic ROI. Keywords: Requirements process improvement, return on investment,cost, quality,schedule

1

Introduction

Process improvement like any other investment is carried through a decision making process based on the calculation of a healthy return on investment (ROI) [23]. The process used to estimate ROI will determine the fairness of the ROI. An accuracy level that renders a process acceptable is agreed upon because process improvements are not precise [23]. Process elements are the appropriate level for ROI calculation [14], however the ROI for process improvement is calculated over a set of process performances over a period of time. Methods and techniques currently used by process improvement practitioners for RPI fail to maximize healthy ROI [25] because many of them lack a systems approach to RPI. These approaches therefore fail to provide RPI teams with the required understanding to assess the effectiveness of the processes in estimating a more realistic range of costs and benefits. This is attributed to the inability to capture the impact of a change in one or more factors on all the factors that influence the ROI of RPI. The RPI practitioners are faced with the challenge of making informed decisions on the effectiveness of carrying out any process improvement. Recent practical experience indicates that RPI process

2

A. Zawedde, D. Williams

attributes are interlinked and dynamic, and should be analyzed in the context of the whole process instead of analyzing them in pairs. The objective of such analysis is to identify the interdependence and feedback structure amongst the process attributes [25]. When deciding how to prepare and present an ROI analysis, therefore, it is best to take into account all the potential risks that influence the decision process[5]. This can be done by use of analytical tools to visualize the interdependence and feedback amongst the process attributes. The visualization enables the RPI stakeholders to talk about and discuss the emerging system behavior as a basis for process improvement. Visualizing these feedback interactions is a prerequisite in helping the RE stakeholders gain shared understanding of the decisions made in achieving effective requirements for process improvement [25]. In this paper we do not address ROI in terms of pure profit but present a case that the analysis of key variables for process improvement can be quantified in terms of cost savings and increased productivity, which are actually two sides of the same coin since both can be reduced to the ROI factor [6]. Effective analysis of the key process attributes for process improvement can result into demonstrable ROI. In this paper we discuss the factors that influence the decision process for cost effective requirements process improvement (RPI) as well as the importance of analyzing the process factors as a whole in order to enhance the understandability and practicability of measuring the effectiveness of our method in estimating ROI for RPI over time. The rest of the paper is structured as follows: Section 2 provides an overview of the theories on which requirements process improvement is grounded; Section 3 highlights the characteristics of the proposed RPI model and describes the most suitable approach to attain this; Section 4 describes the factors that influence RPI as well as the relationships amongst these factors; Section 5 discusses the field findings from RPI in regard to the RPI variables used in practice; Section 6 presents a discussion on how decision making for RPI can be used to maximize ROI; and Section 7 discusses further research directions.

2

Theoretical Grounding of Requirements Process Improvement

The RPI process is grounded on three theories namely: the systems thinking, complexity and decision making. Systems thinking is based on the use of a holistic perspective where the system is looked at as a whole. The system represents a set of elements that are dynamically related in time and each of the elements can affect the performance of the whole system but none of them can have an independent effect on the whole system [20]. In this paper, Systems theory forms the major grounding for the research problem because RPI is majorly dynamic as a result of the interaction and interdependences between requirements processes. In this research dynamism of the requirements processes is investigated using system dynamics, a methodology based on systems theory, given its ability to model problems in a dynamic perspective rather than a static one [20]

Maximizing ROI for Requirements Process Improvement

3

[24]. The systems thinking theory is based on the assumption that structure drives the behavior of a system [22] therefore understanding the structure or the feedback loops in the structure will help one understand the system. Based on these premises, analysis of the structure of a system in a holistic manner helps the analyst understand system behavior and the relationships of the component variables. The purpose of this analysis is to help arrive at a satisfactory state of cost, quality and schedule [24]. This research is also based on complexity theory. Complexity is a property of systems made of many components that interact in a dynamic and non-linear manner. Dynamic complexity has causal relationships that influence the behavior of a system. Analysis of these relationships helps the understanding of how a change in one part of the system is likely to impact the other parts and the outputs of the system [19]. Complexity theory therefore helps this research in understanding how requirements engineering processes, themselves complex can be analyzed to attain a satisfactory state of the quality triangle. System dynamics through model creation and simulation stages contributes rigor and clarity to systems thinking while systems thinking with emphasis on eliciting information from real world participants contributes useful insights into model behaviour [8]. Applying system thinking with system dynamics aids understanding and explaining complex problems or systems by analyzing feedback loop structures of these problems or systems. The research problem is also grounded on the decision theory that is premised on having a range of alternatives from which an overall decision can be made as opposed to having one optimal solution. The none use of this theory is cited by [1] as the reason behind the lack of understandability of requirements engineering decision problems and their impact and consequences when using current RPI methods. To gain understanding of RPI, complexity theory is triangulated with systems thinking and decision theory to arrive at a satisfactory set of alternative decisions for RPI, where uncertainty among states of the system (variables) exist.

3

The Proposed RPI Model

Software system process improvement failures have raised concerns about why investments in improvements often fail to live up to the expectations. As a result, many process improvement planning processes now require some analysis of the costs and returns expected from the proposed improvements. Unfortunately, the RPI stakeholders often lack models that can guide them through such analysis. Given the relative strengths and deficiencies in the current RPI models [25], to successfully achieve a better understanding, an appropriate RPI model should have the ability to do the following: 1. Capture feedback resulting from interaction amongst the requirements engineering processes. 2. Support evaluation and synthesis of the requirements processes in a manner that promotes explanation and insight into the problem under investigation

4


in order to improve communication amongst the requirements engineering stakeholders. This is best done through visualization. 3. Carry out a holistic analysis of the RPI process variability. This holistic analysis is key to understanding the dynamics of RPI and maximizing ROI of the system under development. These characteristics can be attained using the system dynamics approach based on the evidence illustrated below. The System Dynamics Approach System Dynamics (SD) is an approach for modeling and simulation of dynamic behavior of complex systems over time[9] [12]. The complexity of a system is defined by feedback loops, non-linearity and time delays that often affect the system behavior. System dynamics models being a representation of real world situations are well suited to offer explanation and generate insights into the root causes of the behavior of complex systems. The insights generated can facilitate informed decision making before any improvements can be implemented [18]. SD adopts the big picture strategic viewpoint in capturing the overall system structure rather than that of the individual parts of the system [12] [15]. Various techniques and methods have been used to model complexity but their main drawback is that analysis becomes complex with large models [18]. System dynamics analytical tools have the ability to handle large models, identify independent loop sets within models that determine particular behavior resulting from the interaction among model components [18] [10] [9]. SD has an advantage over statistical models because of its ability to incorporate in the model important soft variables, and also yield explanation and foster understanding of systematic problems [18].

4

Dynamics of the factors that Influence Requirements Process Improvement

Seven (7) commonly used variables were identified that influence RPI: (a) Productivity of Engineers, (b) Process Capability Index, (c) Management Commitment, (d) Process Improvement Costs, (e) Customer Satisfaction, (f) Process Rigor, and (g) Errors Observed [25]. The importance of the preceding variables and why they were considered key for the reference model for RPI was justified from field studies as well as the supporting literature considering that process improvement is a factor of cost, time and resources and it is these variables that determine these factors [25]. The factors for process improvement are interrelated and therefore one has to understand how a change in one variable impacts on the rest of the variables in order to make informed decisions on an effective process improvement. The interrelationships amongst these variables are represented using a descriptive model referred to as a dynamic hypothesis [25]. A system dynamics descriptive


5

model can also be described as a causal loop diagram that offers a basis for capturing the mental models of process improvement teams and facilitates the understanding of their decision making strategies during the process improvement [24]. Figure 1 is an illustration of the interrelationships amongst the factors that influence RPI decision making and how they are linked to each other. The linkages depict how the loop structures drive system behavior and the variables involved for each loop structure impacting on the system. Therefore analyzing the behavior of each of the loops is key to understanding the impact of changes in one or more variables on system behavior and the acceptable limits within which cost effective ROI can be achieved for a set of variables. Figure 1 is a visualization of the interrelationship amongst the key variables for RPI as validated based on industrial experiences of RPI practitioners.

Productivity of Engineers (A) +

-

+ +

- Errors Observed (G)

Process Capability Index (B) + +

Process Rigor (F) +

+ Process Improvement Cost (D)

-

Customer Satisfaction (E) -

+

Management Commitment (C)

Fig. 1. The Descriptive Feedback Model for RPI validated by RPI experts

A brief description of the RPI variables as adapted from [25] is as follows: (A) Productivity of Requirements Engineers: This is the rate at which requirements engineers deliver requirements specifications as agreed upon by the requirements engineering stakeholders. (B) Process Capability Index: This is the potential of a process to meet its specifications. The higher the index, the more capable the process is. (C) Management Commitment: This is the extent to which top management is willing to participate in requirements process improvement projects.

6


(D) Process Improvement Cost: This refers to the total cost of resources in terms of wages, documentation, training technology and initial set-up costs. (E) Customer Satisfaction: This is the measure of the degree to which a product or service meets the customer’s expectations. (F) Process Rigor: This is the level of thoroughness adhered to established standards when carrying out or implementing process improvements. (G) Errors Observed: This is the total number of defects identified by the requirements engineering stakeholders during the review process. An example of one of the loops that illustrates RPI is goes through A-G-D-CF-B-A. In this loop an increase in the the Productivity of Engineers (A) will cause a reduction in the Process Improvement Cost (D) after a time delay. A reduction in the costs will in turn trigger the interest of Management Commitment (C) which in turn will increase the Process Rigor (F) which is the adherence to the standards and procedures that the requirements engineers follow during process improvement. An increase in the Process Rigor increases the maturity level of the organization in terms of the Process Capability Index (B) feeding back into further increasing the process productivity of requirements engineers (A). This demonstrates the need to analyze these feedback loop structures in a holistic manner. The double line marks on some of the relationship arrows in Figure 1 indicate that there is a time delay between the variables. These indicate that the effect of that relationship is obtained over time. For continuous process improvement the feedback loops have to be balanced to avoid causing a system burn out or end in chaos. The descriptive model in Figure 1 demonstrates that System dynamics uses feedback loops and time delays to represent complex systems. Causal loop diagrams representing feedback loops are used for conceptualizing the system structure of a complex system and for communicating model-based insights. These feedback descriptive models have explanatory power and insight to aid understanding to RE stakeholders. Despite the difficulty in most approaches to convert qualitative data into quantitative information, this is a key strength of the system dynamics methodology. Inherent to the system dynamics methodology is the ability to convert qualitative data into quantitative values to aid better understanding [24] and this has been seen over the years as the key strength and advantage of using the system dynamics methodology [9]. System dynamics techniques [9] and procedures make it possible to effectively model quantitative data based on qualitative models. This is done methodically starting with mental models through causal link relationships and then use of reference modes. Analysis of the descriptive model offers a useful basis for identifying a number of deductions that were drawn from it [25]. The following deductions were derived from the dynamic hypothesis in Figure 1. The model was presented to process improvement experts, quality assurance experts, requirements engineers and software systems project managers who confirmed the deductions from the model and showed how the descriptive model is applicable in industrial practice.


7

1. The more process rigor in the organization’s process improvement, the greater the decline of errors observed. 2. The extent of decline in the process capability index is dependent on process rigor and the level of management commitment. 3. The extent of decline in the productivity of engineers will be moderated by the technological rigor of tools and commitment from senior managers with the organization 4. The extent of decline in management commitment to RPI will be dependent on the process improvement costs, and the productivity of engineers and feedback from customer satisfaction. 5. The extent of decline in process improvement costs is dependent on the number of errors observed and productivity of engineers. 6. The extent of decline in customer satisfaction is dependent on errors observed during a review. The experts suggested that this is depend on the measure of the gap between what is delivered to the customers and what was expected.

5

Research Design

The Delphi method which is a structured group communication method was used for this study to determine the key factors that influence RPI as well as the relationships among these factors. The main source of information required subjective judgment from experts and such results are acquired using the Delphi method that has been used as one of the best consensus reaching and qualitative methodologies. For this research experts’ opinion was required through group consensus to help in the validation of the most important variables for requirements process improvement as well as the relationships that arise as a result of the interdependence amongst these variables. The target audience included process improvement experts, quality assurance managers, requirements engineers and software systems project managers who had several years of experience in process improvements and software project management. Over 50 experts who were globally located participated in study. 5.1

Data Collection and Analysis

The primary goal in this study was to confirm the relevance of the qualitycost variables and the dynamic relationships considered for RPI in practice as a basis for the development of a simulation model for RPI. This was based on expert opinion. Experts were issued with questionnaires that had cover letters for guidance to respondent’s understanding of the questionnaire, indication of the amount of time required to complete the questionnaire and a summary of the study’s objectives. For this research, we used three rounds of the Delphi method to reach a stable level of consensus. The experts were provided with feedback on how they rated each of the RPI variables from the first round. This enabled the experts to re-evaluate their opinions in view of any additional information provided by their peers. In order to deal with any exceptions, respondents were

8


asked to justify their evaluation of a variable if there was any significant difference between their evaluation and the rest of the respondents’ evaluation. The responses were analyzed in detail to draw conclusions amongst the experts’ opinions on RPI. The results were summarized from open-ended questions and from the notes drawn from the feedback discussions with the experts on which the level of consensus was attained. Due to the qualitative nature of the data, there was no separate statistical analysis that was done. 90% of the requirements engineering experts that were interviewed agreed that the descriptive model for RPI was a good representation of the variables for RPI. A few alterations were made to the descriptive model that was initially sent to the experts and through a series of discussions they came to a consensus that the variables and relationships between the variables of Figure 1 are a good representation of the factors that influence RPI as well as the interrelationships amongst the factors.

6

Discussion

Based on these results it is proposed that any organization planning to embark on RPI should consider the factors that have been confirmed as important for influencing RPI as well as the relationships amongst these factors. At this initial stage of system dynamics modeling, we have been able to identify key variables and their feedback loops that are responsible for the software systems project behavior. As work in progress we are working on a full fledged system dynamics simulation model which will enable us to determine the strengths of the relationships and thus the relative importance of each key factor presented in this paper. In order to improve on the SD simulation results, we further propose combining SD and Statistical Process Control (SPC) to attain an effective process improvement framework that will enable us maximize ROI for RPI. SPC is a statistical technique used for monitoring and evaluating process performance over time. SPC determines the stability or instability of a process by discriminating between common cause variations that result from normal interactions amongst system components and assignable cause variations that are as a result of external factors to the process [25]. Using a few data points, SPC can dynamically determine an upper and lower control limit of acceptable process performance variability. These control limits act as signals or decision rules, and give process improvement teams information about the process and its state of control. SPC does this through visualization to establish operational limits for acceptable process variation. The visualization is in form of control charts that aid in suggesting whether corrective action can be taken to improve quality or in assurance that a process has the satisfactory quality [25]. The output from the control charts will helps RPI stakeholders in predicting results which enable preparation of achievable plans, meeting cost estimates, scheduling commitments and delivering the required product quality within the acceptable limits [4].


7

9

Summary and Further Research Directions

The contribution of this paper lies in providing empirical evidence on the key factors for process improvement as well as the relationships amongst these variables. Integrating SPC within SD is expected to help RE stakeholders gain insights about the RPI feedback structure and be able to make appropriate decisions that will enable them maximize ROI for RPI. Furthermore, it is also expected to help carry out assessment of the potential effects of the high leverage points on the stability of the interacting requirements processes directly from the control charts leading to attainment of cost effective systems under development. More still, RE stakeholders will have ownership of the RPI process together with its results, and implement the changes recommended by the tool [10].

References 1. Alenljung, B.: Envisioning a Future Decision Support System for Requirements Engineering - A Holistic and Human-centered Perspective. Department of Computer and Information Science, Linkopings Universitet, 2008 2. Baldassare, T., Baffoli, N., Caivano, D., Visaggio, G.: Managing Software Process Improvement (SPI) through Statistical Process Control (SPC). In: Bomarius, F. and Iida, H. (Eds.): PROFES, LNCS 3009, 30-34, (2004) 3. Brinkkemper, S., van de Weerd, I., Saeki, M., Versendaal, J. : Process Improvement in Requirements Management: A Method Engineering Approach. In: Proceedings of REFSQ, 6-22, (2008). 4. Caivano, D.: Continuous Software Process Improvement through Statistical Process Control. In: Proceedings of 9th European CSMR ’05, (2005) 5. 6. 7. Ferreira, S., Collofello, J., Shunk, D., Mackulak, G.: Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation. Systems and Software, 82(10), 1568-1577 (2009) 8. Forrester, J.: System Dynamics, Systems Thinking, and Soft OR System Dynamics Review, 10, 1-14, (1994) 9. Forrester, J.W.: System Dynamics and a lesson for 35 years. Sloan School of Management M.I.T, Cambridge, MA, USA (1991) 10. Georgantzas, N., Fraser, J., Tugsuz, E.: Bipartisan process improvement in polymer coating: Combining SD with SPC. System Dynamics, 2, 502-511 (1995) 11. Gustafsson, T., Ollila, M.: Expert consultation in the preparation of a national technology programme. Systems Analysis Laboratory, Helsinki University of Tech, Sept (2003) 12. Harris, B., Williams, B.: System Dynamics Methodology. W.K. Kellogg Foundation, (2005) 13. Okoli, C. and Pawlowski, S.: The Delphi method as a research tool: an example, design considerations and applications. Journal of Information and Management, 42, pp.15-29. (2004) 14. J.Perry, Y.Kim, P.Lenzen, S.Madison, D.Sims.: Planning for Return on Investment for CMMI Process Improvement (2010) 15. Rafferty, M.: Reductionism, Holism and System Dynamics. In: 8th SD PhD Colloquium, (2007)

10


16. Rico, D.F.: ROI of Software Process Improvement: Metrics for Project Manaers and Software Engineers. ISBN 1-932159-24-X, (2004) 17. Ruhe, G., Eberlein, A., Pfhal, D.: Quantitative WinWin- A New Method for Decision Support in Requirements Negotiation. In: SEKE, Italy, July, (2002) 18. Saurabh, K.: Software Development and Testing: A System Dynamics Simulation and Modeling Approach. In: 9th WSEAS-SEPADS ’10, February 20-22, (2010) 19. Senge, P. The Fifth discipline: The Art and Practice of the Learning Organization Doubleday, 1990 20. Sice, P., French, I.: A holistic frame-of-reference for modeling social system The Journal of Systems and Software, 35, 851-864 (2006) 21. Statz, J.: Measure of Process Improvement. Technical Report on Practical Software and Systems Measurement, (2005) 22. Sterman, J.: Business Dynamics: System thinking and modeling for a complex world Irwin Professional Publications, (2000) 23. van Solingen, R.: Measuring the ROI of Software Process Improvement, IEEE Computer Society, 21, 32-38, (2004) 24. Williams, D.: Challenges of System Dynamics to Deliver RE Projects: faster, better and cheaper. In: 21st Int. Conf. of the System Dynamics Society, (2003) 25. Zawedde, A.S., Klabbers, M.D., Williams, D., Brand, M.G.J. van den.: Understanding the dynamics of requirements process improvement : a new approach. PROFES, 6759, 276-290, (2011)