Supporting CMMI assessment using distributed, non ...

Supporting CMMI assessment using distributed, non-invasive measurement and process mining Saulius Astromskis, Andrea Janes, Alberto Sillitti, and Giancarlo Succi Centre for Applied Software Engineering Free University of Bozen/Bolzano, Bolzano, Italy {saulius.astromskis, andrea.janes, alberto.sillitti, giancarlo succi}@unibz.it

Abstract The reputation of lightweight software development processes such as Agile and Lean is damaged by practitioners that claim benefits of such processes that are not true. Teams that want to demonstrate their seriousness, could benefit from matching their processes to the CMMI model, a recognized model by the industry and the public administration. CMMI stands for Capability Maturity Model Integration and provides a reference model to improve and evaluate processes according to their maturity based on best practices. On the other hand, particulary in a lightweight software development process, the costs of a CMMI appraisal are hard to justify since its advantages are not directly related to the creation of value for the customer. This paper presents Jidoka4CMMI, a tool to – once a CMMI appraisal has been conducted – allow to document the assessment criteria in form of executable test cases. The test cases, and so the CMMI appraisal, can be repeated anytime, without additional costs. The use of Jidoka4CMMI increases the benefits of conducting a CMMI appraisal. We hope that this encourages practitioners using lightweight software development processes to assess their processes using a CMMI model.

1. Introduction Promoters of so called lightweight software development processes (e.g., Extreme Programming, Scrum, Lean software development) face the problem that luring consultants claim that if you use a lightweight development process, you do not need to document anything, can solve every problem, produce bug-free soft-

ware, turn your customers into friends, maintain deadlines easily, and will have a 9 to 5 job [1]. Moreover, some companies call themselves Agile, Lean, etc., not because they pursue agility, but because it is fashionable to do so [2]. The picture of a young, dynamic, flexible, and lean team is what customers expect and, as a consequence, what companies convey within their marketing material. In summary, the described development damages the reputation of lightweight software development processes. The conscientious part of the community needs to develop methods and tools to objectively assess their method to produce software and hence, to understand in which context it works and in which not. The availability of such methods and tools will help to distinguish the serious practitioner from the quacksalver. Agile and Lean teams could benefit from having a reference model, accepted by the industry and the public administration, that defines what it means to develop “good” software. Teams that adhere to such model could claim that what they do corresponds to the best practices of software development. The CMMI for Development is such a model that describes typical elements of effective processes [3]. Therefore, a proposed solution to increase the reputation of Agile and Lean teams is to embrace both: CMMI and Agile [3]. The comparison of the development process of a given organization with the recommendations of the CMMI helps to evaluate the maturity of the analyzed process, find gaps between the performed activities and the suggested ones, identify improvement possibilities, and demonstrate their maturity towards criticizers using a model recognized throughout the industry and public administration. According to Hillel Glazer, one of the autors of the paper “CMMI or Agile, Why not embrace both?” [3], a CMMI appraisal for a small business with less than 100 developers that is completely ready to be appraised

can cost from $36.000 to $60.000, the exact price depends on a variety of factors. If the team needs help to prepare everything to get appraised, clients typically spend about $125.000 over about a year’s time or less [4]. Agile and Lean practitioners are attentive on the elimination of any unnecessary activities to improve their efficiency. Hence, there is a certain level of skepticism towards the CMMI since its value is indirect: it does not directly contribute to the creation of value for the client. Only if team is able to use the results of the appraisal to optimize the software development process, and finally the software it produces, it was worth it. For many, a CMMI appraisal represents an investment that, considering the costs, is too risky. We want to encourage practitioners using lightweight software development processes (this are our envisioned users) to assess their processes from the CMMI perspective. Therefore we propose a tool that once configured, constantly evaluates the CMMI compliance of the ongoing processes, without generating additional costs. The goal of our tool, called Jidoka4CMMI, is – similar to JUnit1 or Hudson2 – to provide a framework to define test cases that verify if a process conforms the CMMI recommendations. As JUnit or Hudson, our tool is based on the Jidoka principle (see Section 2.1), i.e, once configured, it does not require any effort to assess a process.

2. Background 2.1. Jidoka One of the pillars of Lean Management is “Jidoka”, the introduction of quality inside the production process and product. Jidoka is often translated with “autonomation” or “automation with a human mind” and is usually illustrated making the example of a machine that can detect a problem with the produced output and interrupt production automatically rather than continue to run and produce bad output [5]. Some authors translate Jidoka with “quality-at-thesource” [6] meaning that quality is inherent in the production system, which is in contrast to the traditional approach that checks quality after the process. In essence Jidoka is composed by two parts: a mechanism to detect problems, i.e., abnormalities or defects, and a mechanism to notify when a problem occurs [7]. 1 http://www.junit.org 2 http://hudson-ci.org

Figure 1 Software illustrates the idea: produced parts are dedevelopment process livered into a box by an assembly line. On the way a switch – a mechanism to detect a problem – ensures that the parts are not to big. If the switch is enabled, the assembly line is stopped and a lamp – a mechanism to notify – informs the operator that something Non-invasive measurement is wrong. Legendwe propose to use the concept of Jidoka In this work development activity to ensure thatSoftware the processes in place correspond to the Transition from one activity to another recommendations of the CMMI. Measurement probe

Figure 1. Jidoka

2.2. CMMI The CMMI for Development is a model that describes typical elements of effective processes [3]. It is used to assess the maturity of a process. A CMMI model describes typical work products and typical practices that - if performed - indicate a certain maturity level. CMMI models are not precise process descriptions. The actual processes used in an organization depend on many factors, including application domains and organization structure and size. The process areas of a CMMI model typically do not map one to one with the processes used in the organization [8]. To automate the assessment of processes against the recommendations of the CMMI, we adopt non-invasive data collection.

2.3. Non-invasive data collection The term “non-invasiveness” originates from the medical field, in which it describes a treatment that does not require ”the entry into the living body (as by incision or by insertion of an instrument) [9]”. We use this term to describe a measurement methodology that does not require any participation by the software developer. As depicted in figure 2, Non-invasive data collection relies on a set of sensors or probes that extract the data from other systems (ac-

tive) or that get triggered by the observed events (passive) [10]. Software development process

Figure 3 contains the same elements as figure 1: a mechanism to detect a problem and a mechanism to notify everybody that a problem exists: in figure 1, the switch under the lamp detects the problem and the lamp notifies everybody. In figure 3, the measurement probes report problems to the data warehouse and the dashboard informs everybody about the current status of the system.

3. The Jidoka4CMMI tool Non-invasive measurement Legend Software development activity Transition from one activity to another Measurement probe

Figure 2. Measurement probes The measurement probe logs events and report them to a message queue over the Internet using the REST [11] protocol. We use a message queue to minimize the workload of the client computers. Developers do not want that their machines slow down because some measurement program is running in the background. By using a message queue (we use Apache ActiveMQ3 ), we can minimize the time the client machine is busy uploading data. The data are then read from the message queue, processed, and inserted into the data warehouse. Due to the large amount of data we collect, we use a NoSQL database, Apache Cassandra4 . The data are then extracted from the data warehouse and processed to feed it to the process mining algorithm and its results visualized on a dashboard (see figure 3). Laptop

Laptop

Workstation Workstation

Message Queue Data warehouse Legend Computer Data transfer

Dashboard

Data processing and transfer

Figure 3. Data flow in the Jidoka4CMMI tool

The software engineering challenge we propose to address, is to allow practitioners to define test cases for their processes that are monitored by Jidoka4CMMI. Such a test case defines criteria to detect a problem and a mechanism to notify stakeholders when a problem occurs. An analogy to our tool is to set up a mouse trap: we want to set it up so that it snaps when there is a problem. The architecture of our system is shown in figure 4. The measurement probes continuously extract data from the development process about the employed resources, the produced output, and the carried out activities. Warnings

Rules Engine

Notifier

Rule execution Data Storage

Measurement

Measurement probes

development process Figure 4.Software Component diagram of the Jidoka4CMMI tool

We implemented measurement probes that record all interactions developer have with popular integrated Non-invasive measurement development environments such as Eclipse5 or Mi6 crosoft VisualLegend Studio , office automation suites such as 7 activity Apache Open Software Officedevelopment or Microsoft Office8 , and other Transition from one activity to another tools developers frequently use (e.g., web browsers or Measurement probe chat applications). Developers interested in adding additional data to the system can develop a plugin implementing a defined interface. 5 http://www.eclipse.org 6 http://www.microsoft.com/visualstudio

3 http://activemq.apache.org

7 http://www.openoffice.org

4 http://cassandra.apache.org

8 http://office.microsoft.com

The collected data (a log of all detected events) are stored in the Data Storage. The Rules Engine retrieves the data from the data storage, analyses them, and generates warnings if a rule is violated. The Notifier executes the actions to perform if a rule is violated.

3.1. How to use Jidoka4CMMI Typically, a CMMI appraisal team identifies practice implementation indicators, i.e., evidence that provides a basis for verification of the activity or practice implementation. These practice implementation indicators (since version 1.3 called “objective evidence”) indicate how well a practice was implemented [3]. The concept of “objective evidence” relies on the idea that the implementation of a practice results in “footprints” that can be observed and that prove the implementation of the practice. A user that wants to write a test case for a process using Jidoka4CMMI has to: 1. specify a Goal Question Metric (GQM) model [12] that defines what to collect, how to collect it, and why to collect it; 2. specify the conditions that the collected “footprints” have to fulfill using JavaScript9 or Java10 ; 3. define how a detection of non-conformance has to be notified. In step 1) we require the user not only to specify what data to collect, but also to document which question this data answers and how to interprete the answer of the question. The result of the test case is then linked back to the measurement goal to understand what the result means. The CMMI model for Development specifies requirements for processes. Jidoka4CMMI contains process mining support to allow the user to evaluate if a process conforms to the defined process or, if we find out that the defined process is not followed, to discover the real process. Process mining is a research discipline that sits between machine learning and data mining on the one hand, and process modeling and analysis on the other hand [13]. Process mining supports: • Process discovery, in which an algorithm generates a process model based on the data coming from event logs. 9 http://www.ecmascript.org 10 http://www.java.com

• Process conformance, in which an algorithm locate discrepancies between the actual and the ideal process. Process conformance requires the definition of the ideal model in addition to the event logs. • Process enhancement, in which the current process is analyzed to find opportunities to enrich the model, e.g., to make it reflect the reality better or to align it to the strategy of the organization. The detection of non-conformance can be notified via e-mail or in a user configurable dashboard. The dashboard summarizes the results of the test cases.

4. Example The initial validation of our approach occurred through the implementation of proof-of-concepts. In this paper we present one. Each proof-of-concept consists of five parts: the description of the context, the GQM model that describes why, how, and which data is collected, examples of the collected data, the description how the data is processed, and examples of possible results of the analysis. The following proof-of-concept demonstrates how to write a test case for the specific practice 1.1, “Objectively Evaluate Processes” that is part of the “Process and Product Quality Assurance” process area [8]. In this proof-of-concept the process to evaluate is the requirements management process. Each requirement has to go through the states “Backlog”, “Selected”, “Development”, “Done”, “Deployment”, and “Live”. The product owner adds ideas for requirements to the requirement backlog. Requirements that should be picked up by developers are set to the status “Selected”. When a developer begins implementing a requirement he or she sets the status of the requirement to “Development” and after finishing to “Done”. When a features gets deployed to the live system, its status is changed to “Deployment” and finally, after finishing the deployment to “Live”. Figure 5 shows the Goal Question Metric model that specifies how Jidoka4CMMI should verify if the requirements management process fulfills the specific practice 1.1 of the “Process and Product Quality Assurance” process area. The model verifies if requirements follow the defined process through process mining. The log of status changes is compared with the defined process model using process conformance checking. Moreover, using process discovery, the number of loops that requirements performed is evaluated. If requirements are moved from “Deployment” to “Development” too often, it might indicate that the team has difficulties with

Analyze requirements management (object) for the purpose of conformance checking (why) with respect to the defined process (focus) from the point of view of the project manager (who) in the context of the software development department (where). Goal Do requirements follow the defined process?

Do we have requirements that take longer than expected to implement?

Question

Question

Metric

Metric

Process conformance between the defined process model and the change log of the requirements

Number of loops a requirement performs between the status “Deployment” and “Development”

Figure 5. An excerpt of a Goal Question Metric model describing how a given specific practice is evaluated

some type of requirements, that the requirements are not precise enough, etc. The data used to evaluate this GQM model comes from a probe that logs all status changes of the requirements. The probe reports data as in table 1. Table 1. Example data Requirement Status change 25 Backlog to Selected 26 Development to Deployment 03.01.2012 25 Selected to Development Timestamp 01.01.2012 02.01.2012

The data is passed to the process mining engine that evaluates the metrics defined in the GQM model. To extract the visual process model from the event log, we use an algorithm called the Heuristics miner because this algorithm is suited for real world data [14]. The Heuristics miner algorithm is able to deal with noisy data [15], ignores rare events and activities, and therefore generates simple models representing only the most common behavior. Examples of evaluation results are:

• Requirement 3 did not follow the defined process, it was directly added with the status “Development”.

5. State of the art From a bird’s-eye perspective, a CMMI appraisal consists of two steps: collecting data and interpreting it according to the CMMI model [16]. Past research can be classified into these two groups, i.e, methods and tools that a) support the data collection or b) the data interpretation. Data collection tools alleviate the burden of manual data collection through forms and templates, extract data generated by already existing systems, or infer properties about the executed processes using techniques based on data mining (e.g., [17, 18, 19, 20]). Data interpretation tools help to infer the achieved maturity level interpreting the collected data (e.g., [21, 22, 23, 24]). Our work differs from previous works in that it combines non-invasive data collection and process mining to support the maturity assessment of an Agile or Lean organization.

6. Conclusion In this work, we present a novel scenario in the assessment of light weight software development processes. We show, how quality assurance can be built into the process, as proposed by lean thinking. We give an example of how to integrate the continuous monitoring of produced artifacts, ongoing development activities, and consumed resources with the production process together with the knowledge about unacceptable values of the monitored data.

References [1] A. Janes and G. Succi, “The dark side of agile software development,” in Proceedings of the ACM international symposium on New ideas, new paradigms, and reflections on programming and software, ser. Onward! ’12. New York, NY, USA: ACM, 2012, pp. 215–228.

• The process conforms to the defined criteria

[2] M. Ceschi, A. Sillitti, G. Succi, and S. D. Panfilis, “Project management in planbased and agile companies,” IEEE Software, pp. 21–27, 2005.

• Requirement 25 exceeded the limit for loops between “Deployment” and “Development”.

[3] H. Glazer, J. Dalton, D. Anderson, M. Konrad, and S. Shrum, “Cmmi or agile: Why not embrace

both!” Software Engineering Institute, Carnegie Mellon University, Tech. Rep., 2008. [4] H. Glazer. (2011, Feb) Roughly how much does an cmmi evaluation cost for a small business (less than 100 developers)? http://www.quora.com/Roughly-how-much-doesan-CMMI-evaluation-cost-for-a-small-busin/essless-than-100-developers. [5] T. Ohno, Toyota Production System: Beyond Large-Scale Production. Productivity Press, 1988. [6] C. Standard and D. Davis, Running todays factory: a proven strategy for lean manufacturing. Hanser Gardner Publications, 1999. [7] Y. Monden, Toyota Production System, 2nd ed. Industrial Engineering Press, 1993. R [8] CMMI Product Team, “Cmmi for development, version 1.3,” Software Engineering Institute, Carnegie Mellon University, Tech. Rep., 2010.

[9] Merriam Webster, “Definition of invasive,” http://www.merriam-webster.com/dictionary/ invasive, 2010. [10] A. Janes, M. Scotto, A. Sillitti, and G. Succi, “A perspective on non invasive software management,” in Instrumentation and Measurement Technology Conference, 2006. IMTC 2006. Proceedings of the IEEE, april 2006, pp. 1104 –1106. [11] R. T. Fielding and R. N. Taylor, “Principled design of the modern web architecture,” ACM Trans. Internet Technol., vol. 2, no. 2, pp. 115–150, May 2002. [12] V. R. Basili, G. Caldiera, and H. D. Rombach, “The goal question metric approach,” in Encyclopedia of Software Engineering. Wiley, 1994. [13] W. Aalst, A. Adriansyah, A. K. Medeiros, F. Arcieri, T. Baier, T. Blickle, J. C. Bose, P. Brand, R. Brandtjen, J. Buijs et al., “Process mining manifesto,” in Business Process Management Workshops, 2012, pp. 169–194. [14] W. M. P. van der Aalst, Process Mining: Discovery, Conformance and Enhancement of Business Processes, 1st ed. Springer Publishing Company, Incorporated, 2011.

[15] J. D. Weerdt, M. D. Backer, J. Vanthienen, and B. Baesens, “A multi-dimensional quality assessment of state-of-the-art process discovery algorithms using real-life event logs,” Information Systems, vol. 37, no. 7, pp. 654 – 676, 2012. [16] SCAMPI Upgrade Team, “Standard CMMI appraisal method for process improvement (SCAMPI) a, version 1.3: Method definition document,” http://www.sei.cmu.edu/library/ abstracts/reports/11hb001.cfm, 2011. [17] T. Sunetnanta, N.-O. Nobprapai, and O. Gotel, “Quantitative CMMI assessment for offshoring through the analysis of project management repositories,” in Software Engineering Approaches for Offshore and Outsourced Development. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, vol. 35, pp. 32–44. [18] I. Garcia and C. Pacheco, “Toward automated support for software process improvement initiatives in small and medium size enterprises,” in Software Engineering Research, Management and Applications 2009, J. Kacprzyk, R. Lee, and N. Ishii, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, vol. 253, pp. 51–58. [19] N. Chen, S. C. H. Hoi, and X. Xiao, “Software process evaluation: A machine learning approach,” in Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. Washington, DC, USA: IEEE Computer Society, 2011, pp. 333–342. [20] J. R. Curanas, “Process mining opportunities for cmmi assessments,” http://www.tue.nl/en/publication/ep/p/d/epuid/276454/?no cache=1, 2010. [21] Accenture. Measuring and managing the cmmi journey using gqm. http://www.sei.cmu.edu/library/abstracts/ presentations/Prasad-SEPG2006.cfm. [22] L. Buglione, “Strengthening CMMI maturity levels with a quantitative approach to root-cause analysis,” in Proceedings of the 5th Software Measurement European Forum, Milan, Italy, 2008. [23] M. West, Real process improvement using the CMMI. Auerbach Publications, 2004. [24] M. Khraiwesh, “Validation measures in CMMI,” World of Computer Science and Information Technology Journal, vol. 1, no. 2, pp. 26–33, Mar. 2011.