2009 35th Euromicro Conference on Software Engineering and Advanced Applications
Implementation of a Software Quality Improvement Project in an SME: A Before and After Comparison Ayşe Tosun1, Ayşe Bener2
Burak Turhan
Software Research Laboratory (SoftLab), Department of Computer Engineering, Boğaziçi University, Istanbul, Turkey 1
[email protected],
[email protected]
Institute for Information Technology, National Research Council, Ottawa, Canada
[email protected]
Abstract— Software process management is a crucial task in small and medium size enterprises (SME) due to scarce resources, small development teams and heavy workload. We have worked with an SME to institutionalize process improvement practices, aligned with CMMI maturity levels 2 and 3. Main goals of this project were to achieve effective resource management and increase the product quality with fewer defects. The project had two phases: Phase I conducts an “as-is analysis” in order to set well-defined processes. Phase II measures the software reliability in terms of time/ effort allocation and defect rates. Our before and after analysis showed that: a) time allocation for requirements, coding and testing steps improved from (4%, 31%, 65%) to (47%, 20%, 33%), b) defect rates decreased from 11% to 6.5%, c) estimated testing effort also decreased from 47% to 30%. Keywords-software process improvement, case study, small and medium size enterprise, CMMI.
I.
INTRODUCTION
Companies would like to control and manage their complex development lifecycles and, hence, they take measures to institutionalize their process improvement practices. Capability Maturity Model Integration (CMMI) provides standard process disciplines for a company to smoothly achieve its business goals [5]. According to many researchers, CMMI represents a “common sense engineering” to software process improvement with its maturity levels, key practices and goals [1, 9, 11, 17]. CMMI has been an effective process improvement model for not only big companies, but also small and medium size enterprises (SME) [7, 8, 15]. SMEs are more customer-driven and their main goal is to respond customer requirements quickly. They have generally a single, complex and modular product with low cost updates from the internet [10]. The development staff usually consists of more or less 5 people who are highly experienced and qualified [10]. Intrapersonal communication is very strong and open in SMEs. The employees are proud to deliver high quality and reliable software in short cycle times [15]. Additionally, small organizations preserve their flexibility when they need to make changes in their processes [15]. On the other hand, SMEs face many challenges. They heavily rely on engineers rather than processes. They work with tight budgets and resources. Considering that they employ highly qualified and experienced people, their unit cost of operation is quite high and it becomes an important threat 978-0-7695-3784-9/09 $26.00 $25.00 © 2009 IEEE DOI 10.1109/SEAA.2009.52
to their growth. Therefore, process improvement and management models, such as CMMI, should be seen as an opportunity for small companies to align their business objectives with practice [1, 8, 17]. In this paper we present a software quality improvement project (SQIP) and its impacts on a company, Bilmed, which has been operating in health care industry for 18 years in Turkey. We have seen that the company suffered from inefficient software processes, i.e. long coding and testing times and unpredictable defect rates. Thus, we have basically addressed the issue of software quality in the company as the maturity of their processes. We have benefited from specific practices and goals of CMMI Maturity Levels 2 and 3 in order to define, implement and measure Bilmed’s key process areas, policies and procedures [5]. We then conducted an internal appraisal on a pilot project to see that their maturity levels map to Level 3 in requirements and change management and Level 2 in other processes. Furthermore, we compared two projects: a past project with virtually no processes and the pilot project after SQIP implementation. II.
PROBLEM STATEMENT AND MOTIVATION
Bilmed provides solutions for hospital process automation to a wide range of institutions, including very large scale public and private hospitals in Turkey. Their software, Hospital Information Management System (HIMS), serves automation of all medical devices, patient file tracking with diagnosis and treatments, lab automation, financial records and supply management. A development team of 10 people (7 seniors, 3 juniors) work for implementing the change requests and new requirements. They have a strong and coherent development team with an average 7 years of experience. They work long hours during the week days and on Saturdays. The company has major challenges that are common in small organizations, such as project management, training of a new employee, conducting peer reviews and measurement [4, 15]. Bilmed claims that they have fast and efficient processes in software development. However, these are not documented. Moreover, the deliverables depend on the heroic efforts of a few people. Company managers define the knowledge transfer as the ‘master-apprentice’ method. However, their response time for a change request is long and sometimes the level of quality of services is poor. In addition, the staff does not have standard practices on software development processes. Each project has been
203
TABLE I.
COMPANY OBJECTIVES IN LINE WITH SQIP
Management objectives Standardization of software processes Efficient resource allocation, predictable project cost and schedule Gradual integration of common methodologies Cost reduction
Goals of SQIP Definition of organizational policies, procedures and templates Implementation of software processes
Conflict resolution with customers Reduce training effort
Requirements management, documentation, reducing defects Documentation of processes
Reduce response times to customer requests
Requirements and change management
Process improvement with CMMI guidelines Decrease testing effort
developed by a single professional. Thus, the success of the project totally depends on the performance, knowledge and capability of that person. Additional enhancements and modifications based on customer requests should also be implemented by the same person. This brings total dependency on a single professional and an increase in the workload of developers. Lack of documentation and formal process management are the most obvious problems when they want to hire new people and/ or train new comers. They do not measure the effort, defects, code quality and schedule. So, they do not have any knowledge about the reliability of their final product. The board of directors in Bilmed realized that their poor process management, human dependency, inefficient and long testing phases and high defect rates prevent them developing reliable software and high quality products. Moreover, these problems prevent the company to grow into new markets and expand their business. Therefore, they have decided to start a company-wide process improvement project to produce reliable software products. We have defined the goals of SQIP and matched them with the company’s management objectives as summarized in Table I. III.
TABLE II.
STEPS OF THE PROJECT
We planned our work in two phases. In Phase I, we aimed to conduct an “as is” analysis of the company. We have concluded that requirements and change management processes are the most critical ones. The outcomes of this analysis have led us define the scope and objectives of the next phase. In Phase II, we chose a pilot project to implement SQIP. We used CMMI version 1.2 as the base framework for our implementation, measurement and evaluation. A. Phase I Activities The first phase of SQIP took place in 7 months. We conducted an 18 hours training to explain the objectives and the scope of the projects as well as the fundamentals of software quality concepts. We have completed 15 meetings in this period with two Softlab members, and three people (quality assurance staff and two managers) from the company. These meetings were done to understand the expectations of the company, to state their problems and required set of actions clearly. We also conducted one-to-one interviews with the staff to learn current level of knowledge and awareness on software
QUESTIONNAIRE I
Assumptions Engineering discipline is used to build quality into of large size and complexity.
Rate
The work is examined by more than one person.
2.4
2.6
Our success depends on other groups.
1.7
The organization uses process definition to transmit the culture’s quality values.
2.4
These processes are applied in practice.
2.4
Process makes a difference in the quality of our activities and product. Surviving in a business world that is constantly changing requires constant adaptation and learning. Our company works with subcontractors or plans to work with subcontractors. Projects are documented at necessary steps. In case of a problem or conflict, the identity of the people to solve this problem or conflict and their responsibilities are known by all staff and they have certain responsibilities. In case of a problem or conflict. the identity of the people to solve this problem or conflict and their responsibilities are documented. In case of emergent changes or new requirement implementations. these are certainly documented.
2.8 2.9 1.5 2.2 2.9
2.6 2.0
engineering concepts. Finally, we have conducted interviews with the field staff at the hospitals as well as the hospital personnel such as doctors, nurses, lab administrators, tellers, call centre agents and technology staff to get feedback from the customers, especially on how they perceived reliable software and their product. On the basis of what we gathered from the company and its customers, we prepared a gap analysis. The gap analysis showed that Bilmed needed to define its policies, procedures and processes and document them. In one-to-one interviews we used a semi-structured approach. We prepared two questionnaires for our analysis (Table II and III) [12]. In Table II, we tried to capture the perceptions of 25 employees in Bilmed by asking them to rate the questions as 1 (never employed in the company), 2 (sometimes) or 3 (always). In Table III, we asked structured questions to participants to express their views about software development practices. We have observed that there is an open communication culture, and staff is open to change and progress. However, lack of documentation, measurement, on-thejob and formal training on new technologies and inefficient resource allocation are negative outcomes that people suffer. The senior management, on the other hand, realize that the root cause for these outcomes are the lack of processes, documentation and the inability to predict project costs, effort and defects. As a result, Phase I of SQIP is concluded with the full documentation of company policies, procedures and processes. These activities are first employed in requirements and change management processes. Both processes are selected as the most critical areas in development lifecycle, since the company needs to immediately deliver what customer wants using their local practices in order to avoid “customer rules” in their development practices.
204
TABLE III. Question What is your customers’ perception of trustworthy software and excellent services?
What are your expectations from our consultancy? If we had a magic wand, what would we change? Which process/ aspect/ entity of your company is the best? Why?
Which process/ aspect/ entity of your company is the worst? Why?
QUESTIONNAIRE II Answers - Fulfillment of requirements. - Support with short response times. - Predicting possible failures. - Ease of use. -Establishing standards. - Predicting possible problems in the enlargement process. -Guidance for institutionalization. - Integration of best practices. -Documentation standards - Distribution of know-how. - The product. - Requirement analysis - Know-how - Support - Installation & Adaptation - Distribution of know-how. - Documentation - Insufficient project planning. - Inefficient use of resources. - Lack of training for new functionalities. - Heavy workload.
Then, we defined the scope of the Phase II as the implementation of requirements and change management processes at the CMMI Level 2. We have identified a pilot project for the process implementation. B. Process Improvement with CMMI In Phase II we have started with requirements management (REQM) and development (RD) followed by change management, configuration management (CM) and technical solution (TS) as in CMMI process definitions [5]. Compared to the first phase, we have seen more resistance in this phase, since we actually started implementing what we had planned in the first phase. The management, for the first time, realized how much devotion is needed to implement a software quality project within their hectic daily routine. We had often long hours of discussions and negotiations to convince them not to ignore important steps. During Phase II, we conducted regular weekly meetings of average 2.5 hours long and weekly task force meetings of average 3 hours each among Softlab and Bilmed members. In these meetings, we provided templates on policies, procedures and plans from Software Engineering Information Repository [18]. In 6 months, we have done 22 regular and 12 task force meetings in total of 91 hours in five months, with three participants from each side. Bilmed has contributed to discussions with a manager, a quality management staff and a software developer. Nearly 80% of these meetings are based on proper requirements gathering, conformance to customer requirements, documentation of requirements analysis and development. After these meetings, final requirements analysis plan for the pilot project has been completed. Additionally, the organizational policy and procedures for requirements management have been defined. Task force meetings were conducted to resolve specific issues such as
sample scenarios, UML diagrams and use cases for tasks identified at the previous meeting. These activities were controlled through specialized checklists at the task force meetings [13]. Traceability matrix of requirements with customer needs was constructed. We have also prepared policy and procedure templates for change management. We have designed the change request process in line with CMMI guidelines. In the redesigned process, change requests from the customers are evaluated by ensuring that they are consistent with functional and project requirements. If they indicate revisions on requirements, software engineering processes such as REQM and TS are revisited. If the request indicates a problem on design, then Technical Solution is assessed and its conformation to requirements is checked. Thus, change management is designed such that it is directly linked to REQM and TS. Although we had scoped the second phase as the implementation of REQM and change management processes, we have also completed the documentation of company policies, procedures and process for other key areas in this phase. At the end of Phase II, we have performed an internal appraisal to evaluate the process maturity of the company. We based our appraisal to CMMI Level 2 and used the process evaluation checklists of CMMI version 1.2 for specific goals of each process area [5]. Basically, we have inspected seven process areas that compose CMMI maturity level 2: Requirements Management (REQM), Project Planning (PP), Project Monitoring and Control (PMC), Supplier Agreement Management (SAM), Measurement and Analysis (MA), Process and Product Quality Assurance (PPQA) and Configuration Management (CM). In order to avoid conflict of interest and bias, we asked one of our colleagues in the department to be the independent appraiser. Our independent appraiser neither had prior knowledge of the company nor the content of our project. Our appraisal results showed that in requirements phase, maturity is Level 3, since the project fulfills the specific practices of Level 3 Requirements Management (REQM). In PP, PMC, SAM, MA, PPQA and CM they have only defined policies and hence they fulfill the requirements of CMMI Level 2. IV.
EMPIRICAL ANALYSIS OF THE PROJECT
To evaluate the effects of SQIP activities, we measured the defect rates and testing effort by conducting an empirical study. Our experiments compared two projects: a project before SQIP (Project I) and the pilot SQIP project (Project II). Both projects are developed in-house by the same developer. Moreover, we have observed team/ process characteristics, such as team size, total duration of the project and the programming language, and product factors per software modules of the projects, such as lines of code, cyclomatic complexity, vocabulary (i.e. operator and operand counts) and architectural complexity, as illustrated in the Process/Team and Product factors of Table IV. In Project I, requirements gathering and analysis activities were completed with four or five meetings in three days (less than 1%). Moreover, coding and testing phases could not be separated in terms of weeks spent. 205
Thus, our educated guess (also agreed by Bilmed) is that coding took nearly 35%, whereas testing took nearly 61% of the project duration. On the other hand, Project II is the pilot project whose total duration is 15 weeks. Requirements gathering and analysis phase took a total of 7 weeks. The rest was spent to coding and testing, which are 3 and 5 weeks respectively. In order to estimate the defect rates, we have applied a defect prediction model and identified modules that have high probability of having defects. Then, we measured the testing effort required to inspect those defective modules in the software. Defect predictors are widely used models by software companies to effectively allocate their testing effort by focusing on defect-prone parts of the software [6, 14, 21, 20, 22]. Companies can decrease their testing effort and enhance the reliability of their software product with the help of such tools. Since the company did not collect any defect or test effort related metrics, we have measured the estimated defect rates. Then we have estimated testing effort required to inspect defective modules. We acknowledged that evaluation of two projects based on estimations would have a bias compared to actual statistics. However, it is intractable to conduct this empirical study based on actual statistics due to lack of metrics. Instead, we used cross-company data [21], i.e. projects collected from other companies. Metrics are static code attributes which can be collected using a simple and automatic tool [16] from projects either within or outside the company. The experiments of previous study [21] confirm that using cross-company data is not unclear or fictitious. We can increase the probability of detection rates by using crosscompany data if the model cannot learn from local projects [21]. In this study, we used public NASA datasets as cross-company data, whose total number of modules is more than 20,000 [3]. However, we have chosen samples from this dataset using k-Nearest Neighbor algorithm to select the most similar projects to local data [20]. Based on these two studies, we selected 90% of available NASA data as the training set. Then, we applied the sampling algorithm to select the most similar project modules in order to train our defect predictor. Finally, we estimated the defect rates of two projects to make a comparison. The results of estimated defect rates and required testing effort to inspect all defective modules are summarized in the Results field of Table IV. Estimated testing efforts of two projects are computed based on a recent analysis done by Arisholm and Briand [2]. They assess the cost-effectiveness of their prediction algorithm in terms of inspection effort. The assumption they rest their work on is as follows: the inspection effort is simply proportional to the lines of code of the estimated defective modules in a system. That is, if we randomly select classes from our code during testing, we need to trace at least 70% of the code to detect 70% of defective modules. If our model decreases this testing effort, i.e. percentage of LOC inspected (last two lines of Table IV), by predicting true modules as defective, we can reduce the effort of predicting defective modules. TABLE IV.
PROJECT CHARACTERISTICS AND ESTIMATIONS ON DEFECT RATES AND TESTING EFFORTS
Project I
Project II
Team size
1
1
Duration (weeks)
16
15
REQM (weeks)
0.6
7
Coding (weeks)
5
3
Testing (weeks)
10.4
5
Programming language
C++
C#
Total LOC
15460
21707
Total # modules
880
1659
LOC
13
18
Cyc. complexity
3
3
Vocabulary
13
17
Arch. complexity
3
5
Defective modules
99
108
Defect rate (%)
11.2
6.5
Defective LOC
7220
6534
Inspection effort (%)
46.7
30
Process / Team Factors
Product Factors
Results
We estimated that a tester needs to trace nearly 47% of the code to detect 99 defective modules in Project I, whereas it is 30% to detect 108 modules for Project II. We also observed that the estimated defect rate decreased to 6.5% from 11.2%. Even though the developer spent 7 more weeks to complete coding and testing of Project I, the productivity in Project II is far better than Project I in terms of defect free lines of code per week. The developer completed the coding of the second project in a shorter time with fewer defects, although the coding practices are the same. Thus, we can conclude that this improvement is achieved using the practices implemented in the second project. We have also compared the two projects with pie charts, to evaluate the schedule distribution among each development stage (Fig. 1). First two charts in Fig. 1 correspond to time spent for each development activity for Project 1 and Project 2 respectively. The last chart represents the industrial average for the same activities [19]. In Fig. 1, we see that the company had crucial problems in coding and testing phases due to inaccurate requirements gathering and analysis. After implementation of SQIP, we have separated these two phases and achieved to decrease the time spent for coding phase to an acceptable level, i.e. closer to the industry average. Moreover, we increased the allocation of resources for requirements management. Time spent for REQM will be much lower in future projects since Project 2 was the first one to implement SQIP and hence we spent considerable amount of time for training and discussions. Our comparison also showed that the time spent for testing phase has decreased, since the developer produced more
206
reliable software with the help of well-documented and analyzed requirements.
Figure 1. Pie-charts for development costs.
V.
LESSONS LEARNED
During the activities of SQIP, we have faced many problems, solved conflicts between the company and our plans and gained valuable experiences that will be useful for similar studies. Below, we list fundamental issues that we find useful to share. A. Best Practices Open Interviews and Training: Interviews and training in Phase I helped a lot to understand the general knowledge of the staff and the specific problems of the company. We were able to show a) why senior managers could not accurately estimate costs and defects, b) why developers had high workloads due to unresolved requirements, c) why it is hard to integrate new employees and d) why all project team should share a common knowledge base in software engineering concepts. P2P (Persistency and Patience): The activities of SQIP showed that internal resistance of the company to new technologies and standards are sometimes so high that it is very difficult to convince them about the overall benefits of these activities. Although they have the motivation to invest in improvement, it is hard for them to balance the cost of process improvement with their business objectives. Since the company is an SME, they want to see the outcome as quickly as possible and they look for the cheapest solution. However, consultants should clearly state the gradual effect of process improvement activities on their software development lifecycle such as reducing costs and defects and managing resources in the long run. Bottom-up Ownership with Management Support: As we observed during the SQIP activities, we once more acknowledged that senior management support is a “must” for process improvement activities. An organization-wide change with standard procedures, policies and processes needs active involvement of senior management. On the other hand, commitment of all employees and their ownership of a SQIP type of project is another critical success factor. B. Things to Avoid Next Time Project Planning: Bilmed did not see project planning as the high priority process area for them. That is why we did not have a thorough study of principles, specific goals and practices of this process area. However, companies should not underestimate project planning. We have negotiated and discussed to set a proper project plan for
activities of SQIP and the schedule of the pilot project. If we had a more detailed project plan, we would have tried to stick to the schedule for completion of improvements for each process area. Thus, we would not have spent many weeks at the beginning of Phase II to do requirement management activities. Internal Resistance: As we mentioned earlier that Bilmed staff is open-minded about changes and process improvement. However, it is difficult for them to get used to some set of actions, which are totally different from their established practices. For example, detailed documentation and UML modeling of requirements put high pressure on the analyst who was working on the pilot project. Because the only analysis done on requirements phases of previous projects were ad-hoc discussions over user interfaces and customer needs with no documentation. Therefore, we had difficulties in convincing them about the required level of detail in the documentation. Tool Support: During requirements development, we spent too much time for detailing use cases, scenarios, data flow diagrams and traceability matrices. We have observed that a dedicated tool will be very useful to decrease the time and effort spent for standard requirements management activities. A dedicated tool will also increase the reuse percentage of developed code and provide an automated standardization for process activities and documentation among projects. Going forward, an off-theshelf product can easily be integrated and extended to serve their needs. Segregation of Duties: Adaptation to process improvement activities cannot be easily accomplished at SME’s if segregation of duties is not clearly defined. Frequent auditing and training should be carried out by management to convince the staff about the benefits of standard processes and documentation. On the other hand, one of the problems that make adaptation harder is when requirements, coding and testing phases are not carried out by different people in the development team. Dependency on single person makes that person precious and he/ she takes the initiative to change the direction of the project. This was one of the reasons for resistance. On the other hand, segregation of duties enables load balancing and leads to process excellence in each stage. VI.
CONCLUSION
In this paper, we have worked with an SME, who operates in healthcare, to improve their software quality and reliability in order to achieve their business objectives. To accomplish this, we have focused on specific process improvement activities such as requirements and change management. We have employed a two phase software quality improvement project (SQIP) to provide standardization in their software development lifecycle activities. We have defined organizational standards, policies, procedures and processes. We have adopted the CMMI guidelines in line with the business goals of SME. We have seen that managing and improving processes lead to decreased testing efforts and better software quality in terms of defect rates. As a result of SQIP, now managers can share their ‘know-how’ with the staff through documentation. Developers ease coding phase with clearly defined requirements and they assure their software reliability with defined processes and measurable products.
207
Apart from process improvement activities, we have compared two projects (Project I: Before SQIP, Project II: After SQIP). We made estimations of the number of defective modules and testing effort required to inspect those for both projects with our defect prediction model. Results showed that without proper requirement and change management activities, the coding and testing phases require large amount of time. Hence, this decreased the code quality and increased the possible number of defective modules. We have measured that Project I has a higher defect rate than Project II. Therefore, the developer/ tester needs to spend more effort and time to trace defective modules in Project I, which leads to additional testing time for the project. Additionally, it is observed that requirements management activities cost much more time than expected because of loose project planning and staff resistance to process improvement. However, these activities have high reuse percentage. They can be integrated to further projects which would reduce time spent for requirements management activities significantly. Besides, we have achieved a balanced distribution between stages of development lifecycle. We decreased the coding time from 31% to 20% as well as testing time from 65% to 33% after SQIP activities. These facts lead us to realize the importance of applying detailed requirements management to fulfill customer expectations and to balance ratio of each stage in the software development lifecycle. Table V lists major benefits of the company for conducting a quality improvement project by comparing the previous and current situation. We have presented an application of academic research in collaboration with an SME. We encourage more companies to cooperate with the academia in order to close the gap between practice and research. Companies can objectively evaluate their processes and development practices, and further, they can learn how effectively they can revise their organizational policies and objectives. They will be well aware about the current research in academia such as new software engineering principles, innovative technologies and global research challenges. Such collaborations will also have a positive effect for academia such that researchers can combine their studies with current industry practices. They can effectively observe the outcomes and practical applicability of their work by investigating the situation in companies. Going forward, Bilmed needs to implement all the software process lifecycle stages as per CMMI Level 3. They also need to measure the code quality, effort and defect rates by regularly collecting metric data and bug data that are matched to the code. We will collect code metrics automatically from source code in order to install a process where each code specifies the nature of the fault in testing. Combining with bug data from local projects, it will be easier to conduct empirical studies and predictor models. Softlab will also conduct periodic reviews to embed continuous improvement within the company. ACKNOWLEDGMENT This research is supported in part by Bilmed, and Turkish Scientific Research Council (Tubitak) under grant number EEEAG108E014. TABLE V.
CHANGES IN DEVELOPMENT LIFECYCLE: BEFORE AND AFTER SQIP
Issues
Before SQIP
Handling requirements
Through e-mail or phone.
Requirements Analysis
Ad-hoc meetings.
Documentation
No or little documentation.
Standardization Organizational policies, procedures and plan templates Separate software development phases Decrease in testing effort and defect rates
No defined standards for processes. No policies, procedures or plans for software development processes. No defined timeslots for each phase. Unpredictable test effort and defect rates.
After SQIP Well-defined requirements management and development processes. Documented scenarios, use cases, UML diagrams, DFDs and GUI templates. Requirements and change management are well documented. Standards based on CMMI practices. Available for CMMI Level 2 Key Process Areas Balanced allocation of resources for each phase. Estimated decrease in the effort and defect rates.
REFERENCES [1]
J. Abbott, “Software Process Improvement in a Small Commercial Software Company”, Proc. Software Engineering Process Group Conference, SEPG’97, 1997, San Jose, CA. [2] E. Arisholm, C.L. Briand, “Predicting Fault-prone Components in a Java Legacy System” Proc. ACM/IEEE Symposium on Empirical Software Engineering, ISESE’ 06, 2006, Rio de Janeiro, Brazil, pp 8-17. [3] G. Boetticher, T. Menzies, T. Ostrand, PROMISE Repository of empirical software engineering data. Department of Computer Science, West Virginia University. Available at http://promisedata.org/repository. Accessed 01 March 2008. [4] J.G. Brodman, D.L. Johnson, “What Small Businesses and Small Organizations Say About the CMM”, Proc. 16th International Conference on Software Engineering, Sorrento, Italy, 1994, pp 331-340. [5] B.M. Chrissis, M. Konrad, S. Shium, “CMMI Guidelines for Process Integration and Product Improvement”, Addison Wesley, 2005, USA. [6] E.N. Fenton, M. Neil, W. Marsh, P. Hearty, D. Marquez, P. Krause, R. Mishra, “Predicting Software Defects In Varying Development Lifecycles Using Bayesian Nets”, Information and Software Technology, vol. 49, 2007, pp 32-43. [7] S. Garcia, S. Cepeda, J.M. Staley, G. Miluk, “Summary of CMMI For Small Business Pilot Study in Huntsville. Alabama, USA”, Presented at Info Seminar, June 2004, Carnegie Mellon Software Engineering Institute, Pittsburgh, USA. [8] S. Garcia, “ Thoughts on Applying CMMI in Small Settings” Presented in Carnegie Mellon Software Engineering Institute, 2005, http://www.sei.cmu.edu/cmmi/adoption/pdf/garciathoughts.pdf. [9] R. Hadden, “Key Practices to the CMM: Inappropriate for Small Projects?” Proc. Software Engineering Process Group Conference, SEPG 1998, Chicago, IL. [10] M. Harris, K. Aebischer, T. Claus, “The Whitewater Process: Software Product Development in Small Businesses”, ACM Communications Magazine, vol. 20, no. 5, 2007, pp 89-93. [11] L. Hoffmann, “Small Projects and the CMM”, Proc. Software Engineering Process Group Conference, SEPG 1998, Chicago, IL. [12] K. Kaputo, “CMM Implementation Guide: Choreographing Software Process Improvement”, Addison Wesley, 2000, USA.
208
[13] K. Kaputo, “Analysis Checklists for Requirements and Commitment Control”, Proc. Software Engineering Process Group Conference, SEPG 2001, NewOrleans. [14] T. Menzies, J. Greenwald, A. Frank, “Data Mining Static Code Attributes to Learn Defect Predictors”, IEEE Transactions on Software Engineering, vol. 33, no.1, 2007, pp 2-13. [15] M. Paulk, “Using the Software CMM in Small Organizations” The Joint Proc. of the 16th Pacific Northwest Software Quality Conference and Eighth International Conference on Software Quality, 1998, pp 250-361. [16] Prest Metrics Extraction and Analysis Tool, Available at http://code.google.com/p/prest. [17] M. Sanders, “Small Company Action Training and Enabling. In: The CMM and Small Projects”, Society for Software Quality Roundtable, 1998, Washington, DC.
[18] Software Engineering Information Repository (SEIR) Carnegie Mellon University, 2008, Available at http://www.seir.sei.cmu.edu. [19] The Standish Group, “The Chaos Report”, 1995, Available at http://www.educause.edu/ir/library/pdf/NCP08083B.pdf [20] B. Turhan, A. Bener, T. Menzies, “Nearest Neighbor Sampling for Cross Company Defect Predictors” (Abstract Only). Proc. 1st International Workshop on Defects in Large Software Systems (DEFECTS'08 Workshop in ISSTA'08), pp. 26, 2008. [21] B. Turhan, T. Menzies, A. Bener, J. Distefano, “On the Relative Value of Cross-company and Within-Company Data for Defect Prediction” accepted for publication in: Empirical Software Engineering Journal, 2008. [22] A. Tosun, B. Turhan, A. Bener, “Ensemble of Software Defect Predictors: A Case Study” Proc. 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM’08), pp. 318-320, 2008.
209
