Software Quality: A Curriculum Postscript? Thomas B. Hilburn Embry-Riddle Aeronautical University Department of Computing and Mathematics Daytona Beach, FL 32114
[email protected] Abstract This paper addresses a central and critical issue in the development of computer software - its quality. The main thesis of the paper is that computer science faculty, in their design and implementation of curricula, do not devote sufficient attention to teaching their students how to develop high-quality software. As in industry, the most common and popular way of assuring the quality of programs is through software testing. In other words, quality is treated as an afterthought or as postscript in program development. The paper presents and discusses a quality model that can be used to incorporate a wide variety of quality assurance techniques within a curriculum. The model also presents a structured approach for introducing software testing into the educational environment. Finally, there is a discussion of how the model has been implemented using two current software process technologies, the PSP and the TSP. 1
Introduction
Computing has become prevalent in almost all aspects of human endeavor; software is now critical to the current state of consumer products, transportation, medicine, law, management, and education. The information age has arrived and we as computer science educators are challenged as never before. Software products are larger and more complex, and their development demands immense resources in people, money, and time. Although we have made remarkable progress in the engineering of software, much of it is developed in an ineffective and inefficient manner - the software development landscape is filled with examples of projects with cost and schedule overruns and with severe quality problems [3].
Massood Towhidnejad Embry-Riddle Aeronautical University Department of Computing and Mathematics Daytona Beach, FL 32114
[email protected] The quality issue can dominate and drive all of the other concerns in the development of software. It is not uncommon in the development of a software system to spend 40% to 60% of its development resources in testing. Of course, this is rarely anticipated, with the result that budgets and schedules are wrecked. There has been increasing recognition of this problem by educators and there are a number of current initiatives underway to develop courses and curricula that will better prepare our students for the challenges of developing complex systems [1, 2, 10]. The emphasis on software engineering in many computing programs introduces discipline, structure, and rigor to the development of software - this is helping to move it from an individual, artisan activity to a team-based engineering endeavor. 2
Software Quality Issues
The three crucial issues in the development of software products are cost, schedule and quality. Although we can touch on cost and schedule in the classroom, it is difficult to address these issues in a realistic and comprehensive manner in the academic environment. However, we do feel that the quality issue can and should be addressed extensively in a computing curriculum. 2.1 What is Quality?
A good question! Quality is one of those terms that everyone uses; but there is probably not a universal, consistent understanding and agreement about its meaning. Software engineers (and academics) apply the term "quality" to both the product being produced and to the process used to produce it. Although these two types of quality (product quality and process quality) are dependent on each other, they involve different techniques, measures, and implications. Broadly speaking, product quality is related to how well the product satisfies its customer/user requirements. Related to this (but maybe not specified) are the usability, performance, reliability, and the maintainability of the
software. Process quality is concerned with how well the process used to develop the product worked. In this case we are concerned with elements such as cost estimation and schedule accuracy, productivity, and the effectiveness of various quality control techniques (e.g., inspection rates and yields). 2.2 What is the Cost of Quality?
Maybe the question should be what is the cost of low quality. If an organization produces low quality software, as a minimum, it risks its reputation and financial well being, and in the worst case it may be risking the lives or security of its users. The software industry is well aware of its quality problems and invests significant resources in insuring software quality. However, most of the resources are devoted to testing or verifying the quality near the end of the development process. It has been estimated [9] over half of a product's defects are injected in the requirements and design phases. The relative costs of removing a defect in the requirements phase versus the test phase can run as high as 1 to 50. (It is even higher for removal after delivery.) Studies [6] have shown that an effective software process, that emphasizes statistical quality control and introduces quality reviews early in the process, can provide dramatic increases in quality, while not increasing the total cost of development. In other words - Quality is Free! 3
Curriculum Issues
To address the quality issues discussed in Section 2 and to better prepare our students for a software development career, it is important that we introduce quality techniques and activities in our undergraduate computing curricula. Unfortunately, most academic software development activities concentrate on development technology (languages and tools, design methods, and the development environment) and emphasize "product" issues over "process" issues. In addition, those quality assurance techniques that are used seem to center around unit testing. Students often end up, like many professional developers, approaching testing in an ad-hoc, trail and error fashion. For the professional, these errors are typically detected in integration or system test or by the user after installation; for a student’s program that never gets used after a course is completed, the program may just be assumed to be of high quality. We propose that computer science curriculum designers and developers treat software as a central element of their curricula. Software quality should be emphasized in the beginning (the very first CS course) and throughout the curriculum. Quality reviews (personal reviews, walkthroughs, inspections) should be incorporated at the requirements, design and coding stages of software development. We do not mean to diminish the importance
of testing in the development process. It is a critical element in the verification and validation that a product satisfies its requirements and it is pivotal for customer acceptance of the product. We do, however, think that more rigor and structure needs to be established in the way testing is taught. Finally, we do not believe that students can learn to produce high-quality software without the proper guidance, organization, and support. Hence, we believe that it is essential that a software development process that emphasizes quality techniques, methods, and analysis be taught to and used by students throughout their program. Such a process would typically consist of a set of process scripts, forms, guidelines, measures, and tools. 4
A Software Quality Model
Pressman [9] argues that software engineering should be viewed as a "layered" technology with quality as the principle focus and process at its foundation. Higher layers of technology (methods, techniques and tools) are based on the quality focus and process foundation. In this section we describe a model that can be used not only for developing quality software and but can serve as the basis for designing and implementing a computing curriculum that emphasizes software quality. 4.1 The V Quality Model
The "V model" is a variation on the traditional waterfall lifecycle model that highlights the correspondence between software testing and the development phases [4, 8]. We have modified the original V model, including additional quality checkpoints, and refer to this model as the V Quality Model. Figure 1 illustrates the key features of the model. The left side of the V concentrates on the traditional development phases, and the right side concentrates on testing. Each box in the model represents a phase in either development or testing, and it is assumed that in order to move from one phase to the next we should complete and pass a quality review. There are two passes through the right side of the model. The first pass is from the top to bottom, and it indicates the development of the software testing framework (strategies, scenarios, test cases, logs, etc.). The second pass is from the bottom to the top and it indicates the execution of testing. The horizontal dashed line between the phases (on the left and right side of the V model) indicates concurrent activities. For example, as the process of requirements analysis gets underway, planning for system testing starts. Therefore the V Quality Model proposes the following activities. Once the project is identified, the requirements analysis gets under way and concurrently the process of system test development begins. The product of the requirements analysis phase is used as the input for the architectural design of the system. Concurrently, the development activities associated with integration testing get underway. The outcome of the
architectural design is used as an input to the detail design phase. At the same time, planning for unit testing begins. Next, as the coding for a unit is completed, unit testing begins. After all units are tested then integration begins which is followed by a test of the entire system. In each test phase, the discovery of defects requires, as a minimum, a return to the left side of the V model in order to continue quality development. In addition, with each development and test phase is associated a quality review of the artifact produced (e.g, a requirements specification, a design specification, source code or a test plan). Figure 1 captures this idea with the "quality review" oval that is connected to each phase box. The type of review (walkthrough, formal inspection, peer review, or personal review) depends on the size and scope of the project and the available resources. One of the major advantages of the V Quality model is the natural multiple validation techniques that are embedded, as
early as possible, in the model, thereby increasing the quality of the product. For example, during the requirements analysis phase, the requirements are reviewed by a group (or individual). In addition, as the framework for the system testing is developed, the requirements are examined for testability. Finally, there is a review of the system test plan itself (e.g., Does it include coverage of all requirements?) This provides yet another opportunity for checking the requirements. As a result, each requirement goes through at least three quality assurance checks before it is released for the architectural design. Of course, once the actual system testing is executed another layer of quality assurance is added, but we expect by the time we reach this stage, we have already eliminated a large percentage of defects from the system.
Requirements Analysis
System Testing
Architectural Design
Integration Testing Quality Review
Quality Review
Detailed Design
Unit Testing
Coding
Figure 1: The V Model for Quality 4.2 Personal Software Process
The V Quality model can provide the conceptual foundation for a curriculum emphasizing software quality, but additional guidance, support and structure is needed to implement this model in a curriculum. We believe the current technology in software processes can help in this area. At Embry-Riddle, for the past four years, we have been using the Personal Software Process (PSP) to introduce and implement a software quality focus in our undergraduate computer science curriculum [5]. The PSP is designed to help students and professional software engineers to organize and plan their work, track their performance, manage software quality, and analyze and improve their personal process. It incorporates some of the elements of the V Quality Model appropriate for an individual developing a program unit. For instance, it includes the development of a unit test plan during the
detailed design phase and incorporates individual quality reviews in the design and coding phases. In addition, it provides for the collection, computation, and analysis of a number of metrics that measure and influence quality. The PSP can be started in the first year of a curriculum. Hence, it provides an early introduction to the concepts and techniques for producing quality software; and it sets the foundation for implementing the full V Quality model in a curriculum. 4.3 Team Software Process
The Team Software Process (TSP) is a defined process for a group of students or software professionals to develop quality software in an effective manner [7}. It provides guidelines, process scripts, tools, methods and techniques for developing a software product by a team. It is based on linear, incremental model that divides development
effort into a set of "development cycles" where each cycle involves producing software that satisfies some subset of the software requirements. The final cycle encompasses integration and test of the entire system (all requirements satisfied). Figure 2 gives a picture of the cyclic model for software development.
The TSPi encompasses all of the essential elements of the V Quality model: testing preparation that corresponds to the requirements phase and the two design phases, and the use of quality reviews for requirements, design, code, and the test documents. We have also added a customer review after each intermediate cycle and a customer acceptance test at the end of the last cycle. Each team conducts formal inspection of both the requirements and design specifications and a personal and peer review of each unit of code. For the inspections students are taught a formal technical review process that includes a process script, inspection guidelines, ideas for developing an inspection checklist, and a template for an inspection report. The quality review for test documents is more informal, but it is still is a scheduled activity in the overall process. Each team has a designated Quality Assurance Manager that has overall responsibility in this area.
Product Need Statement Cycle 1 Launch
Strategy 1 Plan Requirements 1 Design Implementation 1 Test Postmortem 1
Cycle 2 Launch
Strategy 2 Plan Requirements 2 Design Implementation 2 Test Postmortem 2
Cycle 3 Launch
Strategy 3 Plan Requirements 3 Design Implementation 3 Test Postmortem 3
Students are still not getting defect-free software, but they are finding more defects and at an earlier stage that in previous course projects. Table 1 contains data for one cycle from a set of eight TSPi project teams; it illustrates the sort of data that is available when a defined process is used. The pre-compile yield is a measure of the percentage of defects injected before compile that are removed by inspections and personal/peer review. A yield of 70 or 80 % is a good indicator of a quality process, while yields below 50% point to potential problems in the software or its development. Table 1 shows that three of the eight teams have excellent yields. The low yields for the other teams point to possible product quality problems and motivate follow-up analysis by the team or instructor, or both. Fortunately, the TSP provides a number of metrics and benchmarks that can be used for this purpose.
Finished Product
Figure 2: TSPi Structure and Flow
At Embry-Riddle we use an academic version of this technology (TSPi) in a junior level course where we teach software engineering fundamentals and students engage in a team software project. This course is a major prerequisite for a capstone, senior design course. In our eight years of offering software project courses we have found that the TSPi provides the most effective approach to teaching such courses. It gives students clear, precise guidance and support; it provides good data collection and analysis tools and techniques; and it helps create an environment for building successful teams.
Table 1: Spring 1999 Project Work - Cycle 2 - 4 weeks (3/22/99 - 4/19/99) Team
Plan Actual Plan Effort (hrs) Effort (hrs) N&C LOC
Actual N&C LOC
Total Defects
def/KLOC
LOC/hr
pre-comp yield
1 2 3 4 5 6 7 8
315.21 43.10 132.50 159.50 85.34 112.50 125.93 68.50
268.47 163.43 147.30 135.20 90.56 109.50 116.75 59.70
418.00 990.00 130.00 175.00 743.00 300.00 250.00 451.00
1419.00 1185.00 746.00 418.00 839.00 471.00 699.00 556.00
183.00 6.00 57.00 67.00 90.00 33.00 24.00 60.00
128.96 5.06 76.41 160.29 107.27 70.06 34.33 107.91
5.29 7.25 5.06 3.09 9.26 4.30 5.99 9.31
46.10 0.00 38.60 74.62 76.66 48.48 86.36 13.33
avg std dev
130.32 83.52
136.36 62.55
432.13 296.75
791.63 350.34
65.00 54.55
86.29 50.52
6.19 2.26
48.02 30.71
In general, the faculty and students involved in the use of the TSPi believe that it does an excellent job of introducing, emphasizing, and addressing the quality issues discussed in Section 2. For the first time we have the methods, techniques, and data for assessing and improving the quality the products produced by our students. We believe that both the PSP and TSP provide the necessary process foundation for the quality focus embodied in the V Quality model. 5
Conclusion
Although almost anyone would agree that software quality is important, we as educators often do not make it a major concern in the curricula we deliver. We often treat software quality as a secondary concern (surely unintentionally) and by our pedagogy deliver the message that quality should be addressed almost exclusively in testing, at the end of development. We recommend that software quality should not be an afterthought - it should be addressed in the front-end of the life-cycle. We think that software development oriented curricula should focus on quality and that the V Quality model provides the conceptual framework for such a focus. We also believe that the development of high-quality software requires the discipline and guidance furnished by defined software processes. Our experience with the PSP and the TSPi shows that they are excellent candidates for helping to implement a curriculum that emphasizes and teaches about software quality throughout the curriculum. References [1] Barnes, B., et. al. Draft Software Engineering Accreditation Criteria, Computer, 31, 4, (April 1998), 73-75. [2] Bagert, D., et. al. Guidelines for Software Engineering Education, Version 1.0, Technical Report CMU/SEI-99-TR-032, Software Engineering Institute, Carnegie Mellon University, (November 1999). [3] Gibbs, W.W. Software’s Chronic Crisis . Scientific American (September 1994), 86-95. [4] Germany Ministry of Defense, V-Model: Software Lifecycle Process Model (1992). [5] Hilburn, T. and Towhidnejad, M. Integrating Personal Software Process (PSP) Across the Undergraduate Curriculum. Proceedings of 1997 Frontiers in Education Conference (November 1997). [6] Humphrey, W. S. The Discipline of Software Engineering, Addison-Wesley, 1995. [7] Humphrey, W. S. Introduction to the Team Software Process, Addison-Wesley, 2000. [8] Jorgenson, P.C. Software Testing, A Craftsman’s Approach, CRC, 1995.
[9] Pressman, R. Software Engineering: A Practitioners Approach, 4th Edition, McGraw-Hill, 1997. [10] Software Engineering Coordinating Committee, http://www.acm.org/serving/se/.
