The Role of Software Process Definition in the Computer Science ...

-

Doing Quality Work: The Role of Software Process Definition in the Computer Science Curriculum Thomas B. Hilbum and Massood Towhidnejad Department of Computer Science Embry-Riddle Aeronautical University Daytona Beach, FL 32114 e-mail:[email protected], [email protected]

Abstract

encouragement and guidance necessary for developing curricula that meet the needs of the software engineering profession.

This paper discusses the role of personal software process definition in the education of computing professionals and the importance of emphasizing quality in the development of software. After examining recent government and industry efforts in introducing and instituting effective software development processes, there is a description of the Capability Maturity Model (CMM) and Watts Humphrey’s Personal Software Process (PSP) and its use in industry and academia. The rest of the paper reports on a recent project that introduced PSP conceptsinto CSl and CS2. Project methods and activities are described and the results of the project are analyzed. Finally, future enhancements and extensions of the project are discussed.

In order to improve the quality of software and the manner in which it is developed, there have been a number of efforts to increase the effectiveness of software development organizations.The Capability Maturity Model (CMM), created at the Software Engineering Institute (SEI) in 1989 [11], provides a structured framework for an organization to assess the current level of maturity of its SOfhVaE development processes and plan for improvement and advancement. The CMh4 approach has been mandated by some government agencies and has become the basis for sofhvare process engineering in many commercial software organizations [2,6]. The CMM is organized around five levels (Level 1 to Love15) that am used to classify the maturity of organizational processes. An organization where there are few defined processesand where success depends on individual effort would be classified as Level 1 or “initial”, while organizations with more fully defined processes and the ability to assess and improve their processes would be classified at a higher level. Under each level there is a list of the “key process areas” (KPAs), each of which is associated with specific goals and example practices that better define the maturity level. Currently most U.S. software organizations would be assessed at CMM Level 1 or CMM Level 2. Many Department of Defense contracts now require that an organization be assessed at CMM Level 3 or above.

lntroductlon The problems in software development are well-documented [lo]: large software systems are often late; they cost too much; and many are unreliable. Some of this is not surprising if one mkes a look at the state of the software engineering profession [31. There are no established or agreed upon standards for the minimum knowledge and competencies that a software engineer should possess; hence, there are no professional certitication or licensing programs for software engineers. Most new graduates that are hired as software engineers come from computer science programs not “engineering”programs. Of course, many computer science programs now include software methodology and engineering as part of their curriculum,and some have adopted curricula that are similar to the “software engineering emphasis” sample curriculum in Computing Curricula 1991 1121. However, it is debatable whether the curriculum guidance provided by Computing Curricula 1991 or the requirements for certification formulated by the Computer Science Accreditation Board [13] provide the Permission to make digital/hard oopy of part or all of this work for personal or classroom use Is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the ubllcation and its date appear, and notice is given that cop ing Is y permlssion of ACM, Inc. To copy otherwise, to.republish, to pos or to redistribute to lists, requires pnor specific permrssion Yon sewers, & and/or a fee. SIGCSE ‘97 CA, USA 0 1997 ACM O-89791-889.4/97/0002...$3.50

277

The computer science faculty at Embry-Riddle Aeronautical University (ERAU) have developed a computer science program that is centered around an aviation domain of application and emphasizes good software engineering practices. The program is designed to prepare students to work as part of a team on the development of an aviation/aerospace sofhvm system, Software engineering concepts are introduced in the fmt year (in CSl and CS2) and expanded on in subsequent years. In addition to an introductory course in software engineering, three of the courses in the third and fourth years of the program involve a team project of three to five students and emphasizes some life-cycle phase. There is also a capstone “senior design” course. A detailed discussion

.

about the software engineering aspects of the program is contained in [4]. Graduates of the program are employed as softwamengineers at Boeing, McDonnell-Douglas, LockheedMartin,the FAA, NASA, etc. In addition to our undergraduate program, we offer a Master of Software Engineering (MSE) program that concentrates on the definition and use of effective processes for development of software [9]. The CMM has played a major role in the design and implementation of the program. The Personal Software Process In discussing major software calamities, Fred Brooks [l] states in one of his essays that usually “the disaster is due to termites, not tornadoes”. In other words, it is the many small problems of individuals that cause the schedule slippages, cost overruns and the quality catastrophes. Brooks, also says that programmers are eternal optimists and opines that software development seems to “attract those who believe in happy endings and fairy godmothers”. A reasonable conclusion one could draw from Brooks’ statements is that in order for an organizationto improve its software products, it is necessary for its individual software engineers to have well-defined disciplined personal processes that includes effective and realistic planning and that emphasizes a “quality”approach to programming. The Personal Software process (PSP) was created by Watts Humphrey [7] to address the need for individual software engineers to acquire a disciplined and effective approach to writing programs. Humphrey has scaled the CMM down to a personallevel. Ten of the eighteen KPAs from the CMM have been incorporated into four process levels that start with a “baseline” process and evolve into a ‘cyclic personal process”. The philosophy behind the PSP is that an organization’sability to build large-scale software systems depends upon the ability of its individual software engineers to develop high quality small-scale programs in a disciplined, effective manner. The PSP is designedto help students(and engineers) to organize and plan their work, track their performance, manage software defects, and analyze and improve their personal process. The PSP is currently being taught in a number of industrial organizations and has been offered at the senior undergraduate level and the graduate level in various academic programs. The basic PSP content (as described in [7]) consists of ten programmingassignments and five reports. Students start with a simple process that includes estimating and recording their effort for the different process phases. As students progress throughthe assignments,they learn about and use more detailed and mature processes. The PSP relies heavily on data collection, mathematical estimation techniques, formal reviews (design and code), and the use of analysis to improve process effectiveness. At ERAU we are using the PSP as the basis for the first course in our MSE program: it serves as an introduction to software engineering and as a foundation for process activitiesthat are

introduced in subsequent courses. Our experiences in @aching this course and our discussions with industrial managers and other academics have supported the decision to give PSP such a prominent place in the curriculum. We have found that new students entering our MSE program, both recent graduates of undergraduate programs and programmers with many years of experience, come to us with poorly defined and ineffective processes: they have little experience with personal planning (estimating and scheduling); they have no clear understanding of the quality of their work (no data on the size, effort or quality of their programs); and they lind it difficult to articulate how they write programs (no defined approach). We have found the same problems with our own undergraduates. Since it is very difficult to unlearn bad habits we believe it is critical for student programmers (and prospective software engineers) to start off with an approach that encourages a disciplined method for developing software. In the following sections we cxplorc some ideas for advancing such an approach. A New PSP Project In the 19951996 academic year we decided to introduce software process concepts and activities into our CSl and CS2 courses. We used an early draft version of Watts Humphrey’s Introduction to the Personal Software Process [8] to teach the material. The book is written for students new to computer science and software engineering. Its narrative style and its examples and exercises have been tailored to first year computer science majors. The book has a major emphnsis on time management, and the PSP material on estimation and design has been simplified. Also, there arc no prescrlbcd progmmmingassignmentsfor the PSP activities, so the material can be used with a variety of different programming assignment profiles. We call our project “Doing Quality Work”. Our goal is to impressupon students, from the very beginning, how important quality is in their work. Morespecifically our objectives arc as follows: 0

provide students with learning expcricnccs that will help them to understand the importance of a disciplined approach in the development of software;

0

engage students in activities in which they use the elements of a defined personal software process;

0

further and provide a / foundation for development/improvement of the student’s PSP and its use in individual and team projects. CS I Activities

PSP conceptswere first introduced into CSl in the fall semester of 1995. We started with time management and beginning planning activities. We feel that a beginning college student can best assimilate and benefit from the time management activities. However, although all students entering our CSl

__

-__-_.-..

___-..- --.-

the time spent in class;

0

PROGRAMMING

the time erog~g;

0

STUDY

the time spent preparing for class;

0

PSP

spent

-

~. --

~--

Assessment and Analysis In the fall of 1995 and the spring of 1996 au anonymous opinion survey was conducted in CSl and CS2, in order to solicit studentreactionto the PSP project. Responses indicated that students thought they understood the PSP concepts and were capable of doing this type of work. The most positive responses were related to the benefits of the code review process and the reduction in compile and test times. The most negative responses concerned the perceived value of the PSP efforts and some of the mechanics of the PSP activities (e.g., how defect information is recorded, how “to date” data is computed and the number of different forms). These concerns have been addressed by changing the way the PSP material is integrated into CSl and CS2 and how the concepts and activities are motivated. Also there have been revisions of some of the PSP forms and techniques.

PSP assignments consist of time management exercises where students are asked to keep track of the time they spend on the following four activities: CLASS

-. .-.

development process: planning, design, code, code review, compile, test, and postmortem. This included using historical data to estimate size, effort, and quality (defect projection). Actual effort and size data for each project was collected and recorded on a summary sheet. As the course progressed, students were asked to identify and record their defects; they cited the phases in which defects were injected and in phases in which they were removed. A primary focus of this effort was to encourage thorough and productive code reviews. Many of the topics were similar to those in the full PSP, but they were sealed down for first year students.

course have had some programming experience, we do not feel they are ready for a formally defined software development process; at this stage they do not understand or appreciate the problems in modem software development or have any real comprehension about the role of a software engineer. PSP topics of time management, scheduling, and planning are combined with the normal CSl material (introduction to computer science,fundamentalsof programming) [5]. Students arc required to develop and/or use several types of documents: time logs, task schedules and plat&unmary documents. After the introductorywork is completed, students start each week by estimating the effort (time) necessary to complete their CSl tasks, Throughout the week they record the time it takes to complete their tasks, and, at the end of the week, they summarize the time spent on the week’s activities and compare their estimated times with their actual times. For their programming tasks students record their estimated and actual data for both effort (time in minutes) and program size (lines of code).

0

--

on

CSl Data

the time spent on PSP activities.

In the Spring 1996 semester 88 students were initially enrolled in CSl; sixty-two (62) completed the course and took the final exam.; and forty-eight (48) students completed and turned in eight or more of the assignments (out of eleven). Twelve studentswere eliminated from this group because of suspicious data recording. This left a group of thirty-six (36) students that wem used for data analysis. The first week of PSP data was not used because it only involved data estimates, not any actual recorded data.

There were a total of eleven weeks of PSP assignments, where each weekly assignment consisted of two sub assignments: an estimated time that a student planned to spend on any of the four activities during the upcoming week, and the actual time the student spent on each of the four activities during the preceding week. CS 2 Activities

G-at 1: Average Effort Estimaticn Error sssh!dents-cs li5-Sprinq 1996

PSP processes are formally introduced in CS2. We feel that students at this level ( after finishing CSl and their first semester in college) have sufficient maturity and are ready for a more disciplinedway to develop their programs. We also feel that delaying the introduction of PSP for another semester (or year) would allow sloppy and undisciplined programming practices to become more entrenched and make them much more diMcult to change. PSI’activities were first used in CS2 in the spring semester of 1996. The material in the second half of [8] was integrated with the usual CS2 topics (introduction to data structures and algorithms) [Sl. Students were asked to estimate and record data about effort in each of seven phases of a program

Chart 1 presents the average estimation error (for the 36

279

students)for the CLASS and PROGRAMMINGtasks for weeks 2 through 11. The estimation error was computed using the formula: lOO*(actual-estimate)/estimate. The errors for PROGRAMMING indicate more irregularity in the data than for the CLASS data. This can be attributed to the variation in assignment difficulty and course topics, and the fact that students are learning new material for each assignment. On the other hand, the CLASS estimates show little error. This can be attributed to the fact that this task has less variation from week to week and, hence, is easy to predict.

Students use a PSP code review script, and a checklist they develop themselves. Since test defects are the most costly to remove, the value of code reviews seems substantial. Another indication of improvement in the development process is the decline of the percent of effort spent in the compile and test phases, shown in Chart 4.

I

Chart 3: Defect Density 51 students CS215 -Spring 1996

CS2 Data In the Spring 1996 semester, the CS2 course started with 78 students;sixty-eight (68) took the final exam; and fifty-one(51) made a C or better in the course. From the 68 students, seventeen (17), were eliminated for incomplete or suspicious data. This left a group of fifty-one (51) students that were used for data analysis (not exactly the same 51 that made a C or better in the course). Chart 2 shows estimation errors for “effort” and “size” tasks for the development of seven programs. The chart data is based on the total estimated time and the total actual time for all 51 students.

I

150 I

I

; 100 s 2 P 50 d

I

0 1

2

3

4 Programs

5

6

Chart 4: Compile and Test Times 51 students - CS215 - Spring 1996

7

I

Chart 2 indicates relatively low errors for estimating effort and size; however, a look at results for individual students (with best aggregate results) show a lot of variation in their estimationerrors. So, we do not feel that any conclusions about the overall estimating skills of the student group can be made. On a more positive side, the oscillation between overestimating and under-estimating (that can be observed in the chart and in the individual data) indicate that, in making estimates, students appear to be using information from previous estimation efforts. Chart 2: Size & Effort Estimation Error 51 studnets - CS215 - Spring~l996

5

’

1

I

I

I

I

I

L

2

3

4

5

6

7

Programs

Sue F

I

2

3

4 Progams

5

6

Effo~

7

A major objective of the PSP is to improve the quality of programs. Chart 3 shows the defect density (defects per thousandlines of code) for programs 3 through 7 (defects were not recorded for programs 1 and 2). There is a clear drop in defects for the last three programs, both for total defects and for defects found in test. The reduction in test defects can be attributed, at least partly, to code reviews, which were introduced into the process in program 5.

An interesting and beneficial side effect of this experiment has been the large supply of data that is made available to the teacher. The PSP provides data on the amount of time spent to write a program and how that time was spent. It provides a detailed picture about the quality of student programs: the number and type of errors, where they were injcctcd, and whero they were removed. Such data can provoke questions about tho effectiveness and productivity of student programmers and the quality of their work, and then provide the basis for a detailed analysis of such questions. We have found such analysis has made us more effective in providing one-on-one advise and help to our students. Conclusion Based on the objective and subject evaluations of the PSP

--

activities, in the 19951996 academic year, we do not feel that PSP objectives have been fully achieved; however, we do believe that we have made some meaningful progress and we think with additional effort and some changes in our approach we can achieve them in the next academic year. Also, we are working on a project for automating data entry by students involved in PSP activities. This will reduce the burden on students and make it easier to collect and analyze data.

----

-~---.-----~

_~- _

.._._---.

Embry-RiddleAeronautical University, August 1995.

For those students that go into their second year of the CS curriculum we will not introduce any new material in the sophomoro year, but will ask them to continue to use the PSP process they developed in CS2. They will do this in two courses: Data Structures and Algorithms and Computer Organization II. We have not yet decided what will be done in the third and four years; but, in the next year we will be working on a framework that will allow for integration and enhancement of the PSP foundation into a team project environment. Our conclusions can be summarized as follows:

WI

Humphrey, Watts S., Synder, T.R., and Willis, R.R., “Software Process Improvement at Hughes Aircraft”, IEEE Software, July 1991.

171

Humphrey, Watts S., A Discipline for Sofnare Engineering, Addison-Wesley, Reading, MA, 1995.

Bl

Humphrey, Watts S., An Introduction to the Personal Process, Addison-Wesley, Reading, MA, 1997.

Sofhare

191

Khajenoori, S., “Process-Oriented Software Education”, IEEE Sofnare, November 1994, pp. 99101.

[lOI

Neumann,

P.,

“System Development

Woes”,

Communicationsof the ACM, October 1993, p. 146.

0

The text [8] provides an excellent foundation and motivation for teaching PSP concepts.

Ull

Paulk, M., et. al., “Capability Maturity Model, Version Ll”, IEEE Sofnare, July 1993, pp 18-27.

0

There is a need for simplification and automation of forms, logs and record keeping,

WI

Tucker, A. M., Editor, Computing Curricula 19PI: Report of the ACM/IEEE-CS J@nt Cur&&m T& Force, ACM Press, December 1990.

0

We must continue to adjust and modify our teaching techniques to accommodate and integrate PSP concepts into CSl and CS2.

H31

Criteria for Accrediting Programs in Computer Science in the United States, Computing Sciences

Accreditation Board, JMUW 0

PSP conceptsneed to be integrated throughout the CS curriculum.

0

PSP data provides teachers with valuable information for analyzing student capabilities.

References

Cl1

Brooks, Frederick P., The Mythical Man-Month, Essays on Soware Engineering, Anniversary Edition, Addison-Wesley, Reading, MA, 1995.

PI

Dion, Raymond, “Process Improvement and the CorporateBalance Sheet”, IEEE Sofnare, July 1993.

131

Ford, G. and Gibbs, N.E., A Mature Profession of Sofnvare Engineering, Carnegie- Mellon University, Software Engineering Report, CMU/SEI-96-TR-004, January 1996,

r41

Hilburn, T., Hirmanpour, I. and Korneeki, A., “The Integration of Software Engineerfng into a Computer Science Curriculum”, Lecture Notes in Computer Science: Software Engineering Education, Proceedings of the Eight SEI Conference on Sojiware Engineering Education, March 1995.

151

Hilbum,T. and Stimi, A., Ada, From Top to Bottom,

281

1987.