An Ongoing Experiment in O-O Software Process and Product Measurements Sita Ramakrishnan Department of Software Development Monash University PO Box 197, Caul eld East Australia 3145 email:
[email protected] 21 July, 1995
Abstract
This paper reports on an ongoing software measurement experiment which has been set up to monitor and evaluate team based student projects on a progressive basis. Student Assessment Tutorial Ongoing Pro le Sheets (SA-TOPS) have been used to provide a more objective feedback to improve students' learning and for measuring their understanding for grading purposes. The objective of the measurement is to improve the quality of the software process and product by measuring the activities of a scenario based development of an Object-Oriented system by various student teams.
1 Introduction Software process improvements using the top-down process such as a Capability Maturity Model (CMM) and the bottom-up process such as the Quality Improvement Paradigm (QIP) [16] have been promoted by the Software Engineering Institute and NASA Software Engineering Laboratory respectively. Integration of a well organised software process and software measurement with an aim to improve the quality of the process and product is a valid experiment [1] for objectively assessing students' projects [6]. The views expressed by a panel held in OOIS'94 that proper mechanisms for educating new entrants to the O-O paradigm is vitally important for this new wave of development to become mainstream. Teaching a measurement based quality improvement approach for developing O-O projects is an important step. This paper discusses an experiment in O-O process and product measurement. Section 2 outlines the motivation behind the experiment which is followed by a section on measurement and experimentation. The experimental results are discussed and conclusions are given.
2 Motivation A third year one semester undergraduate unit called Object-Oriented Programming Systems (OOPS) at Monash University has been taught by the author since 1991. Students doing the OOPS unit in the past have been asked to work on a set of small BANK exercises to master the O-O terminologies and apply them to implement the BANK exercises. The set of exercises was given as tutorial laboratory tasks after covering the topics in the lectures. This was followed by two major assignments, the rst one where the analysis and high-level design were given to the students and the students were required to work in groups of three to produce a working system in seven (elapsed) weeks. The second assignment 1
was to produce design diagrams and documentation for a given problem. Although students were working from the same high-level design which had been produced by \experts", the quality of the implemented product varied dramatically between student groups. This resulted either owing to the team deliberation going for too long without any detailed monitoring/controlling mechanism, or to non-committed or weak members in a group. The organisation of the software process was adhoc and at SEI CMM's initial process level [9]. The project management processes to elevate the project to be a repeatable experiment was introduced. This experiment has been set up to test the hypothesis that improvements in O-O development process and product can be achieved in a project management framework by following formal inspection procedures.
3 Measurement and experimentation 3.1 Educational Objectives This software process experiment on measurement pays due regard to the educational objectives as discussed by [4] such as mastery of knowledge, comprehension, application, analysis, synthesis, and evaluation. These levels of mastery of a topic, ranging from simple object-oriented terminologies to the ability of project synthesis and evaluation are being measured using Student Assessment -Tutorial Ongoing Pro le Sheets ( SA-TOPS). With the introduction of SA-TOPS, students have been progressively assessed from week 3 of the course as per Bloom's educational objectives. The assessment \carrot" has become the invisible stick and students have been assessed using a prescribed process leading to a more controlled progressive monitoring of their project (assignment) work. Process assessment aims to analyse the process for process improvements. Product assement is meant to address the software quality based on Boehm's quality tree criteria [3]. These activities give students insight into a scienti c and engineering approach to building software systems [19]. Inspection sessions are crucial for keeping a mutual shared understanding of the state of the project [5]. Inspection process is used to track defects and can be used to arrest any common patterns of errors that may be arising due to misunderstanding of concepts. An important di erence between the traditional software systems development and Object-Oriented Systems development is in the area of project management. The recommended incremental iterative cluster based development for O-O software systems forces the development team to move through the various phases of the software life-cycle during the development of a cluster. Students are required to estimate the time required to complete these activities and track the actual time taken so that they can measure the resources spent by various members of a group and also gain appreciation of the diculty in project cost and time estimation [19]. Graphs of class sizes, classes and methods against time are particularly relevant in this O-O scenario based development where classes which participated in more than one scenario had to be reopened in some cases for adding new methods [8]. These statistics are also used to compare various student groups' performance and help in arriving at a more objective measure for their grades.
3.2 Experimental Design An empirical method based on statistical principles of formal measurement in a software engineering setting [2] has been used for the software measurement experiment. As a number of student teams have been developing the same project, the experiment is an example of a replicated project where the basic experiment is repeated by various teams and therefore satis es the principles of experimental design. The results of the initial experiment may be diluted by widening the scope to include the
entire software development life cycle [13]. Hence, the scope of the experiment has been narrowed by providing the students with a high-level design for the assignment. As testing and inspection are empirical activities, they are used as valid areas for software measurement and experimentation [19].
3.3 The Experiment The experimental method has been adopted from [2,3,9,10,11,13 and 18]. Students are given an assignment about a conference management system. It is based on the rst case study given in [17]. They are required to use the high-level design given in that text for their system development and are monitored using project management techniques. They follow the following prescribed process for a scenario at a time : list tasks and estimates in a gantt chart; seek approval from the lecturer for the contract showing each member's obligations and record defects found in the various phases during inspection and testing. Templates of inspection documents based on [7, 15] are given to the students. They are also required to consider external software quality criteria such as functionality, usability, maintainability, reliability and performance in building the system and incorporate internal software quality criteria such as modularity, documentation, assertion and reuse to improve the product quality [12, 3]. They demonstrate their work in the labs and together with their documentation folder form the basis of their evaluation and suggestion for improvement. Figure 1 shows a number of templates that were developed for this experiment: g.1.1 - a contract template for scenarios, g.1.2 - a project management template to be used for each scenario to show the break up of phases, tasks and subtasks in the software development, life cycle and their schedules, g.1.3 - SA TOPS and g.1.4 - defect types. The experiment in this study involves fteen student teams (in groups of three) who are required to follow a prescribed process for an incremental, iterative development to implement a Conference Management System. They are required to use the same high level design and produce a low-level design, coding and have organised inspection meetings to review the team members' work, document the defects using the given defect types [15]. The test plan covers black-box or speci cation testing, program testing, white-box and integration testing [10]. The inspection process is used to detect, record and correct the errors captured at various phases of the software life-cycle. (refer to Figure 3 for the defect types and their descriptions used in the inspection process). Inspection process is used to gather statistics about the time taken to detect and record the errors. Testing captures the errors that went undetected during inspection. The emphasis of the process model prescribed for this measurement experiment is on producing charts for project management information, creating test plans that include Class (program) and integration (system) testing, and reporting various defect types found during inspection.
3.4 Process and Product Measures A combination of process [8] and product assessment [1] measurements have been used in our experiment. The project management metrics collected during low-level design, inspection and testing of classes and methods are analysed using graphs. The software products are assessed by analysing the defects recorded according to various defect types [15] during inspection and testing [1] and by using the quality criteria of Boehm [3] and Meyer [12]. The experiment may produce awed results when the developers are new to an approach as their learning process a ects the results produced [13]. This aspect has been taken into account as the students are new to the O-O development approach. It must be pointed out that our students would already be well versed in setting up test plans for conducting program and system testing. The measurement of the software process and its products for the rst scenario are only used for feedback to students rather than for assessment purposes. The next scenarios form part of the measurement requirements. This helps students to perform the activities of the next scenarios with a
better understanding thereby achieving an improvement in the quality of the process and the product.
3.5 Scenario based incremental development The scenario based development of the system which is followed in our O-O system development means that the team members participate in a scenario at a time and follow the prescribed process and are assessed and their process and products are compared against the other teams. Student teams make a presentation in week 6 of the course and give an estimate of their time/activities as a gantt chart for fully implementing scenario 1 and sign a contract declaring the roles and responsibilities of the team members for scenario 1. Once it has been approved by the lecturer, they are required to implement this scenario in two elapsed weeks and follow the prescribed process. Regular inspection meetings are held by the teams where defects found are recorded and graphs are produced to show the time spent vs defects found during inspection and also during testing. Graphs to show class size vs time and number of methods developed vs time are also produced. A review/evaluation for scenario 1 is led by the lecturer in the lecture so that students can use this feedback mechanism to improve their process from scenario 2 onwards. The incremental iterative model of developing O-O software has been exploited to integrate the process maturity level [14] to metrics collection. [19] gives examples of areas within the undergraduate computing curricula which would bene t from software measurement and experimentation. His model ts with our curriculum objective of teaching O-O paradigm and applying quality improvement measures in preparing students for the new ways of incremental iterative development method of Object-Oriented System development.
4 Experimental Results and Discussion The data collected during the semester regarding the defects found during inspection and testing and the time spent for these activities have been summarised in Figures 2 - 7. The process gures also showed outliers which were used to discuss ways to improve the low-level design and code. For example, team 3 reported 118 defects in total during scenario 1 inspections (refer to g.7). Ofcourse, the whole experiment rests on students' cooperation. Some groups which have not had regular inspection meetings owing to timetable clashes and other constraints of the team members have relied on testing alone to detect errors (refer to gs. 3 - 4). Figures 2.1 - 2.3 show the number of defects found per defect type during the inspection process. Although the defects seem to be spread across all defect types, a closer examination reveals that they are clustered around defects in logic (CL,FU,LO), communication between components (IN), defects in language usage(SN) and human factors (BL, HF and TY). The defect types reported for inadequate or incorrect documentation (DO), forgotten features (FO) and internal data use errors (DA) are on the low scale which is reasonable as the high-level design with detailed static and dynamic diagrams were provided to the students. The very low gures for quality, performance, reuse and robustness (QU,PF,RE,RO) improvements recorded by the teams could be interpreted as reasonable from novice O-O teams. The students' understanding of these qualities were assessed during their formal demonstration of each scenario. The above analysis of each team's inspection and testing (refer to gures 2 - 4) was followed by a statistical process control (SPC) technique [18] to identify poorly inspected and defect prone work products [15] by producing control charts and dispersion charts. In this study, SPC technique was used to view the diversity of data of teams from inspection and testing results and to assess the quality of the work product of the various teams. In fact, this technique could be adopted during the various phases of a development process to manage and control the quality of the product. The control charts (refer to gures 5.1, 6.1) show defect densities per team for the three scenarios.
These charts show the mean defect density (U = number of defects/total work product size), upper control limit (UCL = U + 3 * sqrt(U/average work product size) and lower control limit (LCL = U - 3 *sqrt(U/minimum work product size). Inspections with defects greater than UCL are regarded as potentially defect prone products and those below LCL are considered poorly inspected. The control chart (refer to g.5.1) showed team 3 with potentially defect prone products and teams 10 & 11 with poorly inspected products. The dispersion chart (refer to g.5.2) was done to con rm the ndings from the control chart. This chart was plotted to show the dispersions of the examination rates for the various teams, an average and a upper limit of two standard deviations from the average. Data over this upper limit were looked at more closely. The dispersion chart showed that examination rate of team 3 was within the norm and hence was considered as well inspected and error prone. The two teams 10 & 11 shown as poorly inspected in g. 5.1 had high examination rates, and so did 7, 8, 10, 11, 13, 14 & 15. Since the data recorded by the teams were not complete in some cases (refer to inspection & testing charts - gs.3 - 4 ), a control chart and dispersion chart was prepared by combining the inspection and testing data. The second control chart (refer to g.6.1) still showed team 3 above UCL and team 10 was now below LCL. The second dispersion chart (refer to g.6.2) showed teams 3, 12, 14 & 15 with examination rates above the norm. Teams 10 & 11 were no longer in this high end, and suggests that their inspection and testing process was not perhaps consistently performed or recorded. (Refer to [15] for more details of this analysis technique). The ratio of inspection and testing time (refer to g.7) shows teams 1, 2, 3, 5, 6, 9 & 12 have bene tted from the organised inspection process. This data was used in conjunction with the above control charts and dispersion charts to arrive at teams who may be categorised as those who have achieved quality improvements. These teams would have a common knowledge of understanding about the interactions within each scenario and be better equipped to achieve quality improvements in this incremental development approach. These teams can be said to have a shared understanding of the OO material covered in the course and a shared memory of the project [11]. This shared understanding was proven by team members achieving very similar results in another individually assessed component. A couple of individuals from teams 7 and 15 were high achievers in the individually based assessment component and this suggests that these teams did not develop a shared memory since they did not follow an organised inspection and testing process and perhaps also due to the di erence in the calibre of team members. This experiment has shown that a team based object-oriented development will de nitely bene t from an organised inspection process and cost less resources to develop the system.
5 Conclusion A repeatable process experiment has been set up to assess students on their O-O project. The process requires student teams to work on a scenario at a time and work in an incremental iterative style of development. The students collect data about their development process and the product quality using the templates given for inspection and testing. The data across the student teams have been compared using graphs to show di erences visually and used in class to discuss bottlenecks and outliers. One of the important bene ts of the introduction of a repeatable process in an educational context has been to make the process aspects visible and available for comparison across student teams more objectively than it has been possible in the past. A number of these controlled experiments have to be conducted before measurements on scenario based development undertaken with formal inspection process and testing to improve the process and product may be validated. More data needs to be collected for making such speci c claims. An on going experiment will be considering the tools for storing the metrics data and data analysis. This will ensure that meaningful metrics can be produced by this controlled software process and product improvement experiment.
Acknowledgment Thanks to Monique Spence from our department for helping with data collection by setting up the templates for inspection and for keeping records on SA TOPS. I would also like to acknowledge the contributions of SFT3021 students of semester 1 1995 towards this experiment. An earlier version of this paper is available as a technical report - TR95/8 from the Department of Software Development, Monash University.
References [1] Bache, R., and Bazzana, G. Software Metrics for Product Assessment. McGraw-Hill, 1994. [2] Basili, V. R. The experimental paradigm in software engineering. In Experimental Software Engineering Issues: Critical Assessment and Future Directions, International Workshop, Germany, H D Rombach and V R Basili and R W Selby (Eds.), LNCS 706, Springer-Verlag (1992), pp. 3{12. [3] Basson, H., Haton, M. C., and Derniame, J. C. Use of quality characteristics graphs for a knowledge-based assistance in software quality management. In Proceedings of Software Quality Management, Elsevier Science and CMP, Southampton (1993), pp. 807{818. [4] Bloom, B. Taxonomy of Educational Objectives: Handbook I: Cognitive Domain. David McKay, New York, 1956. [5] Bruegge, B., and Coyne, R. F. Teaching iterative and collaborative design: Lessons and directions. In Software Engineering Education, 7th SEI CSEE Conference, San Antonio, Texas, USA, J L Diaz-Herrera (Ed.), LNCS 750, Springer-Verlag (January 1994), pp. 411{428. [6] Fenton, N. E. Software Metrics. Chapman and Hall, London, 1991. [7] Gilb, T., and Graham, D. Software Inspection. Addison-Wesley, Wokingham, England, 1993. [8] Grady, R. B. Practical Software Metrics for Project Management and Process Improvement. Prentice-Hall, Englewood Cli s, 1992. [9] Humphrey, W. Managing the Software Process. Addison-Wesley, Readings, Mass, 1989. [10] Jacobson, I., Christerson, M., Jonsson, P., and Overgaard, G. Object-oriented software engineering. Addison-Wesley, Readings, Mass, 1992. [11] Konda, B., Monarch, I., Sargent, P., and Subrahmanian, E. Shared Memory in Design: A Unifying theme for research and practice, Research in Engineering design, Vol. 4, pp. 23 - 42. 1992. [12] Meyer, B. Object-oriented Software Construction. Prentice-Hall,Hemel Hemstead, 1988. [13] Mohamed, W. E., Sadler, C. J., and Law, D. Experimentation in software engineering: A new framework. In Proceedings of Software Quality Management, Elsevier Science and CMP, Southampton (1993), pp. 417{430. [14] Pfleeger, S. L., and McGowan, C. Software metrics in a process maturity framework. Journal of Systems and Software 12 (July 1990), 255{261. [15] Strauss, S. H., and Ebenau, R. G. Software Inspection Process. McGraw Hill, 1994. [16] Thomas, M., and McGarry, F. Top-down vs. bottom-up process improvement. IEEE Software 11, 4 (July 1994), 2.
[17] Walden, K., and Nerson, J.-M. Seamless Object-Oriented Software Architecture. PrenticeHall, Englewood Cli s, New Jersey, 1995. [18] Wheeler, D. J., and Chambers, D. S. Understanding statistical process control. SPC Press, Knoxville, Tenn., 1986. [19] Zweben, S. E ective use of measurement and experimentation in computing curricula. In Experimental Software Engineering Issues, International Workshop, Rombach and Basili and Selby (Eds.), LNCS 706, Springer-Verlag (1992), pp. 247{251.
