A multi-level quality management system for software ...

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

A multi-level quality management system for software process improvement J. Simila" & E. Koskela&

o

Finland

ABSTRACT The history of the development of the quality management system of a leading Finnish software house is briefly described paying attention to the premises of setting up a quality management system and the phases of its further development and relating the process to the total quality management approach. The case study history has led to a multilevel quality system where quality is ingrained into all the major functions of a software house including software engineering, marketing and selling, personnel management, company management and administration as well as quality control and development itself. The structure of the multilevel quality system is proposed as a general structure for a quality management system oriented towards software process improvement. Quality emerges into the functioning of a company basically in two ways: in its effects in customer satisfaction and in business results. The basic metrics for measuring quality in the case study organization especially in terms of these effects are identified. Special attention is also paid to software process improvement and the factors affecting it. Finally some remarks are made on the effects of the quality management system on the identified company processes in the case study. The company has been certified according to the ISO 9001 standard in June 1992 as the first software house in Finland. INTRODUCTION What is quality? The concept may be divided into three main categories: product quality, quality of service and process quality (cf. Lillrank [15]). Taguchi [25] and Crosby [2] emphasize objectiveness and measurability when speaking about quality in general. Particularly Taguchi restricts quality only to production-

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

166 Software Quality Management related, measurable issues with an ideal of uniform and controlled quality of a product. This approach may be interpreted as a traditional engineering science view on the subject. Deming [3] believes that quality can be achieved by constantly improving the processes that produce services or products; when processes get better, quality and productivity go up too. In this article we take a definite process improvement oriented approach towards the development of a quality management system. Software process quality and process maturity models as well as assessment procedures have received keen interest during the later years, starting from Crosby's [2] original idea of the 5 stages of quality and culminating at the present in the work of the Software Engineering Institute (SEI) on the Capability Maturity Model (CMM) (cf. e.g. Humphrey [8], [9], Paulk [18], Paulk et al [19], [20], [21], [22]), in the work of the Esprit/Bootstrap project (cf. e.g. Koch [14], Bicego et al [1], Jarvinen and Simila [13]), as well as in the work of the SPICE project (cf. e.g. Dorling [6]) in integrating and standardizing the various assessment and improvement approaches. Quality has been approached also from the point of process standards and certification, most notably in the international standard series ISO 9000 framework (cf. ISO 9001 [11], ISO 9000-3 [12]), not to omit the other approaches (cf. e.g. DoD STD 2167A [5], and ESA [7]). More holistic process oriented approaches include Deming's 14 Points for Management (cf. Deming [3]), Total Quality Management (TQM) (cf. e.g. Zultner [26], Deutsch [4], and Shashkin and Kiser [24]), and Concentrated Quality Management (cf. Moisala et al [17]). In this treatise we concentrate on the fundamental development principles of the quality management system (QMS) as part of a general TQM approach basing our ideas on the experiences of developing a particular operational QMS and using this QMS as an example repository. TQM - A CASE STUDY HISTORY The case study organization was established in 1985 as an independent software house. The company has grown during the almost ten years since its establishment into a respected software producing organization with a personnel of close to 80 systems designers and a steady customer list with domestic and export references. The internal development of the company was and is based on a simple two phase approach: 1. identification of development stage and 2. selection of development steps. The identification of development stage originally utilized the ideas of Crosby [2]. Crosby's model of 5 stages of quality was modified within the

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems

167

company and has been used widely in Finland in consultation work by the company since 1987. The modification is depicted in figure 1. Since 1992 the company has been a partner in the ESPRIT/Bootstrap project (cf. Bicego et al [1]) and the software maturity assessment and development approach developed in the project is presently being ingrained into the company processes.

"JUNGLE WAR"

"PROJECT" WORK req def design coding testing

SOFTWARE PRODUCTION MANAGEMENT

Figure 1: Identification of development stages.

Retrospectively the development of the company may be seen as the implementation of several concurrent and parallel processes with a selective priority order depicted in figure 2. Each of these development steps was taken with an explicit decision in the company directory board and with the explicit idea of "small continued steps towards improvement". The process may be likened to "kaizen" (cf. Imai [10]), i.e. gradual, unending improvement doing little things better, setting - and achieving - ever-higher standards, even though the development decisions were strategic in nature. Just as well applicable is Demmg's fundamental principle: "Improve constantly and forever the system development process to improve quality and productivity, thus constantly decrease costs." (cf. Deming [3], p. 23). Not surprisingly the order of the development steps bears considerable similarity with the SEI maturity model (cf Humphrey [9]). A MULTILEVEL AND MULTIPERSPECTIVE QMS As Seghezzi [23] points out, management concepts which came on the market during the last ten years do not fulfill the requirements of the new

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

168 Software Quality Management paradigm. The new paradigm demands management concepts which multidimensional^ and concurrently consider the factors of quality, costs and time. What is required is not a new independent quality management concept but an integration of quality management as a part of a total management concept.

development level

SELECTION OF DEVELOPMENT STEPS

strategic & financial management personnel & resource management

quality control sales and marketing project management & testing

time Figure 2: Implementation of concurrent and parallel company development processes.

The main components of the multilevel and multiperspective QMS, which is proposed here and which strives towards fulfilling the requirements of the "new paradigm", include the following: quality policy, quality organization, quality control principles and levels as well as the implementation of the principles at each level, and quality manual structure which facilitates the documentation and communication of the previous components in an effective manner. The components are built in or embedded into all the processes of the production unit. The structure and form of the proposed QMS bears some similarity to the model proposed by Seghezzi [23] but has been developed independently as part of the normal organizational development of the case study company. The general structure of the multilevel and multiperspective QMS is described in figures 3 and 4. The general structure of the quality organization represents the multiperspective view of the proposed QMS. Quality is viewed from the perspectives of all the major organizational units of the company. Each view produces an independent, slightly different and complementary outlook on quality. The solution depicted in figure 3 is based on a software producing organization where the production is organized within system groups. A

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems

169

different organizational structure would require a slight adjustment but the basic solution would remain the same. The quality management levels and functions depicted in figure 4 represent on the other hand the multilevel structure of the proposed QMS where the lowest level is formed by the projects and the highest by company management.

[

Projects Project group

| Project directory board System groups persons w customer resp | [ Group managers

|

Quality function | Quality group | | Quality manager | | Quality council |

Financial fu nction Office services

Company management Managing director ] Directory groups | 1 Resource group |

Figure 3: General structure of quality organization.

Each of the components of the multilevel and multiperspective QMS is in the following treated separately firstly on a general level and then providing concrete examples of each using the case study as a repository. Quality policy The quality policy statement is defined by the top management of the company and it expresses the general objectives and attitudes of the organization towards quality. The quality policy should form an integral part of the business concept of the company. The statement of the quality policy is not in itself sufficient, what is also needed is the implementation mechanism of the policy.

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

170 Software Quality Management According to its business concept: CCC is a software house that provides customized consulting and software engineering for both export and domestic markets, emphasizing quality and delivery on time. The business concept of CCC includes the definition of its quality policy: CCC emphasizes in its software production especially quality and delivery on time.

PROJECT LEVEL FUNCTIONS Project

Project management

Technical project

Testing reviews

1 SYSTEM GROUP LEVEL FUNCTIONS Project sales

Contract reviews

Project implementation and quality follow-up

Personnel, financial, resource, and interest group management

1 COMPANY LEVEL FUNCTIONS Management and quality follow-up

Quality system development

Quality system reviews and audits

Personnel training

Figure 4: Quality management levels and functions.

Delivery on time means that the system ordered by the customer satisfies the quality objectives and is delivered in the agreed schedule and in the given frame of personnel, facilities, equipment and software resources. The quality of software production means that the customer gets the system that he/she needs and that the system satisfies the requirements of the customer. This kind of system is useful for the customer organization, it satisfies the end users, it is reliable and it is also technically effective from the viewpoint of both use and further development. The implementation of the quality policy is based on the strong project culture of CCC. Quality awareness of the personnel is the most central factor in the project culture at the individual level. Quality awareness means that each person is responsible for the implementation of the quality policy of the company and performs the tasks according to the given instructions and attempts also to develop his/her own working community. In other words everyone participates in the implementation of the quality policy and in the further development of the quality of CCC.

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems 171

Quality awareness requires that the personnel has very good project delivery know-how, customer business process know-how, information technology know-how, cooperation skills as well as self-development desire and resultorientation. At the activity level the systematicness of procedures is the central factor of project culture. This means that all tasks are performed in a planned and consistent manner using appropriate methods as well as techniques and tools supporting their use. The development of procedures takes place through cooperation between all the personnel and is led by the quality function. The central objective of the development is to increase the systematicness by decreasing rework and overlapping work and so to improve work productivity. The rework is decreased when error preventive design methods are used and sufficient investment is made for an effective testing procedure covering all the work phases with the objective of finding the errors as early as possible. The overlapping work is decreased when the work practices and methods of the organization are such that it can utilize previously done work as much as possible. CCC has a project data system that implements this principle and supports the collection of experience data and different solutions. In the development of the quality system CCC has applied total quality thinking, which means that the whole company is an object of development. This means also that the whole organization of CCC is a quality organization. Quality organization The quality organization defines the organizational structure, responsibilities, procedures, and resources for the implementation of quality management. The responsibilities of the units of the quality organization concern the different levels of the company, i.e. company management, the quality function itself, system groups, and projects. Company management The responsibilities of company management with respect to quality are the following: to determine the quality policy, to review the quality system regularly, to perform quality monitoring at the company level, and to set the resource framework for the quality system Quality function Within the quality function itself the responsibilities are divided between the quality council, the quality manager, and the quality group. The quality council determines the contents of the quality system that implements the quality policy of the company, performs internal quality audits regularly, and manages the development of the quality system. The quality manager is responsible for the development and maintenance of the quality system and implementation of the quality assurance functions in software production. The quality group performs quality assurance work and quality monitoring at the project and system group level, participates in the development of the quality system, and gives quality support and education to the system groups

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

172 Software Quality Management System groups Within the system groups the responsibilities with respect to quality are divided between the group manager, the person with business area responsibility, and the person with customer responsibility. The group manager is responsible for the implementation of the projects of the group according to the quality system and responsible for the personnel management of the group. The person with business area responsibility is responsible for the business activity on the business area, performs quality monitoring of the project and when needed makes corrective actions. The person with customer responsibility is responsible for specified customer relationships and the quality of the cooperation Projects Within the projects the responsibilities are divided between the project directory board, the project manager, project group, and the test support person. The project directory board performs the quality monitoring of the project, and makes decisions concerning the changes of the project contract. The project manager is responsible for the carrying out of the project according to the plans and decisions of the directory board, performs quality assurance according to the quality system within the project, makes the project plan which includes also the quality plan that defines the quality practices in the project, and gives feedback about the quality system. The, project group is responsible for the progress of the project according to the plan, implements the quality policy by performing the project tasks according to the given guidelines, and gives feedback about the quality system. The test support person is responsible for the planning and implementation of testing together with project group. Quality control principles and levels The central principle of the quality control is that quality is created by making not by monitoring. The projects and groups are responsible for the production of quality. The task of the quality function is to support the projects and system groups so that they are able to reach their quality objectives. CCC's quality function is clearly a support activity by its nature. On the other hand it is also mainly an error prevention activity even though its task is also to react immediately to the situations requiring error corrections. From the viewpoint of the customer the quality function offers a channel which he/she can use to express his/her problems and hopes directly to the management of software production. The quality control takes place in CCC at three levels: at the company level, system group level and project level. Quality control at the company level At the company level the quality control concentrates on the evaluation and development of the quality system and the software production process, the implementation of internal quality education and on the quality monitoring concerning the company level.

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems

173

The management reviews and evaluates the quality system yearly from the viewpoint of the business concept. The quality council makes quality audits every six months and evaluates the contents of the quality system The evaluation is based on the results of the quality monitoring that are the reports of quality function, customer and personnel feedback as well as the key figures of the quality measures. The development takes place under the control of the quality council and by the quality manager and quality group in cooperation with all the personnel The guidelines of development are set in the yearly plan of the quality function The management follows the quality of the projects continually on the basis of the reports of the projects and the quality group and makes corrective actions if needed. Group level quality rontrol At the system group level the main areas of quality control are personnel, economy, resource and interest group management management of sales projects as well as the implementation of project and quality assurance. The activities are based on the yearly plan of the system group. ^ The development of quality awareness and the professional skills of the ** ™°* ™P°"*" tasks of personnel management at the system The success of projects requires that the projects have the right resources ine group manager is responsible for the resources of the projects The resource group co-ordinates the resource allocation between the system groups The system groups are responsible for the contracting and implementation of the projects. From the viewpoint of the quality, continuous quality monitoring and quick corrective actions are important. Project level quality control The quality control at the project level concerns the quality ot the production process and that of the product. The quality of the project is sufficient, when it can be carried out in the given frame of resources and when it is possible during the project to develop cooperation between the different interest groups according to the objectives put forward in the project plan. The quality of the product is sufficient, when it satisfies the requirements of the customer. The product must be useful to the organization of the customer, it must satisfy the end users, and it has to be reliable and technically effective from the viewpoint of both use and further Development. . , *e Project level is divided into two parts: project control and the technical project that includes depending on the project: feasibility study specification, design, implementation, and testing phases as well as installation A II (Imaintenance. mnint"£»nanr»£» and

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

174 Software Quality Management The quality assurance is also divided into two parts: the internal and the external quality assurance. They concern the quality of both the product and the project. The project is itself responsible for the internal quality assurance. The quality manager and quality group are responsible for the external quality assurance. The customer may also be responsible for the external quality assurance depending on the contract. The internal quality assurance of the product takes place according to the accepted testing plans and the internal quality assurance of the project takes place in the project meetings led by the project manager. The test support person together with project group are responsible for the preparation of testing plans. The external quality assurance of the product takes place in the testing reviews and of the project in the project reviews. Quality manual The quality manual documents the quality system of the company and also provides the multiperspective view on quality required for its implementation. The quality manual of CCC follows the ISO 9001 and 9000-3 standards. The quality manual consists of 9 parts: I General description of the quality system, II Quality instructions for company management, III Quality instructions for system group, IV Quality instructions for project control, V Quality instructions for design and testing, VI Instructions for the quality function, VII Instructions for the economy function, VIE Instructions for office services, and IX General instructions. QUALITY METRICS APPROACH - BASIC DATA FOR PROCESS IMPROVEMENT The main measurements for measuring quality in the case study are: 1) customer satisfaction profile of a project and over the projects, 2) project productivity profile and summary profile over the projects, 3) project quality profile and summary profile over the projects, and 4) employee satisfaction profile on quality system and the results of supporting functions of the company [cf. e.g. the criteria of the European Quality Award]. Customer satisfaction profile The customer satisfaction profile includes the following measures: satisfaction on the project, satisfaction on the results of the project, satisfaction on the services of the project. The customer satisfaction measure at the project level is based on actual data gathered from the customers of the project using customer questionnaires. The main factors that customers are asked to evaluate include among other things delivered products, co-operation, reporting, working practices, professional skills of workers and their co-operation capabilities. These factors and their sub factors are evaluated using a five value semantic differential. At the group level

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems

175

the feedback of customers is summarised by calculating the numbers of the different values given by the customers to the questions in the questionnaires, over the projects and different time periods within the group and by presenting the results as summary and scenario diagrams. At the com/xwy 7eW the same kind of summary data are produced from the group level data. Project productivity profilp The productivity profile consists of six components: 1) the difference between the planned and real amount of work, 2) the share of rework, 3-5) the effectiveness of specification, design and implementation testing, and 6) the quality index of the product ex post. The basic data are gathered using work amount, and error and correction work amount matrixes. In the work amount matrix we have for each activity the planned and real amounts of work. In the error and correction work amount matrix we have for each phase of testing (i.e. from the specification phase to the system testing phase and use phase during the warranty time) the amounts of errors found and amount of work used for the correction of different types of errors (i.e. specification, design and implementation errors). Using this data we can determine the components of the profile. The rework of a project is the sum of error correction work. The effectiveness of (specification, design and implementation) testing is the percentile share of errors found in the phase in question (specification, design, implementation) compared to all the errors of the same type found during the whole life cycle of the system up to the end of the warranty time. The quality index of the product is the percentile share of the errors found during the development process compared to all the errors found up to the end of the warranty time. The differences and the share of rework are classified into five categories valued 1-5: 1= over 20 %, 2= 16-20 %, 3= 11-15 %, 4= 6-10 %, and 5= 0-5 %. The effectiveness values and indexes are also classified into five categories valued 1-5: 1= under 60 %, 2= 61-70 %, 3= 71-80 %, 4= 81-90 % and 5= 91100 %. At the group and company level the productivity profiles are determined by calculating the numbers of projects belonging to the different classes according to the components of the project productivity profiles over the groups and/or different time periods and presenting the results as summary and scenario diagrams. The productivity profiles are supplemented by economical results (e.g. project net profit). An example of project productivity profiles is provided in figure 5. Project quality profile The project quality profile is based on the subjective assessments made by the quality group on how well the instructions of the quality manual of the company are followed in the projects using a five value semantic differential. Things

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

176 Software Quality Management evaluated are such as project planning, project control, reporting, risk and configuration management, and testing. Based on the values above the quality index is calculated for the project as a weighted average. At the group level the quality index for the group is determined as an average of the quality indexes of the projects of the group and the company level the quality index is an average of the group indexes.

Project productivity profiles in the system group 2

Project 1

Project 2 I Planned vs actual effort • Effect, of design testing

Project 4 3 Effect of specs testing E3 Effect, of impl. testing Q Quality index of product

Project 3 O Share of rework

Project 5

Figure 5: An example of project productivity profiles.

Employee satisfaction profile The employee satisfaction profile is based on the following measures: satisfaction on quality function operations (e.g. project and testing reviews, methodology support, quality training, quality instruction manual), satisfaction on personnel management operations, satisfaction on office services, and satisfaction on economical services. The employee satisfaction data are gathered with the aid of the questionnaires concerning each topic in question and using a five value semantic differential. The individual opinions are summarised by calculating averages and modes for all the factors and presented as summary and scenario diagrams. EFFECTS OF QMS IN TERMS OF COMPANY PROCESSES Even though we have not yet made any thorough formal evaluation of the effects of the developed QMS on the business processes of the company, a quite strong indication of the effectiveness of the soundness of the proposed and implemented approach is provided by the business position of the company. The company has been growing steadily both in terms of revenue and personnel through the most difficult years in recent history in the Finnish economical life. The present depression seems to have only strengthened the business position of the company.

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

Managing Quality Systems

177

REFERENCES 1. Bicego, A., Cacchia, R., Haase, V., Koch, G., Krzanik, L., Kuvaja, P., Lancelotti, R., Maiocchi, M., Messnarz, R., Saukkonen, S. Schynoll, W.] Simila, J. 'Bootstrap: Europe's Assessment Method.' IEEE Software pp 9395, May, 1993. 2. Crosby, P. B. Gwa% ^ Frgg - 7Ag Arf McGraw-Hill, 1979. 3. Deming, W. E. Out of the Crisis, MIT Center for Advanced Engineering Study, Cambridge, MA, 1986. 4. Deutsch, M. S. Total Quality Management in Hughes Aircraft', Proceedings of the Esprit Bootstrap Conference on Lean Software Development Stuttgart, October 22-23, 1992. 5. DOD-STD-2167A, 'Military Standard DOD-STD-2167A', Defence System Software Development, February 29, 1988. 6. Dorling, A. 'SPICE: Software process Improvement and Capability dEtermination.' Information and Software Technology, Vol 35 No 6/7 nn 404-406, 1993. ' " 7. ESA, ESA (European Space Agency) Software Engineering Standards ESA PSS-05-0 Issue 2, February, 1991. 8. Humphrey, W. S. 'Characterizing the Software Process' IEEE Software, Vol. 5, No. 2, March, pp. 73-79, 1988. 9. Humphrey, W. S. Managing the Software Process, Addison- Wesley Reading, MA, 1989. 10. Imai, M. Kaizen: The Key to Japan's Competitive Success, McGrawHill, New York, 1986. 11. ISO 9001, Quality Systems; Model for Quality Assurance in Design/Development, Production, Installation and Servicing, 1987. 12. ISO 9000-3, Quality Management and Quality Assurance Standards, Part 3: Guidelines of ISO 9001 to the Development, Supply and Maintenance of Software, 1991. 13. Jarvinen, J. and Simila, J. 'Maturity Level Definition: A Comparative Analysis.' in Proceedings of the 16th IRIS, (ed. Bansler, J. P., Bodker, K., Kensing, F., Norbjerg, J., and Pries-Heje, J.), Information systems Research seminar In Scandinavia, 7-10 August, 1993, Rapport Nr. 93/16, Department of Computer Science, University of Copenhagen, 1993.

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

178 Software Quality Management 14. Koch, G. R. 'Process Assessment: the 'Bootstrap' Approach', Information and Software Technology, Vol. 35, No. 6/7, pp. 387-403, 1993. 15.

Lillrank, P. Laatumaa (Land of Quality), Gaudeamus, Jyvaskyla, 1990.

16. Manns, T. and Coleman, P. Software Quality Assurance, MacMillan, Basingstoke, 1988. 17. Moisala, U. M., Vuorinen, R. and Moisala, A. CQM tyoyhteison muuttajana (Changing Work Community with CQM), Weilin+Goos, Hameenlinna, 1989. 18. Paulk, M. C. 'U.S. Quality advances: The SEI's Capability Maturity Model', Proceedings of 3rd European Conference on Software Quality, Madrid, November 3-11, 1992. 19. Paulk, M. C., Curtis, B., Chassis, M. B. et al, Capability Maturity Model for Software, Software Engineering Institute, CMU/SEI-91-TR-24, DTIC Number ADA240603, August, 1991. 20. Paulk, M. C., Curtis, B., Chrissis, M. B. and Weber C. V., 'Capability Maturity Model, Version 1.1', IEEE Software, pp. 18-27, July, 1993. 21. Paulk, M. C. et al, Capability Maturity Model for Software, Version LI, Software Engineering Institute, Technical Report CMU/SEI-93-TR24, 1993. 22. Paulk, M. C. et al, Key Practices of the Capability Maturity Model, Version LI, Software Engineering Institute, Technical Report CMU/SEI-93TR25, 1993. 23. Seghezzi, H. D. 'Europe as Part of the Triad', in EOQ 93 (Ed. Anttila, J.), pp. 12-25, Proceedings of EOQ '93 World Quality Congress, Helsinki, Finland, June 15-17, 1993. Salpausselan Kirjapaino, Finland, 1993. 24. Shashkin, M. and Kiser, K. J. Putting Total Quality to Work, BerretKoehler, San Fransisco, 1993. 25. Taguchi, G. Introduction to Quality Engineering, Asian Productivity Organization, Tokyo, 1986. 26. Zultner, R. E. 'Software Quality Engineering: The Deming Way', American Programmer, Vol. 3, No. 11, pp. 2-11, 1990.