Aspects of Process Quality Sivert Sørumgård and Guttorm Sindre Division of Computer Systems and Telematics The Norwegian Institute of Technology N-7034 Trondheim, Norway Internet:
[email protected] Abstract. Quality is a concept which is hard to define and measure. Within software engineering, the term quality is usually taken to mean software product quality. However, with increasing awareness and focus on the software development process, quality considerations should be applied here as well. An interesting approach is to consider if the current way of doing product quality assessments can be applied to software processes.
1. Introduction Quality appears to be one of the most fundamental concepts in our lives (Pirsig, 1974). Quality is ‘hard to define, impossible to measure, easy to recognise’ (Kitchenham, 1989). However, this should not keep us from being conscious about quality since constantly reminding ourselves of the difficulty and uncertainty of quality considerations most certainly will not improve anything at all. Software development can be considered as comprised by three basic components: Products, processes and resources (Fenton, 1991). In the context of quality assessments, only products have received any substantial attention. But during the last years attention to the process has been considered increasingly important as a means for improving product quality. Our interpretation of the term “process” is that it is the way people perform work in order to produce a software product, and it includes tools, methods and techniques which are used (Humphrey, 1989). Process quality improvement obviously requires an understanding of what process quality is. This can be done by simply stating that a process that delivers a high quality product has a high quality. However this definition is far too simple, it does not consider obvious aspects as stability, timeliness, cost etc. Our goal is to look at the process as a self contained entity and establish a notion of process quality based on the process itself rather than its surroundings. Due to similarities between a software product and a development process, our first approach is to consider the applicability of product quality models for processes. 1.1. The contents of this paper Section 2. discusses how the concept of quality has been treated and modelled in software engineering earlier. This discussion will refer to relevant work on quality criteria and quality factors (“ilities”) which have been recognized as important for software products.
1
Section 3. is looking at processes and trying to find what properties and characteristics of a process are important in terms of quality considerations. This section will also examine some existing approaches to process quality and point out why our approach is different. Section 4. is based on the preceding discussion, and will try and determine to what extent a traditional quality model can be applied to assess the quality of a software process. In particular we will identify important quality aspects not covered by a traditional model. Section 5. provides some concluding remarks to summarize the paper, and points out what areas appear particularly important for further research on process quality.
2. Software Product Quality There are two main interpretations of software product quality (Gillies, 1992): One considers a software product as derived from a specification, and thus defines quality as the degree to which the product corresponds to the specification (Crosby, 1979). The other interpretation is based on the view that a software product is a solution to a problem, and thus the quality of the product becomes to which extent the problem is solved (Juran, 1974). We will use the latter definition throughout this paper. 2.1. Quality factors For the purpose of discussing product quality, we refer to work by Deutsch and Willis (1988). Their quality model has a number of similarities with other early work on quality (Gillies, 1992) by e.g. McCall (1977) and Boehm (1978). In their view, product includes code, design and documentation. Their model makes a distinction between aspects of quality that an engineer can affect (quality criteria) and aspects that are important to the user according to the “fitness for use” view (quality factors). There is some confusion about the use of the terms ”criterion” and “factor”, we will try to be consistent with the interpretations mentioned above. The user´s view of quality is decomposed in a way that is similar to the models of McCall and Boehm, as shown in Figure 3. For the four low level quality aspects, the Operational
Functional Performance
User needs Maintenance
Change Management
Figure 1 - User
needs (Deutsch, 1988)
breakdown into quality factors results in a list of 15 entries. For a more elaborate discussion of these quality factors, we refer to Deutsch (1988).
2
The 15 quality factors are based on how the user perceives quality. In order to actually improve quality we need to tie the quality factors to attributes of the product which we can change. These “low level” attributes are called quality criteria, and Deutsch and Willis identified 27 such criteria as well as their relations to the quality factors. Improvement however makes little sense if we do not know what to improve. Thus, we need to be able to measure quality (Fenton, 1991). There have been suggested a number of measures (sometimes called metrics) for the different quality factors - however they often have weaknesses. One of the major problems is that a small number of metrics are used to measure several quality criteria (Gillies, 1992). The way individual measures are combined into an overall quality value is also questionable - a number indicating the quality of a software product is not meaningful unless we know exactly how the number was obtained and how the different criteria were traded against each other.
3. Process Quality Defining what quality means for a software development process is even harder than it is for products. As we mentioned in the introduction, high quality products are a desirable and expectable outcome of a high quality process. However we need to understand process quality beyond this level in order to know how to accomplish process improvement. The existing understanding of what process quality is seems limited. Many approaches focus on time, effort, productivity and defects (Fenton, 1991). Others consider quality to be stability and control, and thus the variability of the output to be a quality indicator (Yeh, 1993). A development process is too complex to be considered as one entity, we need to decompose it. Since we adhere to the view of quality as “fitness for use”, a decomposition based on the “uses” of the process may be a good approach. Since “use” also involves a “user”, another approach may start by finding the roles of people involved in software development. We consider this approach to be preferable because it also reflects the fact that people are the perceivers of quality, and thus importance and meaning of quality factors are likely to stay consistent within the view associated to one role. We can identify the following roles and informal descriptions of their perception of what is important for a development process: • The end user will focus on the product being developed. With respect to the devel-
opment process, the end user expects her opinions on functionality to be taken into account when the system is specified. The product must also be pleasant and easy to use. She expects the system to be delivered on time, and she expects good support regarding training and maintenance.
3
• The customer pays for the product, and will thus expect it to be delivered accord-
ing to the specified requirements, on time and within budget. It should also be developed according to current standards. The customer may require particular development techniques or methods to be used. Service, which includes training and technical support, is also of high importance. It is expected that all information from the customer is kept confidential by the developer. • Developers comprise several people from requirement analysts to test engineers. Their view is that the process should be comfortable to work in, enough time should be left for developing quality products. They dislike working under too much pressure, and they want their effort to be appreciated. Motivation and satisfaction is of high importance. Training and knowledge in tools and techniques is also desirable. • The manager is concerned about satisfying the customer as far as possible, yet still the development should yield an acceptable profit. He wants the organization to be as productive as possible. Use of tools and techniques is judged according to increase in productivity and competitive advantage. The process should be easy to manage, maintain, adapt and improve. The fact that roles and views are important when discussing quality has been recognized before (Gillies, 1992). However, this does not seem to be exploited to any substantial extent in many of the quality models. There is a significant difference in the way the different roles perceive the development process. End users and customers are primarily concerned about attributes of the product being developed, e.g. its price, quality and defect density. As for the process, they consider effort (which determines price), timeliness and communication with the development organization to be important. Managers (i.e. managers of the development organization) are primarily concerned about productivity, cost and quality similarly to the customer. A cost/benefit analysis will have large influence on the manager´s opinion about the process. Since the manager is administrating the process, he requires it to be easy to change and adapt. Developers, on the other hand, have a completely different view of the process. They will be more likely to consider the process as a self contained entity with a notion of quality independent on other entities. Their view is that a process which allows them to use and develop their skills is good. Psychological aspects such as motivation, appreciation and stress are crucial (DeMarco, 1987). 3.1. Current software quality improvement efforts Software process improvement can be addressed from many different angles. Development methodologies and corresponding CASE-tools are supposed to help improving quality - and often do - by providing development guidelines and automating some of the tasks. However, criticism has been raised that the introduction of tools will not help if the development organization does not have the sufficient maturity, and often there is a need to discuss software process quality independently of specific development methods.
4
For this purpose, various standards are being suggested for software process assessment and capability determination, such as ISO-9000 (ISO, 1987; ISO, 1991), SEICMM (Humphrey, 1987; Paulk, 1993), SPICE (Dorling, 1993), TRILLIUM (Bell, 1992). These efforts typically include statements about practices that should be included in a good software process, such as audits, corrective actions, training, process definition (Bamford, 1993). From the knowledge of what key practices are mastered and not in an organization, one will also have an idea about how to proceed to improve process quality. E.g., in CMM maturity is measured in levels, and the first priority will be to attack those issues which are needed to get the organization up to the next level, not capabilities needed at a level way beyond this. However, the above mentioned efforts have not tried to establish any overall definition of what software process quality is, just a long list of required practices where the understanding of quality is at best implicit. This paper is trying to take a more overall view on process quality as such, without connecting this to specific practices in the process. As such it is somewhat similar to the effort of Lindland et al. (Lindland, 1994; Krogstie, 1995) of defining the notion of quality for conceptual models, where the semiotic ladder has been applied to define, e.g., syntactic, semantic, pragmatic, and social quality. However, an important difference is that the main purpose of this paper is not to define quality for process models, but to look at quality for the process as such, which makes a slightly different angle. Nevertheless, the application of the semiotic ladder to the discussion of process quality might be an interesting exercise, but space limitations prevents its discussion here - the main focus being the application of product quality criteria to processes.
4. Product Quality Factors Applied on Processes Software products and processes may appear to be completely different concepts at first sight as far as quality is concerned. But on second thought, they have quite a few similarities: • Quality is related to value in terms of achieving goals. From a business strategic
point of view, the software product is one of the means available for the customer organization to reach its goals. Similarly, the software process is one of the means available for the development organization to reach its goals. • A process may be prescriptive or descriptive. This can also be the case for software products: They may require the customer to perform his work in a different way from what he used to, or they may conform strictly to the existing work procedures. • A software program is an implementation of an algorithmic solution to a problem, often in terms of functional decomposition. The same applies for a process as well. • In the eyes of the customer, the software product is the executing program, which is a dynamic entity. But the static source code itself may also be considered the product, and is in fact often used in order to determine quality. Again, the difference between an executing process and a process model express the same duality.
5
Thus we see that products and processes are similar in several ways, concerning purpose, nature and representation (Osterweil, 1987), although this should not lead to over-optimistic views on the possibility to automate software development (Lehman, 1987). There are differences between products and software processes as well. One example is that the source code and the executing program are conformous in the sense that the execution takes place according to clear rules. This conformance is not present for processes; the “execution” of a process by human beings can at any time be abandoned, temporarily or permanently. In fact a process often has no process model, which is the counterpart to the software product´s source code. Despite these differences we consider the similarities to indicate that the product quality criteria identified in Section 2. should be further investigated for the purpose of assessing software processes. 4.1. Applicability of product quality factors for processes The 15 quality factors suggested by Deutsch and Willis can be given the following interpretation when applied to processes: • Correctness means to what extent the process model conforms to the specified
•
• • •
•
• • •
• •
•
requirements to the process. The requirements to the process can be e.g. time limits, effort constraints, development techniques to be used etc. Efficiency is whether the resources needed to execute the required process are affordable and available. Resources will here mean people (with certain skills), organizations, tools and other facilities (office space etc.). Expandability tells how easy it is to make enhancements to the process. An enhancement here means perfective changes, e.g. adding another testing technique. Flexibility deals with the adaptive changes, i.e. making the process work in different environments. This can be e.g. replacing the people working in the process. Integrity tells whether the process is sufficiently protected from access by unauthorized people. This includes access to the process model and process execution data as well as the products. Interoperability indicates how easy it is to couple the process to other processes, e.g. whether a specific process for software testing will work with another process for implementation. Maintainability tells how easy it is to find and fix errors. Manageability is concerned with how easy it is to manage changes to the process. Portability will tell how dependant the process is on the organization executing the process. High degree of portability means that the process can be executed by different organizations with minor changes. Reliability deals with the rate of failures, a failure can be e.g. incorrcect output (product), too long response time etc. Reusability tells to what extent fragments of the process can be reused in other processes. This usually requires the process fragments to be general, to comply with standards etc. Safety means absence of unsafe conditions, where an unsafe condition can lead to a hazard. This can be e.g. abuse of developers in a dissatisfactory work environment.
6
• Survivability tells how the process will work if a part of the process is affected by
a failure. Will e.g. integration test be performable if the implementation process fails to deliver some modules as planned? • Verifiability deals with ease of verifying correct execution of the process. • Usability tells something about the effort required to learn and use the process. If we try mapping these 15 factors to the views presented previously, we see that the end-users and customers will basically be concerned about efficiency, integrity and reliability. To some extent, correctness, interoperability and verifiability will also be of interest. Managers will of course be likely to find manageability important. Otherwise, the factors concerning change (expandability, flexibility, maintainability, portability and reusability) are important. Efficiency is also essential. Finally, the developer will probably be interested in the two that involves him directly: safety and usability. 4.2. Problems and required modifications We can identify some problems and aspects that are not sufficiently covered by the factors listed above: • There are several different factors that deal with aspects of change. While all the
•
•
•
•
factors may be given an interpretation for a process, it is not clear whether the factors will be useful. Productivity, cost, effort and time: All these four concepts are related, and considered important in particular by the managers and the customers. Efficiency can be extended to comprise time if we consider time a resource. Quality of the product is also missing in the model above. In the same way as a process model often has relations to a product model, a process quality model should be related to a product quality model. Correctness would be more meningful as a quality factor for a process model. For a process, conformance seems of more interest. With conformance, we mean to what extent the process complies with the process model. The relation between a developer and a process is basically the same as the relation between an end-user and a product. And in the same way as traditional product quality models fail at reflecting the user´s view of quality, the developer´s major concerns are not covered by the factors listed above. Usability may cover some of the aspects, but there is no notion of user satisfaction, motivation, pressure or work environment.
We consider the last problem to be hardest and also most important here. Since the developers are the primary sources of cost as well as quality of a software product, their view of process quality should be far better reflected in a process quality model.
7
5. Conclusions and Further Work Models are important in software development. Our opinion is that we need three basic models, for processes, products and resources. A model should be related to the two other models in order to express e.g. that a process produces a certain kind of product. The relations could also be used to express constraints, e.g. that the number of people available restricts the extent to which the work can be done in paralell. In order to model quality, we see a set of quality models, like a shadow of the process, resource and product models as feasible. These quality models should should exhibit the same nature as the entity models: Process quality is related to product quality, and resource quality may constrain the product quality. We have found most of the factors usually present in product quality models to be applicable also for process quality models, but more work need to be done in order to accomodate for productivity, cost, effort, time and conformance. However, by far the most challenging task is to cover the developer´s view of process quality, i.e.factors such as satisfaction and motivation.
6. References Bamford, R. C. and Deibler II, W. J. 1993. Comparing, contrasting ISO 9000 and the SEI capability maturity model, IEEE Computer, 26 (10): 68-70. Bell Canada 1992. TRILLIUM - Telecom Software Product Development Capability Assessment Model, Draft 2.1, Bell Canada. Boehm, B. et.al. 1978. Characteristics of Software Quality, North-Holland, New York. Bollinger, T., and McGowan, C. 1991. A critical look at software capability evaluations”, IEEE Software, 8 (4): 25-41. Brooks jr., F. P. 1986. No silver bullet: essence and accidents of software engineering”, IEEE Computer, 20 (4): 10-19. Conradi, R., Høydalsvik, G. M., Sindre, G. 1994. A comparison of modelling frameworks for software processes and information systems, Proc. 3rd European Workshop on Software Process Technology (EWSPT’94), Springer-Verlag (LNCS772). Crosby, P. B. 1979. Quality is Free, McGraw-Hill. DeMarco, T., Lister, T. 1987. Peopleware: Productive Projects and Teams, Dorseth House. Deutsch, M., Willis, R. 1988. Software Quality Engineering - A Total Technical and Management Approach, Prentice Hall. Dorling, A. 1993. SPICE: Software Process Improvement and Capability dEtermination, Software Quality Journal, 2: 209-224. Fenton, N. E. 1991. Software Metrics: A Rigorous Approach, Chapman & Hall, London. Gillies, A. 1992. Software Quality - Theory and management, Chapman & Hall Computing.
8
Humphrey, W. S., and Sweet, W. L. 1987. A Method for Assessing the Software Engineering Capability of Contractors, tech.rep. CMU/SEI-87-TR-23, Software Engineering Institute, Carnegie Mellon University, Pittsburgh PA. Humphrey, W. S. 1989. Managing the Software Process, Addison Wesley. IEEE Computer Society. 1990. Standard for a Software Quality Metrics Methodology, IEEE draft standard P-1061/D21, April 1990. ISO. 1987. ISO 9001, Quality Systems - Model for Quality Assurance in Design/ Development, Production, Installation, and Servicing, International Organization for Standardization, Geneva. ISO. 1991. ISO 9000-3, Guidelines for the Application of ISO 9001 to the Development, Supply, and Maintenance of Software, International Organization for Standardization, Geneva. Juran, J. M. 1974. Quality Control Handbook, McGraw-Hill, New York. Kitchenham, B. 1989. Software Metrics, in Software Reliability Handbook, Elsevier. Krogstie, J., Lindland, O. I., and Sindre, G. 1995. Defining quality aspects for conceptual models, Information System Concepts: Proc. ISCO3, Marburg, Germany, March 1995. Chapman & Hall, London. Lehman, M. M. 1987. Process Models, Process Programming, Programming Support, Proc. ICSE’9, IEEE/ACM. Lindland, O.I, Sindre, G., and Sølvberg, A. 1994. Understanding Quality in Conceptual Modeling, IEEE Software, 11 (2): 42-49. McCall, J. A. 1977. Concepts and definitions of software quality, Factors in Software Quality, NTIS vol 1 1977. Osterweil, L. 1987. Software Processes are Software too, Proc. ICSE’9, IEEE/ACM. Paulk, M. C, Curtis, B., Chrissis, M. B., and Weber, C. V. 1993. Capability Maturity Model for Software, version 1.1, tech.rep. CMU/SEI-93-TR-24, Software Engineering Institute, Carnegie Mellon University, Pittsburgh PA. Pirsig, R. M. Zen and the Art of Motorcycle Maintenance,Vintage, New York, 1974. van Swede, V., and van Vliet, H. 1994. Consistent Development - results of a first empirical study on the relation between project scenario and success, Advanced Information Systems Engineering, Proc. CAiSE’94, Springer-Verlag (LNCS 811), Heidelberg. Yeh, H. T. 1993. Software Process Quality, McGraw-Hill.
9
