User-perceptions Of Embedded Software Quality R.J. Kusters (
[email protected]), R. van Solingen (
[email protected]), J.J.M. Trienekens (
[email protected]), Eindhoven University of Technology, Department of Technology Management, P.O. Box 513, 5600 MB Eindhoven, The Netherlands This paper has been published in the Proceedings of the STEP97 Conference, IEEE Computer Society Press, ISBN 0-8186-7840-2, July 1997 Abstract Many researchers and practitioners have recognised that the perception of ‘quality’ is largely influenced by personal view and application context. Depending on personal goals, interests and background, the interpretation of the quality concept is different per individual. In this paper, an approach is suggested by which the different user perceptions on quality are modelled, and can be addressed. This Multi-party Chain model supports the explanation of the various views on product quality of all users. Discussion among the parties involved is therefore enabled. Based on consensus the appropriate measures can be selected. The model should operationalise the basic relation between process and product, it enables the definition of metrics to evaluate process improvements, and links engineering activities to user requirements. The usefulness of the model is validated in a case-study.
1. Introduction An information system, especially a complex embedded system, does not end up being ‘high quality’ just by accident. Often much effort and resources are spend to achieve quality. The developers, often of several disciplines, each contribute in the development of a high quality product. In their design and implementation, quality is a self-evident ambition which forms the basis of good engineering practices. As such, quality can be seen as the result of a large number of explicit and implicit control activities, and measures, often deeply implanted in the professional engineering knowledge and actions of the experts involved. In practice however, quality is not always easy to achieve. Two common problems will be mentioned. Firstly, building functionality within plan and budget is often given higher priority than achieving quality attributes such as for instance reliability. Secondly, quality of the complex systems of today must be
achieved by combining the experience and knowledge of the disciplines involved. Combining disciplines means co-operation of experts from different fields, probably causing communication difficulties. Human failures become more likely and can easily lead to lower quality (e.g. less reliable or even unsafe systems). In this paper we present an approach which puts user expectations on product quality as being the ultimate requirement to be fulfilled. An information system should either fulfil all requirements as desired by all parties involved or it should be made clear that some of these desires cannot be fulfilled within the existing constraints. This statement entails at least two types of problem: 1. Requirements are not always made explicit by the people involved, but they do expect the system to adhere to them none the less. 2. We are most of the time dealing with many different involved parties. Differences between users occur related to for example a persons job, responsibility or role and their specific use of the system. Even people that will not “use” software like the manager of the users or the person who is responsible for buying the software are involved. Also individual preferences exist based on intellectual skills, background or interests. The result of these differences is that ‘quality’ is being defined differently among the different users. As a result we have to recognise that quality is a multi-dimensional concept, that it means many things for many people. This concept is explored further in this paper. First we will take a look at the existing definitions of quality. In section 3 we will develop the multidimensional concept of quality. Based on this, in section 4 a control model for defining and implementing quality is presented. This model is developed further in section 5 where the multi-party chain approach is presented. This approach supports tackling the problem of identifying and controlling product quality. It is based on modelling the organisational chain for a product life-cycle and relating the different parties, roles and requirements to
explicit actions to be taken during development. Our approach explicitly defines quality requirements for all users, and facilitates communication of those quality requirements among all parties involved. Therefore it seems highly applicable for application in multidisciplinary development teams. A data model to support this model is also presented in section 5. A number of ways to use the model is described in section 6. In section 7, by means of a case study, the validity and the usefulness of the approach is illustrated. The paper ends with some conclusions.
2. Present approaches and problems. The distinction between functional and quality (or nonfunctional) requirements is a common one in the IS-field (see e.g. Boehm et.al. 1978 [2]; Cavano and McCall 1978 [3]; Deutsch and Willis 1978 [5], Bemelmans 1991 [1]; Delen and Rijsenbrij 1992 [4]). Examples of these quality requirements are quality, user friendliness, maintainability and portability of an information system (Boehm et al, 1978 [2]). Most of the literature in this area is based on the work of Boehm et. al. (1978) [2] and of Cavano and McCall (1978) [3]. In their view quality is expressed in a number of quality attributes of a software product. These attributes can be derived from a set of aggregated quality factors. The next step is to define metrics which can be used to test if the final product adheres to these attributes. Over the years this approach has been further refined by several authors (Willmer 1985 [12]; Deutsch and Willis 1987 [5]; Keller, Kahn and Panara 1990 [10]), and in particular the relation between the attributes have been described in a more realistic manner. The current state of the art can be found in two standards: ISO 9126 (ISO, 1991 [8]) and the IEEE software engineering standards collection (IEEE, 1994 [7]). Both definitions are made for engineers by engineers and provide specifications in engineering terms that can be used as part of a design contract. In doing so, they both provide a vital service. Developing information systems, especially complex embedded systems, will usually take place on the basis of a set of well worked out requirements. However, the very nature of these solid technical definitions also provided a problem. The greatest objection to this type of definition is that principals and potential users are not supported in the formulation of their quality requirements (see e.g. (Kaposi and Kitchenham, 1987 [9]). It is not clear how an organisation can determine the level and type of quality required that is relevant in its specific situation and in which language this can be formulated (Trienekens, 1992 [11]). This implies the need for a wider look at quality than is provided by either ISO or IEEE.
3. Quality: a wider outlook Different parties have different ways of looking at a product from the point of view of quality. It might theoretically be feasible to translate all these different ‘visions on quality’ into one common definition which can then be used to communicate between all those parties involved. This seems the objective of the above discussed standards. In practice however, we deem this to be not feasible. Even if this common terminology could be developed, it is unlikely that it would provide a sufficient basis for discussion with all the parties involved. For one it is unlikely that all parties involved will be willing to adopt this technical ‘engineering’ language. We will therefore take an approach in which explicit account is being taken of locally defined assessments of quality. In this paper we assume that product quality is related to its use. This brings us to the conclusion that to create a reliable product one must first define the function it is intended for, and collect all local definitions of its required quality level. This required quality level can now be derived from the characteristics of the situation within which the system has to function (Heemstra et.al., 1995 [6]). Another approach is to assume that product use is related to the responsibility of the users involved. Different parties, i.e. stake-holders such as buyers, users, project principals, developers etc., have different requirements regarding product quality. Quality appears to be a multidimensional concept, both in theory and in practice. As a consequence different concepts are used to express quality, different measures are carried out to determine and/or to establish quality, and different standards are used to validate and to evaluate the level of quality of a product. This multi-dimensional concept of quality, as described above, is not intended to displace the more traditional quality concepts of ISO and IEEE. These will still be needed to provide accurate specifications for development engineers and will form the basis of any discussion on product quality. However we do claim that a wider look at quality is needed, if one wants to be able to define the type and level of quality required in such a way that users both understand and agree to what is being proposed. We will take this premise as a starting point when designing a control model for implementing quality.
4. A control model for implementing quality
For product development various activities are needed. A typical breakdown of activities for the development of embedded software systems is: • initiation • specification • design • implementation The development activities produce the system and thereby will determine the system characteristics, including its quality. Further phases in the life cycle, such as operation, maintenance and disposal will also change the system quality in time. For simplicity, we will not discuss this and focus on the development phase of the life cycle, since those phases are assumed to have a higher impact. Within the development process, several activities can be noted that specifically contribute to the quality of the system. If we make this activities explicit they can be seen as control measures on all activities within the development process. We can depict this in a control model (figure 1).
Figure 1: quality control model
In this model the flows of information and corresponding processes are: • desired quality is the quality as stated or intended by the different users, the law or by ethics. It might also be influenced by taking into account results from a quality analysis. • control measures are all required activities that should take place in the development process in order that the actual quality will equal the desired
quality. These control measures are selected on the basis of a quality analysis. • actual quality is the quality of the end product or the intermediate results of the development process, as perceived by the actual user of that product. The goal of any development project is to create a system that conforms to a pre-specified level of quality. Depending on both the type of quality required as well as the level required a certain amount of measures are selected to result in the required product quality. In order to select the right mix of measures from the numerous possible options, and to achieve a closed control loop we need to be able to assess the result obtained (actual quality) to be able to determine if the proper mix of measures was selected in order to satisfy the required quality, which has to be stated explicitly. More in detail: 1. Assessment of actual quality. Product use is the ultimate quality check, but during development specific data collection appears to be applicable to give a representation of product quality. Keeping in mind that control measures will be taken, it seems logical to include data collection activities to track
perceived quality requirements for their specific context of use. Whenever this description is available measures can be selected with a clear goal, and evaluation (read: measurement) can be implemented. Such a methodology is not found in literature or practice, but will be described in the next chapter: the Multi-Party Chain model.
5. Multi-party chain model Measures are considered to be activities/actions that aim to increase the quality of an embedded software product. Both measures as well as the effect they will have will tend to be expressed in an engineers’ language. Given this, communication between stake-holders is essential in order to achieve mutual understanding and consensus between the different parties involved which has to lead ultimately to agreed upon operational measures and metrics to realise and quantify the quality of products. To enable this communication the different interpretations of quality have to be understood. Given a
Figure 2: Data model of the development process. whether those measures give their intended effect. 2. Selection of control measures during the development processes. To select the appropriate measures to fulfil quality requirements, a supporting method is required. 3. Make the desired quality explicit while taking into account the multi-dimensional nature of the concept of quality as was discussed in the previous section. The one thing to be kept in mind is that whenever a product must be reliable, measures are required to result in the required level of quality. Selection of measures is a difficult topic, and is done in practice mostly implicit and/or based on experience. What is required in both practice and science is a technique to describe the user-
specific product, if it is known which party specified a specific requirement for this product and especially for which reason, then a basis for discussion has been created. If this specification is available it becomes possible to translate different interpretations and it becomes possible to negotiate and discuss those specified requirements. In the end, this enables us to link specific measures (in engineering terminology) to requirements of the different parties involved as expressed in their local terminology. In this chapter we will present a model to facilitate the identification of all parties involved and of their concerns with quality. This model, the ‘multi-party chain’ model (MPC), can be used in a specific situation to
identify concerns and to facilitate communication among the parties involved. In a data model of this multi-party chain (MPC) the main entity types and their interrelations are represented. The data model supports the identification of the different parties involved, their interrelations, their roles, their requirements etc. The focus of the data model is on the determination, the realisation and the evaluation of quality in embedded software products by taking the right measures. By taking a specific measure, one aims at increasing one (or more) aspect of quality. The key question is therefor which set of measures suit best to the set of requirements for a specific product.
have defined that a development consists of a unique set of actions taken in a specific order. Those actions (measures) are taken on the product, the process, the resources and on the project management. Evaluating the actual quality of a product will consist of data collection on all of those entities. The main entity types in the Figure 2 data-model and their definitions are: • Measure: Activity that influences the type and/or degree of quality that is attained in a given product • Project management: Those activities that guide the execution of the project • Resources: Those resources that are used during
Figure 3: Data model of selection mechanism to fulfil quality requirements. The data model that represents this approach will be shown in Figure 5. We will not impose this model upon the reader directly, but we will transfer the control model of the previous section stepwise according to the points 1-3, into the MPC-model.
5.1. Assess the actual quality of the output from the development process The first step to take is to evaluate (e.g. by means of the ISO standard (external) metrics (ISO, 1991) [8]) the actual quality of the product. Therefor we need to perform data collection on that product. Theory on software metrics has learned that metrics on the total development process are required in order to gain insight into actual quality of a product. Figure 2 shows the data model of the development process. For simplicity we
• •
project development and use Product: An imbedded software product seen as part of a larger system Process: The process of developing, maintaining and managing the product
5.2Select control measures to fulfil quality requirements The second step to take is the definition of control measures that aim at fulfilling specific quality requirements. Those requirements can be very diverse like: all software faults in the user-interface have been removed, or the most critical part of the system is fault tolerant. A certain quality analysis has to be executed on which measures can be selected. Figure 3 shows the data model in which control measures to fulfil quality requirements are opposed upon the development process.
Note that the data-model of Figure 2 is included in this figure. The new entity type in this data-model and its definition is: • Requirement: A statement regarding the type and degree of quality that is required from the product as seen from the point of view of a party in a given role Requirements have to be expressed in terms of a specification. Examples of quality requirements of users: 'the system may not fail (go down) more than two times a month', 'when the system fails (goes down) an engineer must be able to fix it within one hour', or ‘I will not have my customers annoyed by this system’. The first requirement can be expressed (in ‘engineering language’) into the quality aspect "availability". The second
‘representation’ relationship. Take for instance a system for controlling a fuel pump. Some relevant parties are customers, fuel station managers, oil company managers, oil company purchasing officials, and vendor representatives. We see here a chain of representation. The fuel station manager represents the interests of the customer. The oil company represents the interests of both the customer and the fuel station manager. The oil company purchasing official represents al of those. Also, The oil company purchasing official influences (and is influenced in his turn by) the vendor representative. To chart this area a ‘chain’ approach is taken. The main new entity types in this part of the MPC
Figure 4: Data model to make quality requirements explicit from the context of use. requirement can be expressed into the quality aspect "recoverability". The last requirement will need further discussion and clarification in order to find out what is really required, so that this requirement can be translated into requirements in the 'engineering language'. Consequently, measures and metrics (pertaining to project management, process, resource, and/or product) will have to be determined.
5.3. Make the desired quality explicit The final step to take is to make the desired quality of the system explicit. As described before we will apply an approach in which all local definitions of all users are captured. Therefore a modelling technique is required which models all users and their view on quality. As described before we focus on parties, underlying roles and related quality requirements. Figure 4 shows the data-model by which we describe the chain of parties involved. In this way it becomes possible to make quality requirements explicit and it is possible to trace where they are originating from, based on roles and parties. This is a ‘chain’ model, because we assume that a number of customer-supplier relationships exist between the different parties. An important relationship is the
data-model and their definitions are: • Party: An identifiable person, group of persons of organisation who has a legitimate interest in the degree of quality of the product • Role: An area of responsibility of the party determining a view on the type and degree of quality required A ‘party’ was previously defined as an identifiable person, group of persons of an organisation who have a legitimate interest in the degree of quality of the product. This ‘legitimate interest’ may come to light in one or more phases of the product life cycle. Examples of parties are: principals, customers and users (of different types, e.g. employees from marketing, purchasing, production etc.), project managers, engineers, maintenance people. Parties can also be stake-holders that are not directly involved in one or more specific phase of the life cycle, for instance governmental parties that develop and control policies, rules and constraints for quality of branch organisations that set their specific requirements. A ‘role’ was defined as an area of responsibility of a party that determines which view exists on the type and degree of quality required. A party can have more than one role. Example from Schlumberger, a producer of petrol dispensing machinery: The Schlumberger OPCO
(the national sales and service organisation) responsibility to represent Schlumberger customers and vice versa (customers Schlumberger), but also to represent the
has the towards towards national
• Some quality requirements can be ‘translated’ into another definition of quality. This enables communication and agreement between different parties.
Figure 5: Data model of the Multi-Party Chain. government of the country it is servicing and their policies towards Schlumberger. So the OPCO has several roles to play in communication on quality. Relationship types relevant to the chain approach are: • Parties may have different responsibilities. This is represented by the different ‘roles’ that a party may assume. This relationship type allows an inventory of quality requirements which are relevant to this party through the intermediate entity type ‘role’. • Parties can represent, both officially and unofficially, other parties. • Finally parties may control and/or influence other parties. The last two relationship types allow us to build the multi-party chain by identifying the links between the different parties. • Roles may be linked to several parties which in that case would be expected to have identical quality requirements. • A role may be concerned about one or more quality requirements. • A quality requirement can be linked to several roles. In that case these roles, from a different motivating background, are expected to agree as to the required degree/type of quality.
• Requirements can be influenced by measures/actions.
5.4. The Multi-Party Chain Data Model If we combine the steps taken before into the overall Multi-Party Chain model we get the data model as represented in figure 5.
6. Using the mpc model The MPC model can serve different objectives. The model can be used initially to fill in the entities and attributes for a specific embedded software product in a specific organisation, i.e. for positioning the different parties and the different types of requirements that can be distinguished. But it can also be applied for identifying the different types of measures that are executed, for investigating differences in opinions etc. Based on such a 'case-study' the MPC data model can be used for answering questions such as: • how to identify gaps between quality interpretation, and how to bridge them (e.g. between 'fitness for use'
•
• •
interpretations and 'conformance to spec' interpretations; this to enable communication in the area of quality how to distinguish measures, metrics, standards (e.g. ISO9001, usage of test scripts, the gathering of measurement data, informal reviews of developers), and how to store and supply this information in a user-friendly way how to identify and express generic descriptions of quality measures and specific descriptions of measures and how to make distinctions how to make distinctions between 'internal' parties (e.g. developers) and 'external' parties (e.g. customers).
7. Case study: the Multi Party Chain applied at Retail Petroleum Systems practice The Multi Party Chain approach has been applied to make a model of the Schlumberger RPS situation regarding the way that different specialists deal with different interpretations of software quality. First some information is given about the Schlumberger RPS organisation and its products. Subsequently, selected results from interviews with different types parties within Schlumberger are presented to support the basic ideas behind the mpc-model. The second part of the case study shows how it is possible to use the mpc-model in analysing user quality requirements. The case study ends with a number of comments.
7.1. The organisation and its environment Schlumberger is a world-wide leading supplier of services and technology to the international petroleum industry. Schlumberger concentrates its expertise in three groups: Oil field services (exploitation and production services for the petroleum industry), Measurement & Systems (technologies for optimising the distribution of gas, water and electricity), and Omnes (communication and information technology solutions). The interviews have been carried out at Retail Petroleum Systems (RPS) which is a part of the Measurement and Systems group. RPS serves the major service station networks and offers the oil companies and independent retailers, products for dispensing fuel and transferring funds. The customers of Schlumberger RPS include most of the large oil companies. The software products of Schlumberger RPS are typically real-time embedded systems. RPS develops and manufactures systems for self-service fuel stations. Examples of these products are fuel dispensers, forecourt control systems, point of sales, and credit card systems,
training and maintenance programs and 24-hour customer service. RPS is divided in six departments, Quality & Marketing, Research & Development, Sales & Service, Manufacturing, Finance and Personnel. Manufacturing is carried out in three factories, of which Schlumberger RPS Bladel The Netherlands is one. Schlumberger RPS is represented in a large number of countries by so-called OPCO’s (OPerational COmpanies) which serve national markets in terms of product sales, installation, training, and service. To better understand the meaning and the importance of software quality in Schlumberger RPS products, hereafter some relevant information about the products, the customers and the organisational policy is given. The Products All products of Schlumberger RPS contain software. The software is often incorporated in the hardware (in EPROM chips) or is implemented on a hard disk. Products can be divided in three categories: • Dispensing systems (Fuel-dispensers) • Payment Terminals (mostly for out-door purposes) • Cash register devices (for in-door purposes) All systems consist of both hardware and software. Some of these products are called “complete” products because the OPCO’s do not (have to) carry out customisation activities on the software in these systems. For Point of Sales terminals the core software is released separately, and customised by engineers at the OPCO to meet specific market and user requirements, e.g. regarding customer names, logo, currencies, etc. For these particular systems the difference between software and hardware is very relevant. The product consists of a core system, to which additional functionality such as card applications, back-office applications, supply management, can be attached. Some of these systems have sophisticated man-machine interfaces, for example the Point of Sales terminals, but currently also the payment terminals are increasingly provided with such interfaces. The Clients A distinction can be made between two types of clients: • users who really work with a system (e.g. a manager or operator (e.g. cashier) at a fuel-station). • customers who decide to buy a system, e.g. a representative of an oil company. A user does not distinguish between hardware and software. Whenever the product does not fulfil its promised function or does not have the quality that has been promised , it is not relevant for a user to know
whether it is hardware or software that causes the problem. Some customers do distinguish between hardware or software regarding a product. The number of customers who prescribe acceptance criteria and required test approaches is likely to expand. OPCO’s are increasingly specifying those criteria explicitly for the development department. Mostly products are sold to the major oil companies. A percentage of the customers are independent customers which own and exploit a fuel station. The major oil companies are important customers for which the first releases of a product is intensively supported. The Policy Sales at Schlumberger RPS is targeted on high quality products. The importance of software for the fuel station systems has increased during the last years. For example customers ask more and more for systems that operate in local- and wide area networks. Risks of software, and
fuel station systems in general, are in particular economic risks. Risks are that transactions are lost, or nonoperation of a system causes the station not to sell petrol. Schlumberger RPS invests large efforts in reducing those risks to an absolute minimum.
7.2. Interpretations of quality A first objective of the case-study was to determine the way that practitioners in Schlumberger RPS deal with the determination and implementation of the quality of software in their products. The concept of quality has been approached from a user or customer perspective. Interviews with different types of practitioners have been carried out to test the assumption that the different involved parties have different roles and tasks (to meet customer requirements), approach quality in different ways, and consequently carry out different activities to strive at an optimal level of quality.
The different parties, their roles and their particular ways of dealing with (quality) requirements have been identified. Also the interrelations between these parties, in terms of co-operation and translating system and quality requirements from one party to another, are investigated. Consequently a description of the Schlumberger environment is developed that consists of parties and their interrelations. The parties in the Schlumberger RPS cyclic chain are respectively: Client, User, Fuel-station, Oil-company, OPCO, Production, Development, Marketing (Figure 6). In the model both the flow of the demands and the flow of the delivery of the products are visualised. Each step can be considered as a consumer/producer relation. For example an OPCO is often both consumer of
that each party dealt with software quality. The interviews have been carried out in respectively the departments Marketing, Research & Development in The Netherlands and in two OPCO’s in Turnhout Belgium and Ridderkerk The Netherlands. • In the central Development department a project manager has been interviewed who is a technical coordinator for the release of a fore-court control and cash register product. His job contains of training, demo's and presentations, evaluating requirements, and co-ordinating test phases. He has relevant experience in software development. • In the Marketing department the interview was carried out with a marketing manager who is responsible for the release of a fore-court control
. Figure 6: the Schlumberger RPS environment products from Development and Production, and producer of customer-specific products for fuel-stations. In Schlumberger RPS interviews were performed to discuss the interrelations between the different parties and their roles in the chain model, in particular the ways
and cash register product to a large oil company. At OPCO two interviews were carried out. • One interview with an OPCO manager and an operator trainer who has relevant experience in how people use the products. This interview focused on
embedded software products such as Point of Sales terminals, since this is the software that is customised by OPCO departments. • In a second OPCO interview a Sales manager and a support engineer were interviewed. The Sales manager has sound experience, and frequent contacts with Schlumberger RPS’ customers. The Training and Support engineer has frequent contacts with end-users. The fore-court and cash control systems played a central role in each of the interviews. In the following the main interview results are presented in three system development stages that were identified as most important, respectively: • requirements elicitation (c.q. identification and definition), • establishing system/software requirements and • software testing. The quality chain model is used to position the Schlumberger practitioners and their particular opinions and activities in a coherent framework. As such the differences of their view points on quality could be presented, discussed and clarified. Requirements elicitation Marketing is the central entity in the multi party quality chain model (see Figure 6). Marketing supports the process of specifying the system requirements of the “consumers”. In the requirements identification and definition processes Marketing considers ‘all’ requirements from ‘all’ consumers. All users, operators, clients, customers, OPCO’s and developers are being considered as consumers. As a consequence Marketing has to deal with a large variety of different (quality) requirements. Also the complexity of these requirements continuous to grow. Marketing considers its role as a highly interactive iterative and concurrent process of: • investigating new product opportunities and (im)possibilities of developers and producers; • identifying the implicit and explicit customer (e.g. oil-companies) requirements and wishes; • identifying the implicit and explicit user requirements and wishes; • taking decisions about a product’s functionality and quality, how to present a product to the customers and when to the market, and how to describe product requirements for the developers. Marketing specialists are confronted with more and an increasing complexity of the requirements of consumers. Two trends were identified: • customers, in particular the large oil companies, start prescribing acceptance criteria and required test approaches. This trend is likely to expand. Although
formal international standards for development processes are applied, such as ISO9000, product quality standards such as ISO9126 are not applied. • Customers, require more and more information from the fuel-station. This trend demands for systems that operate in large networks, and that provide information to manage several fuel-stations centrally. The role and the position of the fuel station manager is therefore changing. Establishing system/software requirements This section is in particular based on selected results from the interviews of respectively the project manager from the central Development department and the OPCO employees (see Figure 6). Often, the wishes and needs of the customers and users do not reflect a distinction between hardware and software. Customers want to specify a system and to buy a system (functionality) for a certain price. The translation from system requirements to hardware and software is then made by the developers. One of the main issues that has been addressed was the translation of quality requirements into measurable quality characteristics of a product. Although requirements for quality of systems are not explicitly specified, quality characteristics (e.g. for the Quality of a system), including metrics, are becoming important and are specified. Examples of metrics are “Mean Time Between Failure” (MTBF) or “Mean Time To Repair” (MTTR). The quality characteristics have to be checked by an OPCO, during in-house testing. Sometimes a quality characteristic such as Quality can also be covered by an agreement between developers and customers about the application of specific test sets and test criteria. It is difficult to develop systems for all European countries that meet all requirements. The reason is that each country has different demands and government regulations. Therefore Schlumberger RPS develops socalled core-systems that have a specific set of basic functionalities. OPCO’s customise these systems in accordance with the particular wishes and requirements of a specific country (or customer). On the one hand, this creates time saving, because the core-system is developed only once. On the other hand, the time-tomarket is lengthened, because products have to be customised, which takes extra time and effort. Software Testing This section addresses the experiences of Schlumberger professionals with testing. Testing is considered as an important area in that quality has to be evaluated and consolidated. The selected interview results that will be presented hereafter have been gained from interviews with the project manager from the
central Development department and the OPCO employees (see Figure 6). The risks of developing systems with more functionality creates higher risks. As stated before, risks of fuel station systems are economic risks. Developers strive at decreasing the risks involved in using software in the systems by using extended testing and debug facilities, 4th generation languages, libraries of software components. Software development itself is getting more mature also because of the use of quality control systems. Risk reduction has an important place in such a quality system. Risk reduction at Schlumberger RPS is mainly based on thoroughly testing of the software-intensive systems and on intensive co-operation with customers during development and testing. It is recognised that testing should be based on a welldefined test strategy and plan. Based on that the targets for testing can be set and the time and effort planning for the testing can be carried out. Systems should be tested very intensively. For adequate testing it is important to know which the most critical components are of a specific system and what the failures are that have occurred with similar systems before. Only then testing will improve the quality of a product. In the interviews two main issues of software quality have been identified: • failures and causes of failures • testing from a customer perspective To be able to carry out testing in an adequate way, i.e. increasing the quality of the system, practitioners should know about possible failures that (can) occur as well as their causes. There is currently no real consensus among the practitioners about the most important causes of system failures. This underlines the assumption that individual viewpoints influence the perception of product quality. One interviewed person told that most failures are caused by hardware defects. Another one told that most failures of the system are caused by software faults. In his opinion it could be expected that after some time when the stability of the software has increased most defects will be caused by the hardware. If possible, customers should be involved in the testing activities. Actually, they have to be convinced about the quality of a product. At Schlumberger RPS customers respond to the increase of the importance of software in products. Sometimes also a customer is asked by Schlumberger specialists to approve temporary deliverables, and they are kept informed on current status and progress. To reduce the number of fatal faults in the system before it is released to a large number of fuel stations, it is normal to keep field-tests. Field-tests are trial installations of a system in practice, but in a very small number (1 or 2). During such a first field-test, a
developer (engineer) will stay at the fuel-station. There will also be 24 hours support for that station, during the first week of operation. In this way faults in the system, that were not found during test, are detected before the system is installed at a large number of stations.
7.3. Using the MPC model In the second part of this case study we will demonstrate how the MPC-model can be applied. For the sake of clarity we will confine the further discussion to a single aspect of quality, namely reliability. Also, again for the sake of clarity and brevity, we will only look at external (from the point of view of Schlumberger) parties. We continue the case within the following setting: A specific customer of Schlumberger is demanding that each new product is first installed in a pilot installation where it has to be operational for a period of time before the final role-out can proceed. Those pilot installations are called: “Field-tests”, and the question is why this customer is demanding those tests, and whether an alternative solution can be offered that fulfils the same underlying requirements. A first step in using the model is by identifying all relevant parties (see Figure 6), their roles and the reliability requirements that may be derived from these roles. This was done by means of a series of interviews within the Schlumberger organisation. Interviews were conducted with: • an operator trainer for RPS products, with relevant experience in how people use RPS products, • an OPCO manager, who is responsible for all activities performed in a national representative company, • a marketing manager who is responsible for the release of a product to a large customer. • a technical co-ordinator involved in the release of the same product, • an experienced sales manager who has frequent contacts with Schlumberger RPS customers, • a customisation engineer who has frequent contacts with end-users. The results, an operational version of the MPC-model, are presented in tables 1 and 2. Table 1: Overview of external parties and roles Party Roles Government 1. Protect the environment 2. Guarantee safety 3. Collect taxes 4. Translating the law into
Party
Oil-company
Fuel-station
Customer
Roles certification procedures 5. Certifying 6. Buying systems that fulfil the oilcompany requirements, customer requirements, and fuel-station requirements 7. Deliver (fuel) products to fuelstations in time 8. Responsible for revenue of a fuel-station 9. Responsible for fast payment by customer 10. Responsible that exact amount of fuel is delivered 11. Paying
Using these tables we will now analyse the customers’ demand for a field test. The first step to be taken is to identify the roles that originate to the demand for a field-test. Result: The MPC-model shows that a representative of an oil company has two roles: • Buying systems that fulfil the oil-company requirements, customer requirements, and fuelstation requirements • Deliver (fuel) products to fuel-stations in time It is clear that the request for a field-test mainly addresses the first role of the representative of an oilcompany. The difficulty for that specific role is that it has the responsibility to address requirements of many other parties, therefor this person will demand a test (validation of requirements) that is able to check all of the implicit and explicit requirements: practice. This shows the reason behind the demand for a field-test. Based on this conclusion it is possible to offer the person with that specific role an alternative solution which fulfils his needs. For example showing that person the same product already operational for some years on an other fuel-station. Note: In real-life this request came for a product that was in it’s first release to the market, so no operational systems were available, which explains the emphasis of that role for a field-test. Table 2: Overview of (external) roles and related reliability requirements Role Reliability Requirements 1 Safety of environment Environment is not at risk, according to the law 2 Safety of people People are not at danger, according to the law 3 Security of payment transactions
Role
4
5 6
7 8
9
10
11
Reliability Requirements Payment transactions are secure, according to the law Operational inspections that show no unreliability Operational inspections that show reliability Conform certification procedures Availability Maturity No (known) failures Recoverability Support in case of failure All implicit reliability requirements are addressed Availability Availability Performance Support in case of failure Accuracy Usability Sufficient feedback Performance Availability Usability Accuracy Performance Security Performance Accuracy
But before drawing premature conclusions, it is better to focus on the underlying reliability requirements of the representative of an oil-company. In this example this means focusing on the first role. The underlying reliability requirements according to the MPC-model are [8]: • Availability • Maturity • No (known) failures • Support in case of failure • Recoverability • All implicit reliability requirements are addressed Again, it is shown that very diverse reliability requirements are included in this role, which declares why this person will ask for a field-test, since it seems difficult to address those different requirements together in one measure. The overall requirement of a product seems to be that it is continuously operational, which is being addressed by four of the above reliability requirements: availability, maturity, no failures, and recoverability. For a representative of an oil-company: it just has to work. It has to be ‘good’. One can think of an
alternative measures to address these issues, like labtests, formal proof, measurement results, or proven operation on a fuel-station in practice. Two other requirements are still left: support in case of failure, and fulfilling implicit requirements. From the role of the representative the support issue is clear: a fuel-station must be available, so whenever it fails fast support must be guaranteed. One can think of contracts that guarantee support with penalties when those are not kept, but testing this kind of support in field-test is not suitable, because support is mostly important in situations where there is no special attention for a fuelstation, and during a field-test there is special attention because it is a test-case. So it can be concluded that a field-test is not addressing this requirement. The second requirement left, is about implicit requirements which really show the need for testing in the field, in order to verify those implicit requirements. But, it does not imply that the product needs to be tested at a station of the customer. Involvement of end-users during design, or the intensive enhancement of the userinterface in an external usability-lab may be a sufficient measure to address this reliability requirement of the oilcompany representative. An alternative solution to field-testing might therefor be showing that the product is already operating in practice with success. This can be implemented by showing the successful operation of the system at the station of competitors of that oil-company. But the most likely alternative for a field-test is: testing a product on a fuel-station of Schlumberger. Customer specific enhancements of a product are then not validated, but in that case a lab-test of the additional functions may be sufficient to convince the representative that his responsibilities are being addressed. One remark that must be added, is that the requirements above may not always be totally stated by the oil-company representative himself. It is very likely that the demand for a field-test is done by someone else in the company who is translating those requirements. Examples are the marketing manager of an OPCO, who needs to prove that the requirements of an oil-company are met, or the manager of a fuel-station who does not want to take any risk. Anyway it is possible to show the reasons behind a measure (expected effects). Look for example at the roles and reliability requirements of a station-manager, or OPCO-marketing. One will see that the demands for a field-test originate mainly from validating all possible requirements on a fuel-station.
should take all the responsibility. The interviews made clear that there are important differences between the viewpoints on quality of different parties, i.e. types of practitioners, and their roles. From the Marketing point of view quality has to be determined starting from the large variety of different customers’ wishes and needs and their increasing complexity. Central Development and the OPCO’s have to translate these wishes and needs into (measurable) quality characteristics of a product. From the developers’ point of view, both in the central Development department and in the OPCO’s, actions have to be carried out to evaluate and to verify (i.e. to test) the quality of a product. Preferably, these activities should be a joint effort of Schlumberger specialists and customers. The interview results in the first part of the case show that the different points of view on quality of different parties can be visualised and clarified by means of a cyclic multi party quality chain model. This example presented in the second part of the case illustrated how the MPC-model can be used to analyse user requirements. As such it can provide important input into the negotiation process will result into ‘hard’ engineering requirements.
7.4. Case comments
1.
Quality is not a concept that can be dealt with by one particular person, or for that one specific specialist
8 Conclusions It appears that a general definition of quality to which all people: developers, users, managers, etc., agree is not available. It has been concluded that the interpretation and view on quality is influenced by the responsibility, or role of a person. Those different views of quality rightfully exist and cannot and should not be ignored. In this paper a multi-party chain model was presented. It can be used to chart the various views on product quality in such a way that discussion among the various parties involved is enabled. A number of different scenario's for dealing with quality can be distinguished by using the model. Also, the definition of quality issues for different roles can be supported. This will enable the definition of metrics for quality issues and eventually the formation of a link between engineering actions and customers requirements. Further research within the context of the Spirits project (see appendix) will be aimed at populating the data model with more case material, at further empirical research into the uses of the model, and at developing a series of connected tools that support this approach. References Bemelmans T.M.A., Bestuurlijke Informatiesystemen en Automatisering (Management Information Systems and Automation), Stenfert Kroese, The Netherlands, 1991.
2. 3.
4.
5. 6.
7. 8.
9. 10.
11.
Boehm B.W. c.s, Characteristics of Software Quality, New York, 1978. Cavano J.P., J.A. McCall, A framework for the measurement of software quality. In: Proceedings of the software quality and assurance workshop, 1978. Delen G.P.A.J., D.B.B. Rijsenbrij. The Specification, Engineering and Measurement of Information Systems Quality. Journal of Systems Software, 1992. Deutsch M.S., R.R. Willis, Software Quality Engineering, Prentice Hall, 1987 Heemstra, F.J., Kusters, R.J. en Trienekens, J.J.M., From Quality Requirement Factor to Quality Factor: an enduser based method, Proceedings of the 6th European Software Cost Modelling Meeting, Kerkrade, May 1995, pp. 18.1 - 18.19. IEEE Standards Collection: Software Engineering, Institute of Electrical and Electronic Engineers, 1994. ISO/IES 9126, Information Technology - Software Product Evaluation - Quality Characteristics and guidelines for their use, International Organisation of Standardisation, 1991. Kaposi A. and Kitchenham B., The architecture of system quality, Software Engineering Journal, 1987. Keller S.E., L.G. Kahn, R.B. Panara. Specifying Quality Software Requirements with Metrics. In: R.H. Thayer and M. Dorfman (eds.), Tutorial: Systems and Software Requirements Engineering. IEEE Computer Society Press, 1990. Trienekens J.J.M. Quality Management in Software Production, A Customer Oriented Approach. In: Proceedings of the IFIP 5.7 Working Conference on Integration in Production Management (ed. J.C.
12.
Wortmann ed.), North-Holland, Elsevier Amsterdam, The Netherlands, 1992. Willmer H., Systematische Software Qualitätssicherung anhand von Qualitäts- und Produktmodellen, Springer-Verlag, Informatikfachberichte, 1985.
Appendix: about the Spirits project ‘SPIRITS’ is an acronym which stands for Software Process Improvement in embedded IT environments. SPIRITS is a research project of Schlumberger Retail Petroleum Systems (RPS) which is executed in cooperation with two research institutes in the Netherlands. SPIRITS will develop concepts and methods for process improvement to accomplish high and quantifiable reliability of embedded products. The main objectives are the design of: • methods for process improvement in embedded systems development; • methods for measurement and evaluation of the effectiveness of process improvement activities on the reliability of embedded products. The practical application of the concepts and methods is validated in several case studies within Schlumberger RPS. The set of instruments will be based on practical experiences and will be scientifically founded and validated in a number of pilot projects.
