Monitoring the Software Process: A Solution from the Customer Point of View Vincenzo Ambriola* & Giovanni A. Cignoni** * Dipartimento di Informatica, Università di Pisa Corso Italia 40, 56125 Pisa, Italy EMail:
[email protected] ** Link s.r.l. Via S. Martino 51, 56125 Pisa, Italy EMail:
[email protected]
Abstract According to the Italian law, monitoring is a kind of quality control that must be performed during the enactment of contracts related to information systems of the Italian Public Administration. In this paper, the authors describe and analyse the aspects of monitoring as a tool to guarantee the quality of software production and, as a consequence, to assure the customer satisfaction. The paper presents the administrative context where monitoring is applied, its purposes, the application areas, and the relationships with other kinds of quality control. In particular, we emphasise the novelty of monitoring and its characteristic of customer initiative.
1 Introduction Control, as a solution to achieve product quality, is a simple and intuitive technique adopted by the software industry from the common heritage of industrial methodologies. Control, moreover, is keen to be applied both by software suppliers, as a preventive measure, and by customers, as a precautionary measure. Caution by the customer is, by need, directly proportional to the amount of the investments involved. The automation process currently on-going in the Italian Public Administration is an example of great investment. Its importance is twofold: the economical dimension of the projects and their relevance in the improvement of the services offered by the Public Administration to the community.
The Italian law defines how to control the automation process of the Public Administration. To this end, the law establishes an Authority (the Authority for the Informatics in the Public Administration) with the precise purpose of guaranteeing the application of quality control. From a technical point of view, the relevance of the law is a new methodology for quality control to be applied to contracts related to implementation of computerised information systems. The proposed methodology is called monitoring [1]. Similar concepts are already defined in literature – for instance the ISO 12207 standard defines the supplier monitoring – but the Italian law defines more precisely the details and the goals of this kind of control [2] and, worth to be noticed, it states monitoring as a mandatory activity. The key target of monitoring is the identification of the problems that can arise during the enactment of a contract for the supply of software products or computerised services. The typical contract stakeholders, i.e., the customer (the Administration, in this case) and the supplier, are accompanied by the monitor, a third party with an independent role and with precise control functions. In this paper we want to give evidence to and to discuss, in the context of the Italian Public Administration, the need of a quality control process. We want also to show how this need resulted in the definition of a new kind of control. The nature of the argument is bound to the Italian context (as evident in the listed references). The authors, however, believe that monitoring is an example of quality control that can be studied in its specific aspects. Monitoring is a wheel of a complex machinery that controls the automation process of the Public Administration; the next Section of the paper defines the general context where monitoring is applied. The third Section describes the key point of the automation process: the contract enactment; monitoring is discussed at the detail level of the software lifecycle. Monitoring, in its original definition, prescribes four different application areas each aimed to control different facets of contract enactment; these application areas are the subject of the fourth Section. Monitoring is a rigorous kind of control, based on precise reference elements. Among them the most important one is the contract itself; the contents and the importance of the contract are the argument of the fifth Section. The sixth Section discusses monitoring, with a comparison to other kinds of control applicable to software production. The paper ends by showing the results, at the current state of the art, of the application of monitoring to the on-going contracts for the automation process of the Italian Public Administration.
2 The Automation Process Phases When implementing a product or a service related to its computerised information system, a Public Administration follows a predefined procedure: • strategic planning; a strategic plan for the whole Administration is built by defining the needs of the information systems and the actions that must be undertaken to fulfil them;
• programming; the actions defined in the strategic planning are translated in specific projects; each project identifies some products or services; the supply of these products or services is formalised in different contracts; • enactment; the products and the services – object of contracts signed in the previous phase – are realised; contract enactment is the object of monitoring. The strategic plan is finalised to guarantee the coherence of the service level provided by the Administration towards the needs of the whole community. In other words, the strategic plan integrates the new technologies with the organisational models and the processes of the Administration. The strategic plan must be accompanied by a continuous programming process whose primary purpose is to implement, by means of contracts, the projects that have been identified as targets of the strategic plan. The key point of programming consists of feasibility studies. A feasibility study performs a detailed analysis of the planned goals and of the technical, organisational, financial implications related to the initiative. All the contracts for implementing, maintaining, and managing computerised information systems must be preceded by a feasibility study. After a feasibility study, the achievement of a strategic plan target proceeds with the identification of one or more slots – each of them can be the object of a single contract – or a set of products and services that can be realised by a single supplier. The contract assignment follows this administrative procedure: • requirement specification; • contract signature between the Administration and the chosen supplier; the general clauses of the contract must satisfy the specifications; • authorisation to the contract enactment. In parallel with this procedure, there is a similar one for assigning the monitoring contract. The law requires that monitoring must be directly performed by the interested Administration or, in its place, by the Authority, either by an explicit request of the Administration or, automatically, in case of idleness or default of the Administration. In both cases, monitoring can be performed by a third company, specialised in this activity and belonging to a special list – maintained by the Authority – of enabled companies [3, 4].
3 Monitoring and the enactment phase The contract enactment is the part of the automation process where monitoring will be performed. The Italian law contemplates six phases for the contract enactment: for each of them we highlight the limits and the goals of monitoring. • Design; this phase has the purpose of identifying the Administration requirements and of formulating the solution in a definite form; it produces the documents related to the contract enactment, such as the design plan, the quality plan, and the cost/benefit analysis. These documents will be used, in
•
•
•
•
•
the next phases, as reference elements for monitoring. In this phase, the purpose of monitoring is to ensure the correspondence between the design results and the outcome of the feasibility study. Implementation; this is the phase where all the activities listed in the project plan and aimed to the product or the service realisation - the object of the contract - are performed. Monitoring has the purpose of verifying the correspondence between the performed activities and those declared in the project documents. In this phase, the main reference document is the project plan. Acceptance; this phase comprises all the tests, the verifications, and the proofs needed to assure, before installation, the correspondence between the provided products or services and the contract requirements. Monitoring must neither substitute nor overlaps the acceptance test, since this is a standard control activity. Monitoring, in this phase, has instead the purpose of verifying that all the tests, the verifications, and the proofs performed during the acceptance test are adequate to ascertain the product or the service quality levels, as indicated in the quality plan. Installation; it is the phase that comprises the preparation of the operative environment, the application set-up, the personnel training. Monitoring, in this phase, has the purpose to ascertain, for those aspects not covered by the acceptance test, the adequacy of the packaging, of the user documentation, of whatsoever will be needed to guarantee the best operationality of the information system; the main reference document is the quality plan. Management; this phase identifies the operative life and the effective usage of the information system. Monitoring has the purpose of evaluating the gaps between both the actual system performance and the customer satisfaction and the requirements stated in the cost/ benefit analysis document. Maintenance; simultaneous to the previous one, this phase identifies the specific activities that, still belonging to management, are characterised by correction and improvement actions on the product or on the service. In this perspective, monitoring has the purpose of verifying that the changes performed to the system are relevant to the Administration needs and that they will not compromise the characteristics stated in the quality plan.
Monitoring, then, is a form of exhaustive control of the contract enactment phases that does not change the responsibility range of the Administration and of the supplier. The monitor, either belonging to the Administration or to a third party company, has a new responsibility whose limits – a characteristic that must be highlighted – are stated in the reference documents. The Italian law underlines the co-operative nature of monitoring. The monitor, although working in the interest of the Administration to achieve the goals of the contract, is not responsible at all with respect to the final quality of the result. This responsibility is still in the hands of the supplier and, subsequently, of the Administration. The monitor, therefore, has no prescriptive power: when
discovering a gap between the contract enactment and the requirements, the monitor can only suggest the supplier to take proper initiatives.
4 The four monitoring facets We have seen that monitoring prescribes, phase by phase, how to control the performance of the activities defined by the contract and how to validate their correspondence to the needs stated by the Administration. From an orthogonal point of view, monitoring can be seen as an activity aimed to control specific project facets that, still bound to the final product quality, are not directly related to the phases of its enactment. In this “specialistic” connotation, the Authority identified the following monitoring application areas. • Supplier process monitoring. Monitoring is seen in terms of control of the supplier adequacy and reliability. The objects of the analysis are the supplier working organisation, the supplier competence in the specific contract area, the organisational structure, the warranties, even financial, offered by the supplier. The supplier process is evaluated with respect to a reference model that, according to the monitor, can be inspired to a maturity model or to international standards such as, for instance, the ISO 9000 series [5]. The best situation – and for this reason the one to be preferred in contract assignment – is when the supplier adopts a well defined and possibly certified process. In this case, the monitor activity is clearly defined in the analysis of the supplier quality manual and in the successive verification that all the directives therein contained are effectively applied during the contract enactment. • Project management monitoring. The monitor activity is finalised to the control of the correct development of the activities prescribed by the contract. The control focuses on schedules, costs, and risk provision. The main reference document is the project plan. Besides the discovery of gaps, the monitor is in charge to cooperate to identify the causes of the discovered problems. In this specific context, it is important to underline the co-operative nature of monitoring: the monitor can suggest a course of actions, always without any prescriptive responsibility. By contrast, it will be the monitor precise duty to successively verify that the prescriptions recommended by the Administration will be followed by the supplier. • Product quality monitoring. In this view, monitoring is applied in terms of control of the correspondence between the supplied product or service and its specifications, both functional and related to quality attributes. The main reference document is the quality plan: this document, besides the needs stated by the Administration, must specify the indicators to be used for evaluating and controlling the product quality characteristics. The most critical part of this monitoring facet is the risk of overlapping with other forms of quality control external to monitoring as, for instance, the acceptance test.
• Investment benefit monitoring. This monitoring facet has the purpose of controlling that the added value given by the product or by the service fulfils the Administration expectations, mainly with respect to the invested financial resources. The reference document is the cost and benefit identification, as prepared by the Administration. It is worth to note that, in the absence of problems revealed by the previous monitoring facets, the investment benefit monitoring does not put under scrutiny the work done by the supplier but the investment plan of the Administration. These four facets must not be interpreted as a decomposition of this control activity. A decomposition, in fact, could give rise to a partial application of monitoring. In its inception, monitoring aims to be complete and accurate: the identification of the above facets must be understood as an explicit description of the coverage level expeted by the Authority from this form of control.
5 The contract as a reference document Monitoring must be performed only with respect to defined reference documents. In this sense, the main goal of monitoring is the discovery and documentation of the gaps between the activity effectively performed and those prescribed, between the achieved results and the expected ones. The contract, signed by the Administration and the supplier, is the main and, from a legal point of view, the most important reference document to be taken into account by monitoring. The Italian law identifies and prescribes the issues that a contract must define. These issues can be classified in three categories: • financial; these are the issues related to payment formalities, anticipations, and penalties due in case of supplier default; • control; these issues define how the Administration can control the supplier activities; monitoring rules do not necessarily belong to these aspects of the contract that are, on the other hand, specifically devoted to other kinds of control, such as, for instance, the acceptance test; • quality; these issues specify the quality characteristics which are relevant to the object of the contract and to the expected quality levels; the required quality characteristics can be directly referred to the product or to the service or, indirectly, to the production capabilities of the supplier. To be used as reference document, the contract must be very detailed in the definition of the expected results. In particular, the contract must define: • the structure of the supplier process, including schedules and costs; these data are needed both to the process and to the project management monitoring; they can be useful also for the investment benefit monitoring; • quality characteristics of the product or of the service that will be realised; it is important to define the characteristics to be observed, preferably with a reference to existing standards (for instance, ISO 9126); the expected qual-
ity levels must be expressed in terms of a metric framework rigorously defined; this is needed for the product quality monitoring; • quality characteristics of the development process; the product quality stems from the production quality too; the expected quality level must be rigorously specified and related to a defined metric framework; this is needed for the project management monitoring. A large part of this information contained in the above points is more properly located in the quality plan, for all the aspects related to product quality, or in the project plan, for all the aspects related to project management. In these cases, the contract must prescribe the preparation and the publication of these documents, by defining for each of them the scope and the contents. Besides the contract there is a monitoring contract that must define the monitoring terms and formalities. In particular, the monitoring contract must define the range of the control activities and the documents that must be visible to the monitor. Both when monitoring is directly performed by the Administration and when it is assigned to a third party, terms and formalities must be traded with the supplier and explicitly accepted.
6 Comparison with other kind of control Monitoring is a kind of external control performed by an entity, external to the supplier, that works for the customer interest. There are other consolidated kinds of external control; in the follow, we present three of them, classified with respect to time: control can be performed before, during, or after the product or the service realisation. • Certification. It is a kind of preventive control, aimed to ensure that the supplier is able to guarantee a predefined quality level. Since it is not possible to have a product as reference element, the supplier is evaluated against parameters such as the previous implementations, its proceeds, its clients, the adopted technologies. In general, certification is issued by specialised companies or by bodies, often international, that evaluate the supplier in the absence of a contract or a customer. A relatively new approach is the evaluation of the supplier internal organisation and production policies with respect to a process quality standard – for instance the ISO 9000 certification. The limitation of the certification is that it only states the potential capability of a supplier to deliver services or software products of a given quality, without giving any real warranty on the quality of a specific delivery. • Quality assurance. This terminology captures all the controls performed during the contract enactment by or on behalf of the customer. The main problem encountered by the entity that performs the evaluation is the availability of reference documents to be used to asses the contract progress state and the quality of the intermediate products. Apart from the validation of the intermediate development products, quality assurance has more efficacy
when it can compare the performed work against a quality plan previously declared by the supplier and known to the customer. In this case, the evaluator must verify that the supplier is following the subject of agreement. In this perspective, it is of fundamental relevance the contractual clause that requires the customer knowledge and approval of the supplier quality plan. • Acceptance test. It is the last and final control that certifies that the delivered product fulfils the customer requirements. From a contractual point of view, the definition of the customer requirements has a special relevance for the acceptance test. The evaluation must be based on clear and complete requirements. A product cannot be refused on the basis of fuzzy and incomplete clauses. From a more practical point of view, it is important that the acceptance test be performed by the users that will exploit the product and in a real operating environment. Monitoring can be classified as a form of quality assurance. With this activity it shares all the issues of privacy and acceptance by the supplier. If, however, quality assurance is often performed by means of inspection techniques, very focused to well defined goals, monitoring is better defined as a continuous activity with broader purposes. The supplier process monitoring is close to certification. There are, however, some evident differences. Certification always refers to a standard and is relative to a general process, evaluated in its abstract definition. The supplier process monitoring is instead limited to a specific contract and with the purpose of evaluating the supplier capability only with respect to that contract. Furthermore, monitoring does not issue any declaration that can guarantee whatsoever the supplier capability in successive evaluations. The product quality monitoring can be compared to the acceptance test. In reality, it is a different activity that must not overlap with the acceptance test. Firstly, the acceptance test is a final control, while monitoring, by definition, covers all the contract span and interacts with each production phase. Secondly, it is not prescribed that the product quality monitoring include direct and explicit controls (as additional tests on software components), while it is auspicable that monitoring ensure that the controls performed by the supplier are apt to evaluate the product and to guarantee the quality characteristics required by the customer. In this form, monitoring would become very close to an audit.
7 Conclusions The most interesting characteristic of monitoring is its nature of customer initiative: monitoring is a control activity imposed and aimed to guarantee the Administration satisfaction. Although the need of control is admissible in consideration of the customer peculiarity – a public administration naturally bound to the entity that writes the law – this willing of initiative is a novelty. Control quality should be – and in this form is generally presented – a supplier initiative, pushed by competition and market considerations. If it is clear
that certification and acceptance test are two forms of control that fall in the category of natural actions (a form of prudence the former, a supply acceptance the latter), monitoring is a continuous and very peculiar pervasive activity. The introduction of such a control, especially as a customer request, is a signal of dissatisfaction, often followed by badwill suspicions. This recalls the initiatives of customer associations, typical of other markets than the information technology and, surely not, auspicable for the contracts of the Public Administration. Another perspective, yet not very appealing, can be identified by concentrating on the “co-operative nature” of monitoring. The monitor is not a watchdog but rather an angel that the Administration - for its interest, of course - puts besides the supplier so that “possible” problems encountered during the enactment of a contract can be identified, tackled, and solved as soon as possible. In this interpretation, there are no more badwill suspicions, but we are in presence of an inability presumption. The best possible defence that a supplier can advocate is the software crisis. The double nature – control and support – of monitoring is evident even in the discussion, currently very hot [6], on the assignment of monitoring to the Administration or to a third party. Defenders of the former hypothesis want to assign a greater responsibility to the monitor, that, as public officer, can better play a role of control and report of defaults. More pragmatically, defenders of the latter try to find in a third party the competencies needed to ensure the maximum success probability to contracts subject to monitoring. The peculiarity of monitoring has a clear fall out even in the difficulties of application reported up to now. In the strategic plan for 1996-98 [7] and in a recent workshop [8] the Authority reports the weakness of the current monitoring activities, giving evidence of a global delay in its application. The Administrations ascribe this delay to: • the preference to assign the monitoring enactment to the Administration itself, opposed by the lack of professional profiles capable of performing these tasks and of assuming the due responsibility; • the difficulty to find the financial resources needed to cover the additional monitoring activities; • the suppliers resistance in the absence of rules and specific contracts that define and limit the monitor activity; • the lack of criteria and formalities for the monitoring enactment, more detailed than those contained in the law. Even if some of the difficulties found are due to organisational and contractual aspects the main obstacle to the application of monitoring seems to be the lack of criteria and directives for its enactment. In 1995, CNR (the Italian National Council of Research) launched a strategic project for technological transfer to the Public Administration. The goal of the project, started officially in June 1996, are metrics and evaluation tools. In this context, there is a co-operation between the project and the Authority for
defining a monitoring process and for implementing a set of metric tools to be used in the evaluation of the supplier activity [9].
8 Acknowledgements This work has been done in the context of the project “Metriche e strumenti di valutazione per la Pubblica Amministrazione - Protagora” sponsored by Progetto Strategico del CNR “Informatica per la Pubblica Amministrazione”. We wish also thank Aldo Liso and Gianluigi Raiss, both of the Authority for Informatics in the Public Administration, for the fruitful discussions. Anna Maria Gorrasi and Vincenzo Gervasi gave their contribution during many discussions on the technical aspects of monitoring.
9 References Many of the listed references point to Italian laws. In particular [1] is the Italian law in which, for the first time, monitoring is defined; [2] is the first edition of monitoring guidelines; [3] contains the first list of contracts subject to monitoring and the first list of companies enabled to act as monitor; [4] is an updated list of companies enabled to act as monitor. 1. Norme in materia di sistemi informativi automatizzati delle Amministrazioni Pubbliche, a norma dell’art. 2, comma 1, lettera mm, della legge n. 421 del 23 ottobre 1992, Decreto legislativo n. 39, 12 febbraio 1993, 2. Art. 13, comma 2 del decreto legislativo n. 39/1993: monitoraggio dei contratti di grande rilievo relativi a progettazione, realizzazione, manutenzione, gestione e conduzione operativa dei sistemi informativi automatizzati: criteri e modalità, Circolare n. 5/AIPA, 5 agosto 1994, 3. Art. 13, comma 2, e art. 17, comma 2, del decreto legislativo n. 39/1993: prima determinazione dei contratti di grande rilievo e attività di monitoraggio, Circolare n. 3/AIPA, 28 ottobre 1993. 4. Art. 13, comma 2 del decreto legislativo n. 39/1993: elenco delle società individuate dall’Autorità per l’Informatica nella pubblica Amministrazione, alla data del 9 novembre 1995, ai fini dell’attività di monitoraggio, Circolare n. 9/AIPA, 14 novembre 1995. 5. Ince D. , ISO 9001 and Software Quality Assurance, McGraw-Hill, 1994. 6. Cocco F. Il monitoraggio dei sistemi informativi, la circolare AIPA 5 agosto 1994 n. 5, spunti di riflessione, Informatica ed Enti Locali, 4/94 Vol. 12, 1994. 7. Piano per l’Informatica nella Pubblica Amministrazione, Istituto Poligrafico e Zecca dello Stato, 1996. 8. Proceedings of the AIPA Workshop “Il Monitoraggio dei contratti della Pubblica Amministrazione: prime esperienze”, Roma, June 26, 1996.
9. Ambriola, V. & Cignoni, G.A. Metriche e Strumenti di Valutazione per la Pubblica Amministrazione, Proceedings of the AICA Workshop “Metriche e misure nell’ingegneria del software”, Benevento, November 22, 1995.
