certificate. Limited logging meeting. Virtual logging meeting limited number .... vol.2: Building Quality into Software, Computational Mechanics Publication,.
Software inspection - a blend of discipline and flexibility Ilkka Tervonen, Juha Iisakka and Lasse Harjumaa
Abstract: Software engineering staff understand software inspection as a rather formal and disciplined process in general. Unfortunately this formality also detracts from its popularity among software practitioners. If we could add flexibility in the form of virtual meetings, for example, we could remove the barriers to including inspection in the development process. The present paper discusses alternative forms of inspection which provide discipline and flexibility on different scales and allow companies to choose an appropriate inspection method. We also introduce an inspection tool for increased flexibility.
1. Introduction It is largely accepted that formal inspection is the most effective means of finding defects, and thus plays an important role in the software development process. Although the inspection process is defined in an extremely formal way, each company normally has its own way of tailoring it. These tailored versions differ in type and in the degree of formality. This tailoring is understandable, as practitioners see formal inspection as being too strict a method. In the present paper we discuss these extreme aspects of inspection - discipline and flexibility, how they work together and as a blend provide a pleasant and effective way of defect removal. The discipline is defined by means of prescriptive metrics which set goals for the inspection process and a quality model which states the quality hierarchy and the terms. It is suggested, for example, that between 4 to 6 software engineers should participate in an inspection [1]. Other guidelines for individual inspections involve size and effectiveness metrics, for example. The flexibility aspect comes from the implementation of the inspection process and the supporting tools. The preparation for the inspection (individual inspection) and the logging meeting form the major phases in the conventional inspection process. In our earlier paper [2] we discussed reorganisation of the inspection process and introduced four new types, virtual logging meeting, limited logging meeting, pair inspection and no-logging meeting inspections. Although face-to-face meetings typically provide a natural way of negotiating and collecting opinions, they also waste time and are difficult to arrange. A shift to more flexible implementations of meetings is thus an understandable alternative. A pair inspection consists of only two persons, who are very familiar with the material being inspected. A limited logging meeting gathers a limited group together from among the total set of inspectors. It is not necessary for every inspector to come to a logging meeting since a smaller group is able to check the issues. Besides, it may be hard to get all the inspectors to come because of timetables. Consequently the logging meeting can be held in a virtual form, with the inspectors using a communication channel such as the Intranet. When there is no need for group acceptance of errors and the author can him/herself check the status of the issues, the logging meeting becomes unnecessary. WWW technology provides capabilities for a virtual logging meeting. We have experimented with its support by means of a WiT (Web inspection Tool) which provides a set of functions for distributing the document to be inspected, annotating it, searching related
documents, choosing the checklist and gathering inspection statistics. We also briefly evaluate its support for virtual logging meetings.
2. Towards reorganised inspection In our earlier paper [3] we discussed the requirements for a reorganised inspection process, focusing especially on reducing the number of logging meetings. Philip Johnson discusses a similar reorganisation of inspection in his recent article [4], considering the reorganisation problem from a broader viewpoint and introducing seven recommendations for the future of FTR (formal technical review, i.e. inspection): (1) provide tighter interaction between FTR and the development method, (2) minimise meetings and maximise asynchronicity in FTR, (3) shift the focus from defect removal to improved developer quality, (4) build organisational knowledge bases on the review, (5) outsource the review and insource the review knowledge, (6) investigate computer-mediated review technology, and (7) break the boundaries of review group size. When we evaluate these from the discipline and flexibility viewpoints, we find guidelines for the further development of inspection, such as (1) inspection should be more tailorable, (2) inspection should be more independent of time and place, (3) inspection should include more comprehensive metrics for measuring success, (4) inspection should actively focus on process improvement, e.g. evaluate the relevance of checklists, (5) the inspection team should hire external consultants to participate in inspections and buy style guides, for example, (6) the inspection tools should be based on network and web technology and utilise their benefits, and (7) in some cases the number of inspection participants may be over the limit of 6. Figure 1 depicts the two dimensions, discipline and flexibility, and defines them by means of the characteristics discussed earlier. We also illustrate the usability of these dimensions by placing conventional inspection and walkthrough in the figure. The arrows present the two roads for proceeding towards reorganised inspection.
Discipline * roles * procedures - - checklists - - style guides * metrics - - goal values
Conventional inspection
Goal Walkthrough
Flexibility * place and time independent * network and web technology * tailorable to process * varying number of participants, external consultants Figure 1. Two directions towards reorganised inspection
The discipline dimension explains the formal aspect of inspection. It means following the roles and procedures of Gilb and Graham [5], for example. The guidelines discussed earlier suggest more comprehensive metrics and checklists for this dimension, because relevant and up-to-date checklists increase the benefits of inspection. The style guides and language-specific checklists may be examples of insourcing expertise. Prescriptive metrics with goal values are useful for monitoring inspection, i.e. defect removal statistics are not enough. The majority of the above seven guidelines focus on the flexibility dimension, leading to characteristics such as tailorability to the development process, independence of time and place, external consultants, network and web technology and a varying number of participants. We now consider in more detail some new and interesting aspects of the two dimensions. 2.1 New characteristics of the discipline dimension We agree with Johnson that we need more comprehensive metrics for inspection and introduce two examples of them here, namely inspection maturity and grade of competence. They explain in more detail some aspects of the inspection process apart from simple defect removal. Maturity of inspection is a fairly new metric. Grady and von Slack [6] define the equation for the extent of adoption as follows: extent of adoption = inspection process maturity * (percentage of projects using inspections + weighted percentage of documents inspected) * constant. Inspection process maturity is a constant from 1 to 14 based on a five-level model in which levels 1 and 2 are informal peer reviews or formal walkthroughs (weights 1 and 3), level 3 is an industry-typical inspection process (weight 10), level 4 is a best-practice inspection process (weight 12) and level 5 links formal inspections to defect prevention (weight 14). The percentage of projects using inspections presupposes that the number of inspections in a project should be four or more for inclusion in this percentage, while the weighted percentage of documents inspected refers to the fact that different types of defects require different relative costs for fixing them. The equation could be (5.7 * percentage of requirements + 2.5 * percentage of design + 2 * percentage of test + percentage of code)/ 11.2). The constant (e.g. 0.05) is used to fit the extent-of-adoption metric into a range of 0 to 100. Grade of competence explains the skills of the inspector in terms of a knowledge gap or the quality of the inspector's comments, for example. The knowledge gap referred to is the difference between the inspector's knowledge and that of the designer, and can be represented in terms of the number of years of experience and the number of cases worked on in the problem domain [7]. A positive knowledge gap is recommended for successful inspection. Borrajo [8] introduces a more advanced principle based on the author's feedback regarding the improvement proposals (issues). This means that the authors say whether they accept the improvement proposals as representing defects and classify them as very interesting, normal, spelling (do not contribute to the information content), of little interest, or of no interest at all. This evaluation allows the inspection leader to analyse the competence of the inspectors and decide on any training needed for them. 2.2 New characteristics of the flexibility dimension Web technology seems to provide the greatest benefits for more flexible inspection, serving in effect as a base solution which makes other flexibility characteristics such as place and time
independence possible. It is evident that the World Wide Web (WWW) has at least the following advantages [9],[10]: * It is highly platform-independent. WWW browsers are available for all extensively used operating systems. * Because of the recent popularity of the WWW, browsers are already installed in many organisations. Browsers are also moderately easy to use and inexpensive to purchase, so there is no need for extensive investment. * WWW is global. * Building WWW applications is becoming simpler with the aid of sophisticated tools such as Java, ActiveX and their Development Environments. * Information stored on the web pages is (ideally) accessible at any time. Work does not have to be completed during office hours. * The user-oriented nature of the WWW allows end-users to exercise total control over the flow of information: what they want to see and when they want to see it. All in all, web technology is a source of capabilities for flexible virtual meetings, for example. We introduce it and some other flexible inspection types in the next part of the paper.
3. Discipline and flexibility in different types of inspection As discussed earlier we need change the conventional inspection for a more flexible practice. Flexibility can be increased by means of new types of logging meetings and specific supporting tools. In our earlier paper [2] we discussed reorganisation of the inspection process and introduced four new types, virtual logging meeting, limited logging meeting, pair inspection and no-logging meeting inspection. Face-to-face meetings typically provide a natural way of negotiating and collecting opinions, but they also waste time and are difficult to arrange. A shift to more flexible implementations is thus an understandable alternative, although these raise some new problems such as insufficient understanding of the material and communication difficulties. This problem may be solved by means of additional information presented in the form of a design rationale. Insufficient communication can be remedied by means of collaborative tool environments, which can be implemented with web technology, for example. The discipline dimension is based on the formality of the inspection types. We may characterise this by means of the question “How tightly defined is the meeting?“ The flexibility dimension is somehow difficult, however, because we consider it from the viewpoint “How easy is it to organise the meeting?” This means that face-to-face meetings are always less flexible than so called virtual meetings, although from another viewpoint we may argue that face-to-face meetings are more flexible because they involve human resources. In Figure 2 we place the four new types of inspection and the traditional (conventional) inspection and walkthrough on discipline and flexibility dimensions. Conventional inspection, following Fagan [11] and Gilb [5], has no flexibility aspect at all, but is strictly disciplined, having fixed participant roles, agenda and entry and exit criteria. One can divide its entry criteria into two groups. First the inspection leader checks whether the material is ready for inspection. Second, the participants in the meeting have entry criteria, since there is an official minimum for preparations. At the same time there are exit criteria, as there is an exact maximum number of errors per page when the document is approved in the meeting [5]. The older concept of walkthrough is clearly more flexible and less disciplined. The discipline dimension
in the walkthroughs consists of the manners of acting on a public occasion. The main principle is that only one person is speaking at a time. The flexibility dimension in the walkthrough is not complete either, since it entails a very informal but still well established way of acting. One participant acts as chairman, one as scribe, one as speaker and the rest give unprepared comments.
Conventional logging meeting
Virtual logging meeting
limited number of participants
may include a face-to-face part
Controversial issues
Shared & controversial issues
No logging meeting
Discipline
Unique issues
Limited logging meeting
requires a trained moderator
Walkthrough session
Collected issues
no preparation no strict rules
Pair inspection Comments and opinions
a couple check the material and finally write a certificate Certificate
Flexibility Figure 2. New types of inspection placed on discipline/flexibility dimensions 3.1. Virtual logging meeting A virtual logging meeting is at the same time both a disciplined and a flexible way of inspecting a document. It is a combination of public and group inspection [3]. Logging in public inspection may be implemented at the same time but in a different place or at a different time and in a different place, for example. This physical and temporal distribution of the inspection reduces the mutual time required, makes reconciliation of participants' timetables easier and provides the flexible dimension in a virtual logging meeting.
The logging and recording of agreed, unique issues is one responsibility of the moderator. In a public inspection, support is required for flexible communication between participants. This is an area of innovative research in computer-supported co-operative work (CSCW) and organisational memory. In order to manage distributed inspectors and their comments, the virtual logging meeting needs a fairly fixed protocol for acting. Moreover, CSCW tools demand a certain way of acting. This is why the virtual logging meeting has simultaneously a highly disciplined dimension. The moderator's duties in particular are strictly defined. If controversy is evident or discussion is required, a group inspection (face-to-face meeting) may be called, mainly to continue the discussion of open issues, i.e. topics on which the participants disagree. The inspection leader will steer the discussion of controversial issues, the group will discuss unresolved issues and negotiate over them, and their opinions will be recorded. As this phase focuses on controversial issues (i.e. only a small part of the prepared material), the general guidelines for an inspection meeting (e.g. inspection rate) are not relevant. 3.2. No logging meeting The traditional inspection style demands a trained moderator who has a lot of work to do in one inspection, whereas the participants should be motivated to find errors anyway, without anyone to guide them. The inspectors can send their issues and comments to the author privately and he/she can check them alone and decide whether there is any need for action or not. There is no need for a meeting at which a whole group can together say what is wrong with the material. One risk factor in this approach is the competence of the inspectors. Borrajo [8] suggests that the improvement proposals given by the inspectors should also be evaluated by the authors. This would guarantee efficiency on the part of the inspectors. Inspection without a logging meeting is a special case of a virtual logging meeting, where in any case the inspectors do not necessarily meet each other face-to-face. Normally they just send their issues to the common "notice board" where the other inspectors are able to study them and comment on them. If the comments from the other inspectors are irrelevant and the author merely checks the issues, the virtual logging meeting has become a no logging meeting inspection. A no logging meeting is semi-disciplined and totally flexible. Because there is no virtual meeting, there is no need for the readers to comment on each other, and therefore it is not so disciplined as a virtual logging meeting. Even so, the networking demands some rules so that it will not be totally undisciplined. On the other hand, no logging meeting is as flexible as the virtual logging meeting, for it has practically no temporal or physical demands. (Except the deadline for the inspection day.) 3.3. Pair inspection In peer inspection [3] a software designer participates in the inspection and guides the process. His/her role is similar to that of the design rationale in preparation (individual inspection), i.e. he/she explains the background to the design decisions and guides the inspection to focus on the topics that are most essential. The use of two people doubles the amount of time, but their co-operation may increase efficiency. Pair inspection is a special type of peer inspection. Two employees make a designerinspector couple and change roles after a while so that they act as inspectors for each other. As
they form a "permanent" couple and each inspects the other's work, and we can assume that they take it more seriously than when participating in a normal inspection with many other people. The inspection practices of such a pair will evolve step by step during discussions, and thus it is meaningless to formulate any strict rules of action. This means that pair inspection is a very flexible and undisciplined way of inspecting. The recording of "temporary" issues (defects) found in informal meetings is not required, although a pair inspection needs a form or template of its own in the final (formal) meeting on which both the author and the inspector guarantee that the material has passed the pair inspection. This final document is the only disciplined aspect. 3.4. Limited logging meeting Inspectors ought in any case to write issues down before the meeting, as this helps the scribe, and as the issues themselves may be very complicated. In this way they can be sent to the author by electronic mail to be collected and brought to the meeting. If it is hard to find a suitable meeting time for all the participants, only the most important and technically most competent employees may need to participate in the meeting, check all the issues and decide what to do. The limited logging meeting is a specialisation of the virtual logging meeting in the same way as the no logging meeting is, but in this case the common notice board is replaced by a smaller group of inspectors. In principle, the limited logging meeting is a "normal" inspection meeting, and it should therefore be quite disciplined, but the limited logging meeting can be a specialised form of pair inspection. When only a couple of people -- the author and another person (e.g. the chief designer) -- check the issues together, they can work iteratively, focusing more on issues than on the document to be inspected. In this sense, it is some times a very flexible way of inspecting the material.
4. Supporting tools for virtual logging meeting We advertised earlier in this paper the advantages of web technology. We can now give some examples of the benefits that it provides and the supporting tools that we have for a virtual logging meeting. Web technology facilitates the collaborative aspects of inspection, because the web is global and work is not limited to working hours. This means that it is easy to distribute the artefacts for inspection. The web also supports the collection of inspection results. Thus the inspection leader may decide whether a face-to-face meeting is necessary or not. Even the early review/inspection tools, such as CSRS (Collaborative Software Review System) [12], supported the collection and recording of review data. Likewise, the web-based prototypes such as Code Review [13], developed at the University of Arizona, and WiP [14], developed at the University of Oulu, support inspection of text documents with WWW browsers. The on-line commenting facilities which they provide are nice and easy. These tools are similar in many aspects, as they both support inspection by means of checklists, collect defect data, provide links from code lines to these data and allow readable summary reports to be produced.
Based on experiences with WiP [15], we are now developing a more comprehensive inspection tool, called WiT (Web inspection Tool), the main feature of which will be the ability to inspect any HTML document. As depicted in Figure 3, the inspector may mark a suspect item with a vertical line at the left, which follows the metaphor of correcting paper documents. The most popular word processors today can easily convert documents into HTML, so that virtually all written material produced with the aid of computers becomes available for inspection. Instead of reading a document through in an applet or application which is embedded into a web page, we now are able to inspect the web page itself. This makes the organisation of the client side of the tool much more convenient and complies better with the web metaphor, which has proved to be easy to use and has been much appreciated by millions of World Wide Web users. By tightening the integration between the tool, web page and browser we should be able to achieve a more understandable and favourable environment for the inspection. Thus, the approach is to emphasise the document orientation of the inspection, obscuring the technical aspects of the tool.
Figure 3. Marking a suspect item in WiT tool
5.
Conclusions
Face-to-face meetings typically provide a natural way of negotiating and collecting opinions, but they also waste time and are difficult to arrange, so that a shift to more flexible implementations of meetings would be an understandable alternative. The present paper discusses the problem of flexibility in the context of inspection. We must remember that successful inspection requires defined procedures, so that flexibility always has some limits arising from the need for discipline. We characterised the discipline dimension with the question “How tightly defined is the meeting?” and flexibility with the question “How easy is it to organise the meeting?” In this space of two dimensions we evaluated four types of inspection: virtual logging meeting, limited logging meeting, pair inspection and no logging meeting inspection. The virtual logging meeting is the most interesting type of inspection, because it provides at the same time both a disciplined and a flexible way of inspecting a document. It is a combination of public and group inspection. Logging in a public inspection may be implemented at the same time but in a different place, or at a different time and in a different place. This physical and temporal distribution is an example of flexibility and reduces the mutual time required, makes reconciliation of the participants' timetables easier and provides the flexible dimension for a virtual logging meeting. The existing web-based prototypes support inspection by means of checklists, collect defect data, provide links from code lines to these data and allow readable summary reports. Based on experiences with our earlier prototype, we are now developing a more comprehensive inspection tool called WiT (Web inspection Tool) which will support the inspection of any HTML document.
References [1] [2] [3] [4] [5] [6] [7] [8]
Tervonen I., and Iisakka J., Monitoring Software Inspections with Prescriptive Metrics, Software QA, vol 4., no. 3, 1997, pp. 16-23 Iisakka J., and Tervonen I., Painless Improvements to the Review Process, Software Quality Journal, 7, 1998, pp. 11-20 Tervonen I., and Oinas-Kukkonen H., Reorganizing the Inspection Process, Software Process - Improvement and Practice, 2, 1996, pp. 97-110 Johnson P.M., Reengineering Inspection, Communications of the ACM, vol 41, no 2, 1998, pp. 49-52 Gilb T., and Graham D., Software Inspection, Addison-Wesley, Wokingham, England, 1993 Grady R.B., and van Slack T., Key Lessons In Achieving Widespread Inspection Use, IEEE Software, vol 11, no 4, 1994, pp. 46-57 Fan C-F., and Yih S., Prescriptive Metrics for Software Quality Assurance, Proceedings of the First Asia-Pacific Software Engineering Conference, IEEE Computer Society Press, Los Alamitos, CA, 1994, pp. 430-438 Borrajo Iniesta J., A Tool and A Set of Metrics to Support Technical Reviews, in Ross M., Brebbia C.A., Staples G., and Stapleton J.(eds.), Software Quality Management II vol.2: Building Quality into Software, Computational Mechanics Publication, Southampton, U.K., 1994, pp. 579-594
[9]
[10] [11] [12]
[13] [14] [15]
Appelt W., and Busbach U., The BSCW System: A WWW based Application to Support Cooperation of Distributed Groups, 1996, Proceedings of WETICE 96: Collaborating the Internet: The World Wide Web and Beyond, IEEE Computer Society Press, Los Alamitos, CA, 1996, pp. 304-310 Berners-Lee T., WWW: Past, Present, and Future, IEEE Computer, October 1996, pp. 69-77 Fagan M.E., Design and Code Inspections to Reduce Errors in Program Development, IBM Systems Journal, vol 15, no 3, 1976, pp. 182-211 Johnson P.M., and Tjahjono D., Improving Software Quality through Computer Supported Collaborative Review, in De Michelis G., Simone C., and Schmidt K. (eds.), Proceedings of the Third European Conference on Computer Supported Cooperative Work, Kluwer Academic Publishers, Dortrecht, Netherlands, 1993, pp. 61-76 The URL of Code Review demo is http://128.196.205.213/startrev.html, user account from Tom Rodgers The URL of WiP demo is http://rieska.oulu.fi/~marcus/test.html, user account from Lasse Harjumaa Harjumaa L., and Tervonen I., A WWW-based Tool for Software Inspection, Proceedings of the 31st HICSS Conference, vol III, IEEE Computer Society Press, Los Alamitos, CA, 1998, pp. 379-388
