1994, John Wiley. Sons. p. 469-476. [4] Basili, V., M. ... [12] Paulk, M.C., C.V. Weber, B. Curtis, and M.B. Chrisses, The Capability Maturity. Model: Guidelines for ...
A PROCESS-CENTRED EXPERIENCE REPOSITORY FOR A SMALL SOFTWARE ORGANISATION By Louise Scott, Lucila Carvalho and Ross Jeffery
Report No. 02/5 Centre for Advanced Software Engineering Research (CAESER) School of Computer Science and Engineering University of New South Wales Sydney 2052 Australia © 2002 CAESER
A Process-Centred Experience Repository for a Small Software Organisation Louise Scott, Lucila Carvalho and Ross Jeffery Centre for Advanced Software Engineering Research (CAESER) School of Computer Science and Engineering University of New South Wales Sydney 2052 Australia Abstract This paper presents the design, implementation and evaluation of an experience repository in a small software organisation. The experience repository was developed and installed as part of an ongoing software process improvement effort and uses software process to structure experience and make it available for reuse. The experience repository is accessed through a web-based process guide with experiences related to particular tasks linked directly to the pages describing those tasks. This way experiences, including examples of documents, checklists or unstructured experiences such as anecdotes and lessons learnt can be easily entered and retrieved by users when required. This paper presents the design of the repository, the implementation and preliminary results regarding its acceptance and use. 1. Introduction Small organisations have and always will be under immense pressure to produce quality software quickly and competitively. To do this they need to take as much advantage as they can of approaches such as process improvement and knowledge management. In this paper we consider the installation of a basic experience repository in a small software development organisation. The experience repository is designed to assist with general software development tasks while taking into account the smaller budgets, resources and time frames typical of smaller organisations. The repository is installed as part of an ongoing software process improvement effort [14] and uses the existing (and/or emerging) software development processes as a metamodel. Experiences (such as templates, examples, checklists etc.) are linked directly to tasks in the processes and can be easily entered or retrieved through a web-based tool. Rather than adopting an experience factory or explicit experience collection approach, users are simply encouraged to enter experiences they think may be useful to colleagues as they arise. The simplicity of the system makes the initial investment and ongoing maintenance costs small for the company as well as quickly achieving a usable system. In this paper we present the design, implementation and installation of the approach and usage results and feedback from the first month of installation. The results are very encouraging, showing consistent usage and suggesting that the users are finding the tool genuinely useful in their day-to-day activities.
The next section presents background relevant to the paper. Section 3 presents the design and implementation of the experience repository tool and Section 4 presents the installation of the system. Section 5 describes the evaluation and presents the preliminary results of usage and feedback. Section 6 discusses the results and Section 7 concludes the paper. 2. Background In the software domain the growing importance of knowledge management is reflected in the establishment of approaches such as Experience Factory [3] and the inclusion of knowledge management as a key factor in many software process improvement approaches (such as CMM [12], Spice [7] etc.). The International Workshop on Learning Software Organisations [2] also encourages research and discussion on the use of knowledge management approaches in software companies. The result is a better understanding of what kinds of experience can be reused (e.g. [1]), how to collect experience (e.g. [13]), how to structure and store experience (e.g. [8]), tools for knowledge management (e.g.[4]), methods for implementing and maintaining learning organisations (e.g. [16]) and reports of practical knowledge management (e.g.[9]). Unfortunately for small organisations most of the work reported focuses on knowledge management in large organisations. Approaches developed for large organisations are generally unsympathetic to the limited resources, including time, money and personnel, available in small organisations. It is unlikely, for example, that a small organisation could afford to establish an Experience Factory with its separate “knowledge” group and the amount of time it takes to process and make experiences available for use. Despite this, sensible application of knowledge management could greatly benefit small organisations by addressing problems typical of small organisations such as short time frames for projects (little time to search for or reinvent solutions), fast changing technologies (difficult to keep up with the latest changes) and high staff turnover (difficult to compensate for lost knowledge and train new staff). For the company described in this report the idea of an experience repository followed on from a software process improvement effort that used a web-based electronic process guide with links to electronic templates to improve project documentation. Use of this system led to calls to provide examples of documents, which led naturally to the consideration of more general knowledge management. Because of the link to software process, the work of Maurer (MILOS) [10], who has used “process oriented knowledge management” to support the creation of the learning organisation was considered. This approach provides relevant, task-related knowledge as a task is being executed. Knowledge may include generic process models, project plans and project data. The claimed advantages of this system are that processes are “natural” entities for structuring knowledge related to projects and that the structure provided by the process reduces the need for browsing and searching, improves navigation of experiences and reduces information overload. The process-centred approach follows the philosophy that people do not simply need more information, but information pertinent to the tasks they are carrying out at the time.
This paper reports on the implementation and evaluation of a process-centred experience repository in a small organisation. The goal of the experience repository is to capture key management and development experience and make it quickly and easily accessible to users through their web-based process guide. The goal of the evaluation is to determine if and how knowledge management can be beneficial in a small organisation. 3. Design and Implementation The Experience Repository (ER) is implemented as a simple extension to the Electronic Process Guide (EPG) already in use at the company. The EPG is a web-based presentation of the company’s software management and development processes. The EPG is generated directly from UML activity models of the processes [15]. The models define the activities, objects, control flows and object flows of the process. The EPG generator takes the UML models, reformats them and provides navigation and links to assist with the execution of the process. Each page in the EPG consists of three frames as shown in Figure 1; a general navigation tree, a diagrammatic overview of the process and a description section. The navigation tree supports fast navigation to any process entity (activity or object). The diagrammatic overview displays the UML diagram of the process and supports browsing and hyper-navigation of the process. The description section displays the description of the currently displayed entity and any textual attributes (such as entry and exit criteria, responsibilities etc.). The EPG is designed to assist process practitioners with executing defined software processes. The ER uses the defined software process as the meta-model to structure experiences with experiences directly linked to relevant entities in the processes. This structure is both simple to implement and simple to use. Since it eliminates the need for searching or browsing the experiences it eliminates the need for complex meta-structures and databases for experiences. This reduces the initial cost of the system and (hopefully) encourages its use through its simplicity. The combined tool is called the “ EPG/ER” . The ER itself is structured into four categories; checklists, examples, templates and generic experiences. The generic experiences category allows unstructured entries such as anecdotes, lessons learned, code fragments, links to helpful web-pages or books, etc. or whatever else is considered useful by the users. There are no restrictions on generic experiences except that they must be entered by paragraph text or a link (to a file or another web-page). The architecture of the EPG/ER is shown in Figure 2. The figure shows at the top an entity page of the EPG. The description section of the page contains links to “ quicklink” pages that interface with the ER. There is a quicklink page for checklists, templates, examples and experiences. The quicklink pages contain an alphabetical listing of all entities with the associated information (i.e. with a checklist, template, example or experience) with their associated ER links. Invoking a quicklink page opens the page at the entity that is currently displayed in the EPG. Easy access to ER entries for other entities is available by scrolling the page. General experience links can be made through the experiences quicklink page or, as shown on the diagram, directly through the
description part of the EPG page. This provides flexibility for entering and retrieving general experiences.
Diagrammatic Overview
Description Section PageSeeder Tools Quicklinks
Navigation Tree PageSeeder Seed
Figure 1: Basic EPG activity page.
The ER is implemented using the PageSeeder tool for annotating web pages [11]. PageSeeder “ seeds” HTML pages with links that allow users to add comments or new links to a web page. The revisions are instantly visible to other users of the web page and, if required, notification of changes can be automatically e-mailed to group participants. PageSeeder can be used to dynamically update web-pages with new information, to ask questions about topics on web-pages and/or to have on-line discussions and exchange ideas about the content of web-pages. In the EPG/ER it allows users to dynamically link new ER entries to EPG pages, automatically notify other users of the new entries and access the ER entries through the PageSeeder links. To implement the EPG/ER tool the original EPG was extended to include the quicklink pages. The pages were then strategically seeded to provide potential links to ER entries. An example of a seeded activity page and a quicklink page are shown in Figure 1 and Figure 3 respectively. New ER entries are made by clicking on the seed in the activity or quicklink page to which the new piece of experience applies and entering a textual comment or new link using the form shown in Figure 4. If desired other users can be
automatically notified by e-mail of the new entry. Users can access ER entries by following the PageSeeder links, either directly from the e-mail or through the EPG. If the link is a textual question or comment they can also for discussion threads by replying to the entry using e-mail. ,!-!. / 02123 4 - 56789
!
" "# "$ ,2- .*: ,2;=< >@? 7 A B 5C 7 / 0!123 4 - 5678 9
* %
&%
' )(
&%
" )+
" # "$ &% ,!;=,2>? A D 78 / EGF A H'I -2J*K'IL>@8? A L C@? LA 7 H)? 7 M? I N D >O*? F 7M ? 7 A > 5 N P&7"QR 567'I7 ? CS 9
Figure 2: Architecture of the EPG/ER.
As well as being simple to implement the system offers flexible and simple usage. Users can access the ER by browsing the EPG to locate experiences linked to particular tasks or by use the quicklink pages to access ER entries quickly and efficiently. They can also use the e-mail discussion feature to keep track of ER entries that may be useful and access them directly from the e-mails. It is hoped that the simplicity of the PageSeeder system will encourage both provision and retrieval of ER entries. 4. Installation The ER was installed at the company linked to processes for general project management (“ start-up” and “ wrap-up” ) and a component-wise development project for medium-tolarge scale (by the company’s standards) web application development. The processes had been reverse engineered from task lists and created from discussions with experts within the company. All processes had been reviewed in a workshop discussion with five key people in the company (including management and development staff). Altogether the processes contained fifty activities, twenty-one objects and eight diagrams. To make the system usable from the start and see benefits quickly, experiences were collected from two experts to populate the ER. One expert provided checklists and general experiences for the “ start-up” management process while another provided
templates and examples of documents for all stages of the process. Initially there were ninety-six experiences in the experience base (mostly document templates and examples). Nine activities (18%) and eleven objects (52%) contained experiences associated with them.
Links to ER Entries
PageSeeder Seeds
Figure 3: A part of the "Templates" quicklink page.
Figure 4: The PageSeeder “comment” entry form.
Before the EPG/ER was made generally available, users attended a training session about the EPG/ER and the new processes. In all, there are twelve users of the system including both management and development staff. During the training they were introduced by a researcher to the concepts behind process engineering and process improvement, shown the EPG/ER tool and instructed in its use and told about the research project and the data that would be collected. Then they were shown the processes (including the experiences) and instructed in their use by representatives of the company. After the training session the EPG/ER was made available for general use. Users were encouraged to browse the EPG/ER and enter and retrieve experiences as required rather than via a formal knowledge management process. A separate task was set up on the company’s time recording system so that users could use the EPG/ER without suffering penalty to their assigned tasks. At this point the company was left to use the EPG/ER as they desired. 5. Evaluation A longer term goal of the project is to evaluate the use of the EPG/ER over six months to determine if it is used, how it is used, what it is used for and what benefits may be gained from having the EPG/ER. This paper reports on usage from the first month of installation of the tool to give some ideas of its success or otherwise. The data comes from a Technology Acceptance Model (TAM) [6] survey carried out after the training, EPG/ER server logs, PageSeeder logs, the EPG/ER task record and user surveys. The relationship between the data and the goals of the evaluation is shown in Table 1. Question Is the EPG/ER used?
Data EPG Server Logs Task records
EPG Server Logs User survey PageSeeder Logs User survey User survey
Measure Page accesses over time Amount of time spent using EPG/ER Number of new PageSeeder entries Reported usage Perceived usefulness Perceived ease of use Predicted usage Most accessed pages Reported most used features Type of new PageSeeder entries Reported use Reported benefits/detriments
User survey
Suggested improvement
PageSeeder Logs
Is the EPG/ER well accepted?
How is the EPG/ER used?
What is the EPG/ER used for? What are the effects of using the EPG/ER/ What improvements could be made to the EPG/ER?
User survey TAM survey
Table 1: Table of evaluation data.
The usage data from the EPG server (graphed as pages accessed (“ page hits” ) vs time in Figure 5) shows consistent use of both the EPG and Pageseeder (ER) tools. It is difficult
to see any trends (such as increasing or decreasing usage) in the data over such a short time span, but the data shows clearly that the tool is being used. The company’s task records show that thirty hours of time were booked to the EPG/ER task over the four weeks of evaluation. This amounts to almost one person/week over the month and shows that users spent a serious amount of time browsing, familiarizing themselves with and using the EPG/ER. Page hits over time 100 Number of page hits
90 80 70 60 50 40 30 20 10
24 /0 5/ 02 26 /0 5/ 02 28 /0 5/ 02 30 /0 5/ 02 1/ 06 /0 2 3/ 06 /0 2 5/ 06 /0 2 7/ 06 /0 2 9/ 06 /0 2 11 /0 6/ 02 13 /0 6/ 02 15 /0 6/ 02 17 /0 6/ 02 19 /0 6/ 02 21 /0 6/ 02
0
Time (days) Epg pages
Pageseeder pages
Total pages
Figure 5: The number of Page hits on the EPG including PageSeeder hits over the four weeks of the initial evaluation.
The TAM survey collected immediately after the training session, showed an overall positive view of the EPG/ER tool. All twelve users reported a positive perception of both the potential usefulness and ease of use of the tool. The survey predicted moderate levels of usage and acceptance for the tool. The server log data reveals the most used pages. Besides the home page (62 hits) the templates index page (25 hits) and examples index page (19 hits) were the most popular pages. An interesting feature of this data was the high usage of code example index pages (Batch files – 10, Java – 8, Omnimark – 10, XSLT – 5), indicating that users were finding code examples a useful feature of the ER. The PageSeeder logs record new PageSeeder entries during the time of the evaluation. Over the month fifty-seven new comments were entered by nine different users. This shows a reasonably large usage by, encouragingly, a large proportion of the users (nine out of twelve possible users). A large number of the new entries were code examples (22 new entries). The PageSeeder tool was also used to store new templates (6 entries) and to make external links to other web pages (3 entries). Another popular use for the PageSeeder tools was as a discussion tool, with nine entries discussing a new template,
eight entries discussing the process, four entries discussing the coding examples and three entries providing instructions on how to use PageSeeder in the context of the ER. A user survey was carried out at the end of four weeks. The survey asked open questions about usage and users opinions of the tools, including any benefits or detriments of using the tool and how the tool might be improved. The first responses were very positive. Of the twelve possible users nine reported using the tool. Five users found the tool beneficial to their work while four users reserved judgment (e.g. “ not yet” beneficial). The most used features (as reported) were the development processes and the ER in terms of adding, browsing or downloading templates and examples. Two users cited the time required to add experiences to the ER and to access and use the experiences as minor detrimental overheads to using the EPG/ER. The most requested improvement was a searching facility for the ER. 6. Discussion The results of usage and acceptance of the EPG/ER after the first month of general usage were extremely encouraging. The company is using the tool and users seem to be finding it genuinely useful. The participation rate (9/12 users) is also good. The data reveals some interesting insights into the application and use of the experience repository in the company. With respect to how the EPG/ER is being used, the templates and examples (including code examples) in the ER are the most popular features. This is probably because they deal directly with the tangible outputs of projects. Lack of use of the generic “ experiences” category (no new entries over the month) supports previous research (e.g. Schneider (LIDS) [13]) that generic experience is hard to collect and make explicit without a very proactive approach such as mandatory collection and/or an experience factory. The question remains whether this kind of experience (lessons learned, anecdotes etc.) is really useful for a small organisation. It would be interesting to encourage the input of some of these kinds of experiences, perhaps with the help of a researcher, to see whether they were used. This could give some insight into whether it is worth making some extra effort in small organisations to collect these kinds of experiences. The most beneficial effect of using the EPG/ER reported in the surveys was that it helped with creating documentation by providing templates and examples of documents from previous projects. It is interesting to note that some users could identify tangible benefits in such a short time. This suggests that not only can an experience repository be beneficial to a small organisation but that this kind of approach can work within the short time frames of small organisations. It is also worth noting that only two users bothered to record extra effort (time spent) as a possible drawback of the EPG/ER. The responses to the open survey questions reveal some interesting insights into the adoption and success of the EPG/ER. For example, the user who reported that he could not find what he wanted in the experience repository but, when he found it elsewhere, put it in the experience repository “ for the future” , suggests that the company already has
some important features of a learning organisation considered critical for successful adoption of an experience base [5]. Another comment by a user that the ER was a much more useful place to store information than e-mail folders because he knew where to find it next time is a hint from this data that Maurer may be right in his claim that process provides a natural way to structure knowledge about projects. We also note that critical success factors as reported by [5] have been present in this project, including an incremental approach, small initial cost, fast realisation of benefits and management commitment and stability. 7. Conclusions and future work In this paper we have presented the design, installation and usage results of a processcentred experience repository in a small organisation. The experience repository is a simple extension to an electronic process guide with experiences linked to tasks and objects in the guide. The experience repository has been installed in the company with a training session but without any formal experience management processes. The results from the first month of usage are very encouraging, showing reasonable usage of the tool by a high proportion of the possible users. The data shows that templates and examples (particularly document and code examples) are currently the most useful aspects of the experience repository. The data suggests that a simple, low cost experience repository can be useful in a small organisation and can realise benefits within a short time. We will continue to evaluate the use of the EPG/ER over the next six months. A longer evaluation time will reveal any trends in usage (increases or decreases) as well as provide more data on the most useful aspects of the experience repository and its benefits or detriments. We intend to build a profile of both experience repository entries and retrievals to answer important questions about what kind of experience is useful to small organisations. Acknowledgements We would like to thank Allette Systems (Australia) for providing the context for this study and the staff of Allette Systems (Australia) for participating in the study. Thanks also to Felicia Kurniawati for implementing the EPG/ER tool. Funding for this project was provided through a collaborative linkage grant by the Australian Government (DEETYA) and Allette Systems (Australia). References [1]
[2]
Althoff, K., A. Birk, S. Hartkopf, W. Muller, M. Nick, D. Surmann, and C. Tautz, Systematic Population, Utilization, and Maintenance of a Repository for Comprehensive Reuse. Proceedings of the 11th International Conference on Software Engineering and Knowledge Engineering, SEKE’99, 1999: p. 25-50. Althoff, K.-D., R.L. Feldmann, and W. Muller. Advances in Learning Software Organisations. in Third International Workshop on Learning Software Organisaitons. 2001. Kaiserslautern, Germany: Springer, Berlin.
[3]
[4]
[5]
[6] [7]
[8]
[9]
[10]
[11] [12]
[13]
[14]
[15]
[16]
Basili, V., G. Caldiera, and H.D. Rombach, The Experience Factory, in Encyclopedia of Software Engineering, J. Marciniak, Editor. 1994, John Wiley Sons. p. 469-476. Basili, V., M. Lindvall, and P. Costa, Implementing the Experience Factory concepts as a set of Experience Bases. Proceedings of 13th International Conference on Software Engineering adn Knowledge Engineering, 2001: p. 102109. Conradi, R. and T. Dingsoyr, Software Experiences Bases: a Consolidated Evaluation and Status Report. Proceedings from the 2nd International Conference on Product Focused Software Process Improvement (PROFES 2000), 2000. 1840: p. 391-406. Davis, F.D., Perceived usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 1989. 1989(September): p. 319-339. El Emam, K., J.-N. Drouin, and W. Melo, SPICE: The Theory and Practice of Software Process Improvement and Capability Determination. 1997: IEEE Computer Society Press. Houdek, F. and H. Kempter, Quality Patterns - An approach to packaging software engineering experience. Proceedings of the 1997 Symposium on Software Reusability, 1997. 22(3): p. 81-88. Houdek, F., K. Schneider, and E. Wieser, Establishing experience factories at Daimler-Benz. An experience report. Proceedings of the 20th International Conference on Software Engineering (ICSE), 1998: p. 443-447. Maurer, F. and H. Holz, Process-oriented knowledge management for learning software organizations. Proceedings of the 12th Knowledge Acquisition Workshop (KAW ’99), 1999. Pageseeder. 2001: Weborganic Systems www.pageseeder.com. Paulk, M.C., C.V. Weber, B. Curtis, and M.B. Chrisses, The Capability Maturity Model: Guidelines for Improving the Software Process. 1995, Reading, MA: Addison-Wesley. Schneider, K., LIDs: A Light-Weight Approach to Experience Elicitation and Reuse. Product Focussed Software Process Improvement: Second International Conference, PROFESS 2000, 2000. LNCS 1840: p. 407-424. Scott, L., R. Jeffery, L. Carvalho, J. D’Ambra, and P. Rutherford, Practical Software Process Improvement - The IMPACT Project. Proceedings of the 2001 Australian Software Engineering Conference, 2001: p. 182-189. Scott, L. and F. Kurniawati. WAGNER - Web-bAsed process Guide aNd Experience Repository. 2002. Report No. 02/1: Centre for Advanced Software Engineering Research (CAESER), University of New South Wales, Sydney 2052 Australia. Tautz, C., K. Althoff, and M. Nick, A Case-Based Reasoning Approach for Managing Qualitative Experience. Proceedings of the 17th National Conference on AI (AAAI-00) Workshop on Intelligent Lessons Learned Systems, 2000.
