1
A Software Engineering Lessons Learning Repository Warren Harrison
[email protected] Department of Computer Science Portland State University Portland, OR 97207-0751
ABSTRACT Most software organizations possess a large, but informal, corporate memory. Large, because the experiences of every software engineer and manager contribute to the whole. Informal because there is seldom an explicit mechanism for disseminating this wisdom except “over the water cooler”. As a consequence, the same problems and mistakes continue to resurface and the same solutions are re-invented. In order to better exploit this informal collection of wisdom, the key points of each experience can be placed into a Repository for later dissemination. We describe a web-based Lessons Learned Repository (LLR) that facilitates contribution of Lessons, and their subsequent retrieval.
1. Introduction Most software organizations possess a large, but informal, corporate memory. Large, because the experiences of every software engineer and manager contribute to the whole. Informal because there is seldom an explicit mechanism for disseminating this wisdom except “over the water cooler”. As a consequence, the same problems and mistakes continue to resurface and the same solutions to the same problems are re-invented. The major factor that contributes to this state of affairs is lack of communication between “generations” of software engineers and managers within a given organization. This lack of communication is due to a variety of reasons – volatile work force, use of temporary contractors, geographically dispersed project groups, etc. Some organizations have attempted to address this issue by providing explicit opportunities within the software development process to share Lessons Learned by engineers and managers. For instance, a number of organizations attempt to capture Software Engineering “Lessons Learned” by holding project post-mortems or post-project review meetings [1]. The purpose of these exercises is to provide a formal mechanism to transfer the team’s experiences to the “corporate memory” upon completion of a project when the experiences are still fresh in the minds of the participants. This activity is integrated into the organizational software process as the final step before a project can be considered “complete”. Because of the potential value from this experience transfer, some corporations are willing to
2 invest significant resources into Post Project Reviews (PPRs). For instance, such a process has been used in one form or another at one company we have been closely involved with for the past twenty years. At this company, the process involves a set of specially trained facilitators, a budget for off-site meetings, and the time of the entire project team for anywhere between 4 and 20 hours for every project that ships (for a large project team this could be equivalent to over one person-month of effort). A significant portion of the PPR is devoted to collecting the project team's experiences with respect to what went right and what went wrong on the project. Copious notes are taken during the review and a complete PPR report outlining “lessons” and “experiences” is prepared by the facilitators. However, we have found that much of this hard-won, expensive wisdom is filed away and never again accessed once the product ships. For instance after reviewing almost twenty years of PPR reports, we found several specific “Lessons” such as “Have software and hardware Engineers communicate during design of the hardware” and “don’t let hardware engineers modify the instruction set after the design is finalized” occurred with predictable frequency Upon our initial analysis it appeared that the PPR reports may have not been used because of lack of awareness. However, it quickly became clear this was not the reason. Every software engineer at the company who had ever been part of a project that actually shipped has participated in a PPR, and the facilitators in fact distribute a draft of the PPR report to each project member for review before it is published. Thus, the existence of PPR reports is well known within the organization. The overriding reason why the PPR reports were seldom consulted is because no formal mechanism to ensure dissemination of the information beyond the participants of the review meeting itself existed. PPR reports were not conveniently accessible because the official “repository” was a filing cabinet in the office of the Vice-President of Software Engineering. Even attempts to put the reports on the corporate intranet in PDF format were not successful. We finally concluded that rather than the access to the reports being the issue, the real issue was access to the information within the reports. Users had to wade through dozens of PPR reports to locate all the experiences that pertained to a particular situation. This is because the information was organized by project rather than topic, work product, etc. For instance, if a user wanted to benefit from Lessons learned about negotiating with customers, they’d have to scan the twenty years or so of PPR reports to see (a) which ones included such Lessons and (b) what exactly the Lesson was. Clearly, as the repository of reports became larger and larger their value to the organization declined. We addressed this issue by creating a web-based “Lessons-Learned Repository” (LLR) of organization-specific software engineering knowledge. The goal of the LLR was to increase the Return on Investment of the PPR process by making such knowledge convenient to
3 contribute and use. While the exact web pages and CGI scripts that were used to implement the LLR are of some passing interest, the real value of such a system is the contents of the Repository itself, the Lessons.
2. Characteristics of a Lesson Learned The goal of formalizing an organization’s corporate memory is the broad dissemination of information about experiences that will discourage the use of work practices that lead to undesirable outcomes and encourage the use of work practices that lead to desirable outcomes. We call these experiences Lessons. Given the focus of this effort, a Lesson Learned can be defined as either •
a good work practice or innovative approach that is captured and shared to promote repeat application, or
•
an adverse work practice or experience that is captured and shared to avoid recurrence.
Generally speaking, the dissemination of Lessons Learned can take many different forms, from “low-tech” newsletters and memos to electronic mailing lists. A “Lessons Learned Repository” does not necessarily imply a web, or even computer-based mechanism for dissemination. Because it is quite easy for Lessons Learned to degenerate into poorly thought out speculation or grumbling we further restrict useful Lessons to those that are: •
Implemented. The work practice or approach being described must have really been exercised - it is tempting to express a speculation such as: if we’d had included software engineers in the marketing plan, we would have come up with a more feasible design, but, until we actually try this out, it is a speculation, not a Lesson. The Lessons in the Repository should reflect “tried and proven” experiences so adopters will know the Lessons have worked before.
•
Applicable. The Lesson must be phrased in a general enough manner so that it is transferable to other circumstances, yet it must be specific enough to identify a particular action that should be taken. For instance, a contribution such as: look out for “Goldplating” in the requirements really isn’t specific enough - this experience may be better expressed as: in collaboration with marketing, identify essential features and those features that would just be “nice to have” - as the deadline approaches, eliminate any “nice to have” features which have not yet been implemented. On the other hand, a comment such as we really didn’t need the ability to generate all seven reports is too specific for others to use.
•
Valid. The contribution must have a significant impact on some outcome and be
4 factually and technically correct. It would be difficult to defend a contribution such as: instead of wasting time analyzing requirements, we just started coding as soon as the group was formed. The particular project in question may very well have been successful, but it is unlikely the success was a result of this behavior.
3. A Web-based Lessons Learned Repository A Lessons Learned Repository may take on a number of different formats. For instance, a primitive “Lessons Learned Repository” might simply consist of a filing cabinet full of written reports, whose contents are disseminated by the user browsing through the files. On the other hand, Lessons could be stored in an electronic database coupled with a sophisticated query mechanism. Regardless of the form they take, most Lessons Learned Programs encompass the following three activities: •
Collecting the knowledge. As individuals work in an organization, they gain experiences, some of which can be shared with others to the benefit of the organization. The value of a Lessons Learned Program is partially a function of how much of this experience people are willing to contribute. The most significant impediment to collection of experiences is getting individuals to take the time to record them in an appropriate format. This is especially true if the “appropriate format” is a multi-page report.
•
Storing and maintaining the knowledge. As knowledge is contributed, it must be organized for access in the future, otherwise, important experiences may be effectively lost to the organization. In an environment with dozens, or even hundreds of Lessons, this “indexing” is non-trivial. Likewise, since many shared experiences will be situation specific, guidelines on how to transform the experience into a more general Lesson are important to maximize applicability.
•
Retrieving and distributing the knowledge. Regardless of how much past experience is collected and stored, a repository of Lessons Learned will not have any value unless others are able to access and profit from them. A frequent problem with this activity, is that the user often-times must wade through a number of irrelevant experience reports until they come across one which applies to their situation due to the way in which the Lessons are organized.
We addressed these issues in the following ways: Collecting the knowledge. Access to the Repository is via the web. Suppliers of Lessons
5 use a forms interface to submit experiences for addition to the Repository. The information entered consists of: •
A generalized 2-3 line summary of the Lesson.
•
A restatement of the summary in a “Checklist format” (see the discussion of “Checklist Generation” later in this section).
•
A discussion of the Details relating to this Lesson.
•
A series of "classifications" consisting of Domain, Beneficiary, Phase, Work Product and Functional Category. The classification information is selected from a pre-defined list of choices. Because the classification information is used later for accessing the Lessons, having predefined classifiers maximizes consistency for retrieval. Notwithstanding the importance of consistency, in the event the user finds no pre-defined entry that meets their needs, they may select Other, at which point they have an opportunity to add a nonstandard value. The user may associate multiple classifications with a particular Lesson.
•
Optional electronic mail/name of the Supplier.
•
Optional Project Name.
These items aid in providing the contributor guidelines in terms of what should be recorded as well as supporting the organization of the Lessons for later retrieval. Additionally, a series of user-contributed comments consisting of others' relevant experiences, comments, etc. is associated with each Lesson. When viewing the content of a Lesson, a user may follow a link to view or add to the comments. Storing and maintaining the knowledge. Each Lesson is stored as a marked-up web page consisting of a series of both HTML and XML tags. HTML tags are included so a given Lesson can be accessed outside the context of the LLR environment and XML tags are used for targeted retrieval. Thus, these pages can be accessed equally well from the LLR as well as directly via a web server from any web page or client allowing users to e-mail pages to each other and/or provide links that do not require use of the LLR. Each page consists of the Lesson text, as well as “XML tag pairs” such as and which bracket the “classifier keywords” by which the Lesson is logically organized and retrieved. Retrieving and distributing the knowledge. Individuals use the LLR by retrieving experiences contributed by others. Access to the contents of the Repository is via the web. Lesson Retrieval is of four kinds:
6 Targeted Retrieval. The value of the LLR is greatly enhanced if it can be depended upon to deliver pertinent information without requiring the user to wade through a number of irrelevant Lessons. The user may specify up to four classifications: Domain, Beneficiary, Phase, Work Product and Functional Category. The user selects one or more classifications from a set of scrolling menus. In response, they are presented with the Summary information from each Lesson matching one or more of the classifications along a “match score” indicating how strongly related the Lesson is to their initial query. Each Summary is linked via the web to the full Lesson that may be viewed by clicking on the Summary. Users may contribute comments describing their own experiences that are then added to the Lesson for future access. Full Text Retrieval. In some cases, the user may choose to retrieve Lessons based on the appearance of a specific word or words in the body of the Lesson. For instance, the user may wish to retrieve all Lessons that mention "variable". The user may specify up to four such terms and retrieve the Summaries from each Lesson that contains one or more of these words in a manner similar to Targeted Retrieval. Repository Browsing. At times, users may wish to browse the entire repository. As with the previous options, the user is presented with the Summary information from each Lesson. Each Summary is linked to the full Lesson. Checklist Generation. The value of written checklists to the peer review process is well known. One important use of information from the LLR is its use in helping to prepare review checklists. For instance, if experience gained in past projects suggests a particular action should or should not be taken, including this item on a checklist can help ensure the continuation of the practice (or avoidance of the practice, as the case may be). A Checklist consisting of information from one or more Lessons (see the earlier discussion on “Collecting the Knowledge”) can be automatically generated from the LLR for a given class of Work Product and/or Process Phase. For instance, a Checklist entry such as “Did Hardware Engineering sign off on the Project Plan” may very well avoid problems later when Hardware wants to modify the instruction set. The checklist can be printed or saved to a file.
4. Administering the Repository Obviously. the LLR is intended to be used for many years. Thus both mechanical issues as well as issues of policy are significant. Effective performance of the following administrative tasks will be necessary to maintain the value of the Repository: Deleting Lessons. Lessons may be removed from the Repository for a variety of reasons. In
7 some cases, the Lesson may be institutionalized by making it part of a documented process, rendering it obsolete within the Repository. In other situations, the content of a Lesson may be found to be wrong or misleading. Sometimes, a duplicate Lesson will be discovered so one copy has to be removed. This raises a significant issue in that ultimately some party must be responsible for deciding if a “Lesson is misleading or wrong”. Likewise, this may also imply some sort of filtering of Lessons to be added to the Repository. Making corrections and updates to Lessons. Sometimes contributions will be found to have a misspelled word or typo, a poorly phrased passage or simply incorrect content. This is related to the issue of deleting items, but somewhat less sensitive. Updating category entries and/or extending the LLR to other Domains. Through use and experience with the LLR, new classification entries may be discovered. For instance, a new Beneficiary may be identified or a new Phase in the Development Process may be introduced. In a similar vein, though a bit more complicated, the LLR may be extended into other domains (e.g., Marketing) which will have its own Work Products, Phases, Beneficiaries, etc.
5. Expanding the Scope of “Lessons Learned” While the LLR is valuable as a repository of Lessons derived from formal post project reviews, there is definite benefit to expanding participation beyond formal post project reviews. Engineers throughout the organization may access the LLR, not only as a source of information, but also as a repository of experiences to which they can contribute individually. This is yet another benefit that is derived from the LLR. Experiences may be added to the Repository without the need of a formal post project review. This yields more timely information since experiences can be added before the end of the project when post project reviews most often occur. On the other hand, this also introduces additional issues with respect to the administration of the Repository. Can Lessons be added at will? Should they be reviewed? Must a Lesson meet any particular “significance bar” before being added to the Repository?
6. Lessons Learned About Lessons Learned Repositories The LLR was implemented several years ago. We have made several observations from our experiences and analysis during this process. 1. We cannot anticipate every possible organizational structure - this is an artifact of
8 inconsistent vocabularies and/or levels of knowledge among engineers. We address this by providing multiple organizational views of the Lessons - for instance, we used Domain, Beneficiary, Phase, Work Product and Functional Category. Nevertheless, it is clear that we cannot anticipate in advance every Lesson which may be added, nor every way in which the Repository could be used. Therefore, the user must be given the opportunity to add a classifier that does not appear in the pre-defined lists. On the other hand, even if vocabularies or perspectives of contributors differ, we want related to Lessons to be found. For instance, Thus the repository must be able to map between terms, or at least offer the user alternate, but related terms when formulating a search.. 2. Lesson matches are rarely binary outcomes - there are plenty of gray areas in which a Lesson may “match somewhat”. Because of the potential to return so many matches we found that links to pages (rather than the content of the page itself) helps minimize the information overload that might occur were the entire contents of each page presented to the user. We address this by returning a short (1-2 line) summary of each Lesson that matched at least one classifier along with a score (percentage of matches hit) to help provide some guidance. A more sophisticated scoring algorithm may in fact reduce the user’s load by better filtering of which pages are matched. 3. Users will not utilize an empty Repository. While they obviously will not retrieve any Lessons from an empty Repository, they are also reluctant to contribute Lessons unless the Repository already contains some Lessons. Therefore, we found it necessary to “prime the pump” by extracting Lessons from prior Post Project Review reports, and adding them to the LLR. Other ways to help populate the repository, both manual as well as automatic should be explored.
7. References 1.
R. D. Anderson and B. Gilmore. “Maximizing Lessons-Learned - A Complete PostProject Review Process”, to appear in the Proceedings of the 1997 Pacific Northwest Software Quality Conference, Portland, Oregon.
