Experiences with Conducting Project Postmortems - IEEE Computer ...

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

Experiences with Conducting Project Postmortems: Reports vs. Stories and Practitioner Perspective Kevin C. Desouza Univ. of Illinois - Chicago, USA [email protected]

Torgeir Dingsøyr SINTEF, Norway [email protected]

Abstract The most popular unit of work in organizations is a project. Managing knowledge in and about projects is salient for successful project management. Explicit knowledge is easier to manage than tacit knowledge as it is an outcome of work. Tacit knowledge is abstract and is managed in a cursory mode in projects. In this paper, we will discuss how postmortems can be used to capture tacit experiences in projects. Conducting a postmortem, either after a milestone or at the end of a project, is salient in order to gauge what has been learnt, what were the main issues faced, and what can be used to improve the processes of work in the future. The conducting of postmortems aids in articulation of tacit experiences into explicit forms, this enables for experiences to be better reused in the future. Re-using of postmortem findings depends heavily on the nature of the postmortem outcome. We will compare two kinds of postmortem outcomes – traditional reports and stories. Management must choose the right kind of postmortem report to calibrate depending on the project and learning outcomes. We also highlight lessons learnt from conducting postmortem reviews in several software organizations.

1. Introduction Work in software organizations is conducted in projects [3]. Projects can be considered as organizations within the organization. Unlike projects in other domains, such as construction or architectural engineering efforts, software projects are highly knowledge based. Knowledge makes for the raw-material, the process, and the output of software engineering efforts. Failure to appropriately leverage knowledge within and around software projects could lead to disastrous project outcomes and economic losses for the sponsoring organization. Given the importance of managing projects in the software domain, it is striking and quite disturbing to find that optimal management of software projects still remains elusive. Software projects are constantly over-budget and behind schedule [23, 24, 29]. In addition, many software projects fail to deliver on their intended goals [1, 2, 25]. We posit these dismal findings can be traced to poor organizational learning mechanisms in software organizations [23, 29]. A manager at the Motorola Labs in Illinois, USA put it best, “we make mistakes, and we know these mistakes

Yukika Awazu The Engaged Enterprise, USA [email protected]

were done in the past, yet we constantly re-use mistakes”. Re-using knowledge is appropriate in software organizations [15, 17], however, we must re-use past knowledge to improve future operations not to repeat past mistakes. Re-using knowledge calls for an apt organizational learning procedure to be in place. The organization must be able to draw on its extant knowledge resources to inform future behaviors. Software projects by nature are knowledge intensive [17, 21, 38]. Each member of the software project team is an expert in his/her domain. The success of the project depends on how well these experts integrate their “knowledge” to attain the goals of the project. Software projects rarely have any slack resources, as such there is minimal time for project members to engage in interactions, relationship building, and socialization, all of which can add to better knowledge sharing if conducted [2]. The issues of outsourcing (both offshore and onshore) complicate software projects further, as now we must account for differences in culture, language, time barriers, etc [35]. Learning in and around software projects is not an option, it is a must for organizational survival. We cannot afford to constantly re-use our past mistakes, it is just too expensive. In order to prevent repeating mistakes, we must pay attention to the process of software projects. We must learn from the past processes and seek ways to improve them. The process is tacit, and in the conduct of projects, we engage this tacit [34]. By tacit knowledge we mean knowledge that a human is unable to express, but is guiding the behavior of humans [32]. In order to foster organizational learning, these tacit insights need to be captured in an explicit format so that they can be re-used with ease in the future. Tacit insights, in this context, can be likened to project management lessons. In this paper we will explore the concept of postmortems as a means to elicit tacit insights in the context of project management. Conducting a postmortem, either after a milestone or at the end of a project, is salient in order to gauge what has been learnt, what were the main issues faced, and what can be used to improve the processes of work in the future. The conducting of postmortems aids in articulation of tacit experiences into explicit forms, this enables for experiences to be better re-used in the future. Postmortem analysis is only of value, if insights are engaged to guide future project management efforts. We assert that re-using

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

1

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

of postmortems findings depends heavily on the nature of the postmortem report; to this end we will compare and contrast two kinds of postmortems – traditional reports and stories. These two reports differ on a number of dimensions such as structure, context, specificity, etc, and are among the two most popular formats found in software organizations. Both types have their pros and cons, and management must choose the right kind of postmortem report depending on the project and learning outcomes. The paper will also highlight lessons learnt from conducting postmortem reviews in several software organizations. This paper makes the following contributions to the literatures of project management, knowledge management, software engineering, and management of information systems: • We discuss the significance of postmortems as an aid to capturing tacit knowledge in and around projects. • To the best of our knowledge, ours is the first paper to explicate the two kinds of postmortems reports, factors that affect their re-use in the future, and also management considerations when choosing which report to commission after a project. • We highlight several practitioner recommendations on how to better conduct postmortem analysis in organizations. The rest of the paper is organized as follows. In the next section we will describe the concept of post-mortems and the intricacies of conducting postmortems. We will draw on our experience of working with several software organizations in conducting postmortems to inform the material presented. Next, we summarize two case studies conducted to highlight the two kinds of postmortem reports. In the section that follows, we will compare the two kinds of postmortems reports. Concluding the paper we discuss lessons learnt from us conducting project postmortems, recommendations for software companies, and directions for future research on postmortems.

2. Postmortems George Santayana, put it succinctly, “Those who cannot remember the past are contemned to repeat it” [31]. Postmortems are mechanisms to remember and recall what transpired during a project and use these lessons to inform future behavior and actions. Postmortem is a collective learning activity which can be organized for projects either when they end a phase or are terminated [7]. The main motivation is to reflect on what happened in the project in order to improve future practice – for the individuals that have participated in the project and for the organization as a whole. The intention of a postmortem should be learning, and not project evaluation. Evaluation can lead to people not sharing experiences that they think can embarrass them. Learning

through postmortems must occur at three levels – the individual, the team, and the organization. Individuals i.e. project members, involved must learn from their conduct during the project. This learning will call for taking a retrospective look at how they performed key tasks, the difficulties they faced, barriers they had to overcome, mechanisms that were helpful for goal attainments, and other details. The postmortem affords the opportunity for individuals to receive structured feedback on their performance on the project. At the individual level, postmortems also help to identify areas of managerial or technical competencies that need to be developed in order to be more successful in conducting future projects. For instance, an individual may realize that during the project his knowledge in Enterprise JAVA Beans may not have been up-to-par that resulted in him slowing down the project. The individual could take this insight to improve his skills to prepare for future work assignments. Similarly, individuals could learn that they need to improve their managerial skills such as oral and written communications, project management skills, etc. For the project managers, postmortems allow for reflection of management approaches, styles, conventions, and their associated effectiveness in attaining project targets. For the team, a postmortem provides a venue to discuss how the collaboration and coordination were organized. Seldom does a project complete without conflicts among group members. Conflicts do not have to be major incidents; they can be disagreements over work practices, methodologies, output deliveries, etc. During the course of our research we found that most commonly team members disagreed on the amount of testing required, the scope of re-use, process and methodology issues, and communication issues. It is important that once a project is completed, team members reflect on these situations so that lessons can be drawn and mistakes can be avoided in the future. For the organization, a postmortem allows for the tacit insights from a project to be captured and made available to the rest of the organizational members. A postmortem provides an opportunity to reflect on company-wide policies, processes strategies and models. In order to be effective for organizational-wide learning, lessons learnt will have to be de-contextualized from the project-at-hand and be made applicable to the activities the organization engages in. At the organizational level, we are concerned with eliciting knowledge from projects that can be used across current projects underway in the organization and for future projects [14, 18]. A postmortem analysis only has value if the lessons learnt (knowledge) is used to inform future practices of project management. In a later section of the paper, we will focus on two kinds of postmortems reports that are used to promote organizational learning – reports and stories.

2.1. Conducting Postmortems

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

2

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

There are several ways to perform postmortem reviews. Some companies use a simple checklist in a meeting, while others use half-day workshops, three day meetings away from the normal work area and some use a combination of methods [7, 26, 39]. Some do not organize a postmortem meeting, but gather data through interviews. Apple Computer has used a method which includes designing a project survey, collecting objective project information, conducting a debriefing meeting, a “project history day” and finally publishing the results [10]. At Microsoft much effort is put into writing “Postmortem reports”. These contain discussion on “what worked well in the last project, what did not work well, and what the group should do to improve in the next project” [11, 12]. It is important for an organization to have a welldefined process in place when conducting postmortems [10]. While postmortems can differ in the range, scope, and depth, based on the characteristics of the organization and the project, most postmortems follow a basic process - project survey, collecting objective project information, conducting a debriefing meeting, project history day, and publishing the results [7, 10, 19]. The project survey is used to gather subjective data on how the project was conducted; this is followed by gathering of objective data – costs, time to completion, defects, etc. The debriefing meeting and project history days help in eliciting tacit insights, engaging in dialogues, and extracting lessons learnt, the culmination of the postmortem is the publication of the postmortem report. It is important to consider the temporal aspect when conducting postmortems. Postmortems can either be conducted at the termination of the project or the completion of a milestone. For small projects and noncomplex endeavors it may suffice to have a postmortem upon project termination. For large-scale projects it is advisable to conduct postmortems after the completion of milestones. In this way, lessons learnt can be incorporated to improve the processes for the remainder of the project and also improve future efforts in other projects. Tiedeman [39] suggests three types of postmortems, related to the waterfall model of software development, one for “planning”, one for “design/verification” and one “field postmortem” to provide feedback after the developed system has been in use for some time.

2.2. Postmortem Participants Participation is tied to the intended goals of the project postmortem. If the goal is to foster individual learning, the postmortem may be restricted to the members of the project team, probably with a facilitator. The role of the external facilitator is to structure discussions and lead different exercises to stimulate reflection. For team learning, it is important to have an external facilitator to moderate the discussions among team members and also keep the project manager in check. In our experience, it is

common to have postmortem analysis turn into disasters without a facilitator. Without a facilitator, the leader of the postmortem discussion becomes the project manager, who is biased and will not take gently to criticisms and may show favoritism to individual project members. The success of a project postmortem is heavily tied to how open participants are to acts of perspective taking and perspective making [8, 33]. The use of an external facilitator can greatly contribute to open discussions, as the members perceive the external facilitator to be unbiased, neutral and with no hidden agenda. If the intention is to foster organizational learning, it is possible to either invite people from other similar projects that are starting up, or people who have an organizational-wide role such as chief knowledge officers or people from the quality department, which is a more common role in software companies. Postmortem participation will also depend on the nature of the project. For novel projects it is advisable to involve members from outside the project so that lessons learnt can be easily communicated and diffused throughout the organization. Daft and Lengel [13] point out that information and knowledge that is high in equivocality should be shared via face-to-face contact to reduce ambiguity and enhance communication. If a postmortem is being conducted for a fairly routine project it may suffice to share the documented lessons learnt with the rest of the organization as the members are familiar with the kind of project, the context of activities, and also the problems faced in the past. Reading the documented report can help them gauge what is new and what can be implemented in the future.

2.3. Outcomes of Postmortems Outcomes of postmortems can vary in their length, scope, and depth. We have seen postmortem reports that range from 3 pages to over 100 pages. Some postmortem reports are written by individual team members, some are written jointly by teams, and even others are authored by the project manager or the external facilitator, depending on the nature of the project. If the main purpose of the postmortem is to foster individual learning it is best to have the individuals prepare personalized reports. This allows for the members to reflect and present their findings untainted from that of other members of the group. The amalgamation of individual reports can make up the postmortem report. If the aim is to foster team learning, it is best to have a jointly co-authored report. A natural outcome of jointly working on the report is engaging in debates, socialization, joint reflection, and controversies. These subtleties help to add to the richness of the report and also aid in mutual understanding of the cause/effect for critical issues. It is important to give the team ample time to write the report, and also budget the time required into their work assignments. Failure to do so could affect the

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

3

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

outcome of the postmortem report. If the project is routine and non-novel, it may be sufficient to have the project manager write the report. The project manager can elicit lessons learnt from discussions with the group and compile it into a report. Project managers have the advantage of taking a macro look at the project, which helps in identifying critical issues. These critical issues can be compared with past project experiences when creating the postmortem report. It is also conceivable to have the external facilitator prepare the postmortem report. This normally occurs, when the project is “heated” in the sense that it is difficult for group members to arrive at consensus during the project review and also when there is a clear need for an independent examination of the project performance. Lessons learnt must be taken outside the context of the specific project that led to their generation and put in a context that can be understood by the organization as a whole, even by members who were not part of the project team. There are two main kinds of postmortem reports. The first is a traditional report. A traditional report, as the name implies is like any other report, it is a structured presentation of lessons learnt during a project. Reports are built using pre-defined organizational templates, i.e. for every project a report has the same layout with the difference being in the content. A typical template for a report includes an introduction with information about the project and how the postmortem was conducted, lists of issues that went well in the project as well as analysis of the most important issues, lists of issues that did not go well with analysis, and finally appendixes with full listings of topics that appeared in the postmortem and possibly transcripts of discussions. The second is a narrative or a story. Denning [16] defines a narrative as, “anything told or recounted; more narrowly, something told or recounted in the form of a causally-linked set of events; account; tale,: the telling of a happening or connected series of happenings, whether true or fictitious”. Through the use of narratives we can develop learning histories. Learning histories is a method to use narratives to transfer experience, taking into account the importance of storytelling and myths in organizations. Storytelling in organizations has been applied to a number of areas in the management and organizational sciences [16, 22, 30]. A learning history is created by interviewing participants in an individual project, and then working on the material to create a story, following some basic principles [36, 37]. A story starts with a curtain-riser, a kernel paragraph, an exposition containing the main points of the story. The main story is presented as a jointly told tale, with narrator’s voice in one column and direct voice in another column. Usually, the main story is organized in thematic chapters with their own kernel paragraph and exposition. The story finishes with a closing up, which connects to the curtain-riser.

We will now describe two case studies, one that used a traditional report and one that used a story to document the findings of the postmortem analysis; following this we will compare the two types of postmortem outcomes.

3. Two Cases Studies We have chosen to describe only two case studies due to space limitations. The first case study we present was used to foster team learning and the second one was meant to be used in the broader context of organizational learning. Our goal in conducting postmortems was to analyze the nature of postmortems, their effectiveness, critical impediments that affected their success, and also what factors affected incorporating of lessons learnt in future endeavors. Our involvement with the organizations was significant, and for the most part we adhered to the guidelines of action research [9, 20].1 Action research, also known as collaborative inquiry, involves collaboration between the researcher and research subjects in a cyclical manner in order to build learning and knowledge about how to change a given social system [4, 5]. The field of software engineering is apt for incorporating the action research methodology as it is an applied field, and as the role of IS researchers and consultants is to support the organization being studied [6]. One of the most salient benefits of action research is that it results in producing relevant research findings, which as Baskerville [6] notes is an critical measure of the significance of IS research.

3.1. Project Norway This postmortem was done on a project to develop a web-based ticket ordering system for a major transport company in Norway. The project was critical for the transport company, as it introduced fundamental changes to their revenue management process. The project team at the end of the project consisted of eight people, who all took part in the postmortem meeting (the project had involved three more people earlier, but they were removed from the project because of budget issues). The company that was running the software project is a large software house with approximately 500 employees. The postmortem analysis followed the approach described in [7] except for starting with doing a timelineexercise [26] as the project had lasted for almost two years. All participants were asked to remember key events, and write down the names of the events on stickers and attach them to grey paper on a wall, rectangular stickers for events and round stickers for 1

Action research was the primary research method used in the two cases described in this paper. With other postmortem analysis projects we also followed case study and structured interview methodologies for data collection.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

4

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

dates. Important events in this project were tasks like: choosing a platform, deciding on coding standard, choosing the database, intense work period, etc. The timeline produced is shown in figure 1. Participants were asked to write down up to four positive and negative experiences they faced during the project. These notes were then put on a whiteboard and grouped into categories or themes. Issues that went well were: teamspirit, competence development, human competence, will and ability to solve problems, customer responsibility, good products and improved customer relation. Issues that were problematic were: testing, technical investments, lack of knowledge, and immature technology.

ownership of solutions and that it was easy to have a good overview of the group. Similarly, we found the following reason for problems with testing: lack of automated tests, difference in development and production environment, test process was not followed, and testing did not measure the right features. Upon completion of the postmortem analysis, two facilitators wrote an 18-page report. The report began with an introduction which provided the background of the project and the purpose of the postmortem analysis. This was followed by providing an overview of the activities conducted in the postmortem meetings. Next, the results of the analysis were described. The results section enumerated the critical issues, both positive and negative, and put them in rank order, and also contained transcripts of the postmortem meeting. Mind Maps was to document root causes for the main issues (see figure 2).

Figure 1: Timeline for the Project We will now analyze two of the issues that went well and two that did not work out well more in detail.2 We will show excerpts of what people said about the issues: Team-spirit: “If you look at the person gallery who were involved in this project, you see that we are very different, but are anyway able to work well together. I think that has been unbelievable, I see so many other places that this does not work”, “I would also like to emphasize that it has been very nice socially in the project, although there have not been much after working hours ... professionally there have been people whom you could ask all the way, people have not had enough with their own problems”. Testing: “The greatest mistake we did is that we said ‘no’ to more load testing before we went to production”, “I think we ought to have done more automatic testing earlier, and should have done load testing earlier. We also should have had a better understanding of what load testing means – we have at least two different views of it”. Through the conducting of data collection and analysis activities, root-cause analysis was performed. Ishikawa diagrams, also known as fishbone diagrams, were sketched to determine the contributing factors for critical issues [7]. In a root-cause analysis, main causes for teamspirit were found to be good mix of people, solutionoriented people, co-location of the project team, 2

Due to space constraints we are not presenting details on all issues, more information on the projects can be received by contacting the authors.

Figure 2: Mind Map Showing Reasons for the Issue of “Competence Development” The appendices contained a list of participants in the project (and postmortem) and a timeline showing important events, list of 37 keywords from brainstorming on “what went well” and 35 keywords for issues that “did not work well”, transcript of presentations of issues that “went well” and “did not work out well”, and references to material about how to organize postmortem reviews.

3.2. Project Misunderstanding This postmortem was conducted with a medium-sized IS consulting organization based in Illinois. The organization has 300 employees, based in six global locations. The organization had as its goal to achieve CMM level 4 certification; to this end they wanted to systematize their postmortem analysis efforts. The focus was to create postmortems to aid in the generation of lessons learnt that could be incorporated enterprise-wide. The organization chose to use narratives as a means to write lessons learnt in the form of stories. The story was written in hypertext, for ease of reading and was made available on the organization’s Intranet. PROJECT MISUNDERSTANDING involved 20 software engineers, 5 systems analyst, 2 quality assurance personnel, and one project manager. The goal of the project was to build a web-interface for a backend system and work on inter-operability issues between two legacy systems. The project lasted for eight months and involved

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

5

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

interaction between staff members at two locations, one in the United States of America and the other in India. Upon completion of the project it was clear that the project had peculiarities which warranted the need to document it in the form of a story. The project was one of the first efforts the organization had undertook in which the offshore staff had control over design as well as coding of the interface, in the past it was customary for the US office to control design and the offshore staff to work on coding. Another peculiarity of the project was the fact that five of the software engineers in India were working on their first project, hence they had not been exposed to the organization’s culture, work practices, etc. The project, as can be deduced from the name, was laden with misunderstandings; these comprised of misunderstanding requirements, communication misunderstanding, and scheduling misunderstanding. The organization wanted project managers to learn from this experience to better manage future endeavors. The narrative written was comprised of three main chapters, which highlighted the central themes of the story – requirements misunderstanding, communication misunderstanding, and scheduling misunderstanding. In addition, there was one introductory chapter and two concluding chapters, along with several appendices. The introductory chapter set the stage of the story, by introducing the various characters, the project, and other contextual factors such as budgetary details, delivery dates, workload of the characters, etc. The two concluding chapters were titled – “the positives and the negatives” and “reflections”, these represented the lessons learnt and food-for-thought for reflection purposes. The appendices contained reflections of the original team members to the story, notes on how might future projects gain from the lessons, and a definition of terms. The story can be best characterized as focusing on how the various characters acted over time, all of them with the right spirit, but their intentions were misconstrued due to lack of common understanding. While it is difficult to present experts of the story in context, we would like to point out some interesting misunderstandings:3 Since the design and development was handled offshore, the US based work group had the responsibility for requirements gathering, final testing, and delivery. Over 20 critical misunderstandings occurred between the phases of requirements gathering and design, much of these could be attributed to the poor standards of documentation, using of localized jargons, and making assumptions on what knowledge was possessed by the team in India. The initial meeting between the staff based in the US and India never occurred. The reason was simple – no one 3

Interested readers can contact the authors for an abridged version of the story.

specified the reference time. An excerpt of the e-mail from the US project manager, “let us schedule a meeting for 4 – OK”, the response from the team leader in India, “sure…see you then”. Never was it clearly specified 4 AM or PM and what time zone. During the postmortem, the root-cause analysis for this problem was established to be no clear agreement on communication conventions. One of the takeaways from the project postmortem was to clearly post communication conventions such as the use of English or Metric system, 2400 hrs or 12.00, reference time zone, etc Misunderstanding occurred also when interpretations needed to be made regarding sensitivity of issues. The two teams had problems signifying items such as urgency, need for punctuality, explanation facilities, and responses to e-mails. For example, if the Indian developer wanted clarification on an item he/she would almost never use the word “URGENT” but would frame the e-mail as “I would appreciate it if you could answer this at your earliest convenience”, to the American, “earliest convenience” meant that it was not of urgent nature and a delayed response would suffice. The organization brought in a professional writer who played the role of a tool, to document the story. The writer interviewed project participants, triangulated the findings, documented it, and sought clarifications as necessary. Currently, the story is being used to sensitize employees to issues of communication, management, and interaction in a global world.

4. Reports vs. Stories Postmortem analysis can be documented as reports or stories. The choice of whether to use a report or a story is far from being trivial. Table 1 highlights the five salient dimensions on which reports and stories differ. Table 1: Reports versus Stories Reports Stories Knowledge Highly Structured SemiStructure Structured Codification Cost Low High Knowledge Low High Richness Ease of Easy Medium Comprehension Ease of Difficult - Medium Easy Recallability Reports are highly structured in their presentation as they follow pre-defined formats, as such they are easy to prepare. Stories, on the other hand, are semi-structured at best. Stories are similar in their general form, each story begins with an introduction (setting the stage), followed by the main plot (the critical phases of the project), and the conclusion (lessons learnt). Besides these high-level similarities, stories can vary in forms, plots, depth, characters, etc. Stories are not created using cookie-cutter

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

6

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

approaches, each story is unique and hence the cost involved in writing a story is high. The structured nature of reports leads to the perception of objectivity, whereas stories are have a subjective and emotion ring to them. Reports contain knowledge of low richness in comparison to stories, this is a function of the way stories and reports are written. In PROJECT NORWAY, the excerpts on team spirit and testing provide us insights on the high-level lessons learnt; however, they do not provide us with the necessary context surrounding the events. Stories are apt to capture context of the project. Stories are more detailed and can hence offer richer knowledge as one has the ability to incorporate contextual details into the plot of the story. Reports are easy to comprehend. Postmortem reports, if stored electronically, can be accessed with ease using standard search and retrieval technologies, for e.g. using a keyword search. Reports are structured and hence reading one report is very similar to reading another. Stories are more difficult to comprehend. For one, they are larger in length and require more cognitive resources of the reader. Moreover, due to the lack of structure and directness, a story can be analyzed in multiple ways; this equivocality leads to multiple interpretations between individuals, an individual can also have multiple interpretations of a story after reading it two or more times. A report will tell you exactly the key take-a-ways, stories leave it up to the reader to derive their own lessons learnt. Stories are not as amenable to the use of automated technologies as reports, for instance, it is difficult to run automated searches on stories as one may not know the context to search on. Findings of reports are less easy to recall when compared to stories. Stories are dramatic and hence make lasting impressions on memories. Due to the homogeneous nature of reports it is hard to recall the specificities of a given report. For example, in PROJECT MISUNDERSTANDING, on reading about the failure of the first meeting to occur due to misunderstanding based on poor communication of reference time, it is very likely that the event resonated with the reader and will make a lasting impression. On the other hand, if it was documented in a report it would be in the form of a bullet point that would not make a sufficient impact. What factors determine whether a postmortem analysis be written as a report or a story? Through our work with a number of software organizations on postmortems, we feel the following critical guidelines must be adhered to. Nature of the project – if the project is unique and significant peculiarities have transpired during its life it is best to capture the postmortem analysis as a story, else a report will suffice. If an organization has a history of working with a specific kind of project for e.g. building web interfaces, it will be a good idea to capture lessons learnt from all such projects as a report. This will enable for building traceability, we will be able to track if lessons reports on a previous project were taken into account in a

future iteration. Moreover, we will be able to track lessons learnt and see if there are patterns that emerge (we will discuss this in the next section). If the organization is conducting a web-building interface for the first time, a story would be ideal as it could be used to ground ideal behavior in all future projects. Cost – writing a story is more costly and time consuming effort, hence the effort must be warranted by expected benefits i.e. the nature of the project. If resources are scarce, it is better to write the postmortem analysis as a report due to the low cost, trying to write a story with an insufficient budget will be a futile effort. Impact – Stories make more organizational impact than reports. Stories lead to organizational wide discussions, debates, and interpretations. Hence, they are apt for capturing lessons of a significant magnitude, reports on the other hand are more simple and humble, and they are apt for capturing lessons learnt on routine endeavors. A point to note, choosing when to write a story is very important so as not to fall prey to the “Cry-Wolf” syndrome i.e. calling attention to non-significant stories. If an organization writes a story for every small novelty, stories will lose their significance in the organization. Lesson being conveyed – stories are an excellent means to communicate norms, core beliefs, and values of the organization, they are less suitable for conveying rules or policies, these are more easily conveyed using reports. Stories are good at driving home moral lessons whereas reports are better suited for communicating operational lessons learnt directly and succinctly. The above guidelines will enable managers to better choose the output of postmortem analyses. Unless the right postmortem output is chosen, management will not attain desired learning outcomes. Reports are best used to build on past efforts and can help in incremental learning efforts, stories, on the other hand, are used to attain radical learning outcomes as they are dramatic and call for unlearning of previous behaviors. The challenge for the organization is to find the right mix of postmortem reports and stories so as to improve the practice of project management.

5. Postmortem Lessons Learned We will now discuss other lessons learnt while conducting project postmortems. The lessons learnt are aimed to further both the practice and research aspects of project management, software engineering, management of information systems, and knowledge management.

5.1. The Environment Creating a suitable environment for reflection, dialogue, criticisms, and interaction is salient to the conducting of a postmortem. In order to capture tacit knowledge in a postmortem review, it is necessary to create an arena where people can reflect openly on both problems and successes. The environment must be conducive to supporting the activities of socialization, combination,

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

7

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

externalization, and internalization [32] that are required to have a vibrant knowledge creating cycle. Crating an environment to support socialization requires openness and respect among the participants. Participants need to feel comfortable to share their experiences, views, and insights, by the same token they should respect their rights of their peers to do the same and refrain from critiquing or dominating the discussion. To promote combination, it is important to have the right integrators in place. Integrators can be electronic or human. In PROJECT NORWAY, the whiteboard and wall served as integrators. Participants captured their experiences using post-it-notes on these surfaces, during the course of the project review and analysis these experiences were grouped into themes and categories that enabled for sensemaking. In PROJECT MISUNDERSTANDING, the storywriter served as the integrator. Through discussions and interviews with project participants the writer was enable to combine experiences and insights to build the story. As previously discussed, externalization can be in the form of a report or a story. Internalization calls for reusing postmortem analyses to inform future management behavior. It is important for the senior management to make consulting postmortem analyses before planning a new project a requirement in organizations. Taken together the acts of the knowledge creating cycle, can lead to an effective postmortems and a budding learning system that improves the practice of project management.

5.2. Interpretation Management In PROJECT NORWAY, agreeing on team-spirit and testing as issues that went well and not so well means that these issues are more likely to be handled differently in future projects. Probably, many of the participants had a feeling that testing was problematic before the postmortem meeting took place. But after the meeting the participants were able to agree on a set of reasons for why this was problematic. These reasons given in the postmortem report might not be the “true” reasons, but it is the start of a reflection process – which probably will lead to another way of thinking about testing activities in future projects. It is important to note, that postmortem outcomes should not be looked at as the truth, and neither should interactions among participants be focused on eliciting the truth. We must understand that participants are by nature biased and they can only report on their version of reality. As such, interpretation of the subjects between participants should be managed appropriately so as to prevent one version of reality dominating the conversation. Postmortems are best viewed as an integrated collection of realties on lessons learnt.

5.3. The Time Factor Postmortem reviews provide an opportunity to reflect, learn and capture knowledge for the future. In our experience, most software engineers do not have the time

to “finish” a project before they are thrown into the next one. This is rather unfortunate as it prevents one to pause and reflect. The chances of making errors are high when postmortems are not conducted [27]. It is crucial to have sufficient time for people to engage in a fruitful reflection and analysis process. By using different thinking aids and approaching the project in different manners, new aspects of the project appears this leads to new findings and emergence of knowledge. In the method described in the first case study, significant amount of time was devoted to analyzing the two main issues that went well, and the two main issues that were problematic. Dialogues were conducted until participants were satisfied with the main and sub-causes of the critical issues. Similarly, writing of narratives takes considerable time and effort. Time spent on postmortems should not be looked upon as a cost but as an opportunity – an opportunity to learn. The opportunity costs of not conducting a postmortem could be high if we continuously repeat mistakes and perform ineffectively. Resources should be allocated to conduct a postmortem after a project ends and to review existing postmortems analyses before embarking on a new project. It is also advisable to consult existing postmortems when a project faces critical issues that are difficult to manage.

5.4. Cost/Benefit Analysis Postmortems are expensive to conduct; they call for an investment of time, people, and resources. As such large organizations cannot afford to conduct extensive postmortems after every project. We suggest that an organization consider a cost/benefit approach to guide postmortem decisions. If an organization is at the initial phases of conducting software development efforts, postmortems are warranted for all beginning projects. There is much to be learnt as an organization has no history to rely on and must use the lessons learnt for improving project management. In an experienced software engineering organization, postmortems are not necessary for every project. Novel, highly complex, and projects dealing with new territories such as a new programming language, systems methodology, etc will require postmortems; here the gains associated with learning are significant to warrant the cost. Using the cost/benefit approach wisely will help an organization focus its resources so as to get the maximum return out of postmortem efforts. It is important to conduct a postmortem on projects that differ on critical dimensions such as team composition, team size, complexity, length, location of project members, etc; only then can each additional postmortem add to our understanding of how to improve the current project management practices in the organization. Diversity and richness of lessons learnt is always better than the sheer volume of lessons; it does the organization no good to have the same lesson be learnt multiple times and be documented over and over again.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

8

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

5.5. Patterns in Postmortems Postmortems are knowledge objects in their rudimentary sense, and they need to be mined to detect patterns in behavior. It is hence, important for an organization to have a structured collection of postmortems so as to enable for easy analysis. The organization should conduct semi-annual or annual evaluation of postmortem reports to extract macro lessons. These lessons can help make strategic changes to how an organization conducts its project management effort. For example, an organization might be able to approximate what is an ideal team size to avoid communication problems. An organization may be able to learn how much co-location is needed for successful project completion. It is important to build a history of lessons learnt, as lessons that need to be learnt several times over, are not being learnt effectively. For example, if all projects suggest that more slack time needs to be accounted for during the testing phase of product development, and this has never been implemented in future projects, there is a fundamental management problem here which needs to be addressed. Patterns help in re-thinking existing behaviors and changing the governing principles on which the organization operates.

6. Conclusions To conduct postmortem reviews is an easy and efficient way to collect and distribute company-relevant knowledge. In addition to conducting postmortems, software organizations should also get in the habit of conducting post-implementation audits [28]. These audits should aim to evaluate both hard and soft benefits of the new technology or system that was developed. Conducting of post-implementation audits, can complement the lessons learnt from postmortems. Most organizations make the mistake of avoiding postimplementation audits, just like they avoid post mortems. Key reasons being that they consider such efforts too costly in terms of time and resources. While it may not be economically feasible to conduct an audit of every single project, it is always wise to choose projects that are novel, high risks, and also those that call for drastic changes to traditional practices for audits. Only when we conduct an audit will we be able to learn from our mistakes. This paper has examined the concept of postmortems as a viable method for capturing tacit insights from projects. We have elaborated on two types of postmortem analyses outcomes – reports and stories. The paper has also explicated how to decide between writing a report or story for a given project, and has suggested practitioner insights to improve the calibration of postmortems. While the focus of the paper has been on software projects, our findings are generalizable to other kinds of projects. Postmortems if done properly can help an organization come to grips with conducting effective and efficient

project management. Postmortems must be woven into the fabric of current project management practices. Future research can examine the governing dynamics of postmortems. Specifically, we hope to see a study that looks at the economics of postmortems, i.e. exploring in detail the cost/benefit issue. Research can also examine how the different types of postmortems affect the incorporation of lessons learnt in future projects – are stories more successful than a traditional report? Research can also look into linking postmortems to future project performance, i.e. do conducting postmortems help lower the risks associated with future projects and improve the chances for successful project completion, currently there is plenty of anecdotal evidence but a rigorous empirical investigation is lacking. In conclusion, we hope this paper has opened an avenue for interesting discussions regarding the viability of postmortems in capturing tacit knowledge from software projects.

7. References [1] Abdel-Hamid, T.K. and Madnick, S.E. (1990). “The Elusive Silver Lining: How We Fail to Learn from Software Development Failures”, Sloan Management Review, 32 (1), 3948. [2] Awazu, Y., Desouza, K.C., and Evaristo, J.R. (2004). “Stopping Runaway Information Technology Projects,” Business Horizons, 47 (1), 73-80. [3] Basili, V.R., Caldiera, G. and Rombach, H.D. (1994). “The Experience Factory,” in Encyclopedia of Software Engineering, Vol. 1, J. J. Marciniak, (ed.), John Wiley, New York, 469-476. [4] Baskerville, R., and Wood-Harper, A. T. (1996). “A Critical Perspective on Action Research as a Method for Information Systems Research,” Journal of Information Technology, 11 (3), 235-246. [5] Baskerville, R. (1999). “Investigating Information Systems with Action Research,” Communications of The Association for Information Systems, 2, Article 19. [6] Baskerville, R. (2001). “Conducting Action Research: High Risk and High Reward in Theory and Practice,” in Qualitative Research in Information Systems, E. Trauth (ed.), Idea Group Publishing, Hershey, PA, 192-218. [7] Birk, A., Dingsøyr, T. and Stålhane, T. (2002). “Postmortem: Never Leave A Project without It,” IEEE Software, 3 (19), 43 – 45. [8] Boland, J.R. and Tenkasi, R. (1995). “Perspective Making and Perspective Taking in Communities of Knowing,” Organization Science, 6 (4), 350-372. [9] Brydon-Miller, M., Greenwood, D., and Maguire, P. (2003). “Why Action Research?” Action Research, 1 (1), 9-28.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

9

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

[10] Collier, B., DeMarco, T. and Fearey, P. (1996). “A Defined Process for Project Post Mortem Review,” IEEE Software, 4 (13), 65-72. [11] Cusumano, M.A. and Selby, R.W. (1995). Microsoft Secrets - How the World's Most Powerful Software Company Creates Technology, Shapes Markets, and Manages People, The Free Press, New York, 1995. [12] Cusumano, M.A. (2004). The Business of Software, The Free Press, New York. [13] Daft, R.L. and Lengel, R.H. (1986). Organizational Information Requirements, Media Richness and Structural Design,” Management Science, 1986, 32 (5), 554-571. [14] Damm, D. and Schindler, M., “Security Issues of A Knowledge Medium for Distributed Project Work,” International Journal of Project Management, 2002, 20, 37-47. [15] Davenport, T.H., Thomas, R.J., and Desouza, K.C. (2003). “Reusing Intellectual Assets,” Industrial Management, 45 (3), 12-17. [16] Denning, S. (2000). The Springboard: How Storytelling Ignites Action in Knowledge-Era Organizations, Butterworth Heinemann, Boston, Massachusetts. [17] Desouza, K.C. (2003). “Barriers to Effective Use of Knowledge Management Systems in Software Engineering,” Communications of the ACM, 46 (1), 99-101. [18] Desouza, K. C., and Evaristo, J. R. (2004). “Managing Knowledge in Distributed Projects,” Communications of the ACM, 47 (4), 87-91 [19] Dingsøyr, T., Moe, N.B. and Nytrø, Ø. (2001). “Augmenting Experience Reports with Lightweight Postmortem Reviews,” in Third International Conference on Product Focused Software Process Improvement, Lecture Notes in Computer Science, vol. 2188, F. Bomarius and S. Komi-Sirviö, (eds.), Springer Verlag, Kaiserslautern, Germany, 167 - 181. [20] Greenwood, D. J. and Levin, M. (1998). Introduction to Action Research, Sage Publications, California. [21] Huang, J.C. and Newell, S. (2003). “Knowledge Integration Processes and Dynamics within the Context of Cross-Functional Projects,” International Journal of Project Management, 21 (3), 167-176. [22] Kaye, B., and Jacobson, B. (1999). “True tales and tall tales - The power of organizational storytelling.” Training and Development, 53 (3), 44-52 [23] Keil, M., Mixon, R., Saarinen, T., and Tuunainen, V. (1995). “Understanding Runaway IT Projects: Results from an International Research Program Based on Escalation Theory,” Journal of Management Information Systems, 11 (3), 67-97. [24] Keil, M., and Flatto, J. (1999). “Information Systems Project Escalation: A Reinterpretation Based on Options

Theory,” Accounting, Management Technologies, 9 (2), 115-139.

and

Information

[25] Keil, M., and Robey, D. (2001). “Blowing the Whistle on Troubled Software Projects,” Communications of the ACM, 44 (4), 87-93. [26] Kerth, N. (2001). Project Retrospectives: A Handbook for Team Reviews, Dorset House, New York. [27] Kumar, K. (1990). “Post Implementation Evaluation of Computer-Based Information Systems: Current Practices,” Communication of the ACM, 33 (2), 203-212. [28] Levinson, M. (2003). “It ain’t Over…Until You Do the Post-Implementation Audit,” CIO, October 1, 2003, 74-81. [29] Lyytinen, K. and Robey, D. (1999). “Learning Failure in Information Systems Development,” Information Systems Journal, 9 (2), 85-101. [30] Mitroff, I., and Kilmann, R. H. (1975). “Stories Managers Tell: A New Tool for Organizational Problem Solving.” Management Review, July, 18-28 [31] Munson, T.N. (1962). The Essential Wisdom of George Santayana, Columbia University Press, New York. [32] Nonaka, I. (1991). “Knowledge-Creating Company,” Harvard Business Review, 69 (6), 96-104. [33] Nonaka, I. and Toyama, R. (2003). “Knowledge-Creating Theory Revisited,” Knowledge Management Research and Practice, 1 (1), 2-10. [34] Polanyi, M. (1967). The Tacit Dimension, Anchor Books, New York. [35] Power, M., Bonifazi, C., and Desouza, K.C. (2004). “Ten Outsourcing Traps to Avoid,” Journal of Business Strategy, 25 (2), 37-42. [36] Roth, G. and Kleiner, A. (1999). Car Launch, The Human Side of Managing Change, Oxford University Press, New York. [37] Røyrvik, E. and Bygdås, A.L. (2004). “Knowledge Hyperstories and Context-Sensitive Knowledge Enabling,” in Living Knowledge, The dynamics of professional service work, K. Carlsen and G. von Krogh, Palgrave Macmillan, UK. [38] Rus I. and Lindvall M. (2002). “Knowledge Management in Software Engineering,” IEEE Software, 19 (3), 26-38. [39] Tiedeman, M.J. (1990). “Post-Mortems – Methodology and Experiences,” IEEE Journal of on Selected Areas in Communications, 8 (2), 176-180.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

10