Developing and evaluating the requirements for a blended learning lesson in project risk management. Bassam A. Hussein. Dr. ing. Associate Professor Norwegian University of Science and Technology, NTNU. Trondheim, Norway
[email protected] Abstract The main goal of the paper is to develop a project risk management lesson that is capable to take into account practical challenges that project managers have to deal with during managing project risks. Interviews with experienced project managers in project risk managers were conducted. The list of challenges and associated tactics to deal was then mapped into 10 requirements representing the intended learning outcomes of the lesson. The requirements were then mapped onto the design using 4 instructional methods; briefing lecture, team-based assignment, a computer simulation and a debriefing lecture. All these methods are linked by a real life project case and executed in a gaming context to improve motivation and engagement. The uniqueness and strength of the design presented in this paper comes from its ability to engage the students actively in the entire risk management process. The design also provide students with ability to simulate some of the risks they have identified themselves during the team-assignment. This gave the students a feeling of ownership to risk management process during simulation. 1.
Introduction
Project management has become a core business process for many firms both on strategic and operational levels. The PMBOK (PMI 2004) identifies 44 processes that fall into 5 basic process groups and 9 generic knowledge areas. Project risk management is one of the 9 project management areas and focuses on describing the processes that are important in order to conduct proper risk management on a project. Interest in risk management has increased as the size and complexity of projects have grown and as competition between firms has intensified (Maytorena, Winch et al. 2007). As a result, numerous best practice standards, guides, and specialist tools and techniques have been developed focusing on a more effective project risk management process. Colloquially, project risks are defined as uncertain events or conditions that may lead to positive or negative effect on at least one project objective, such as time, cost, health and safety, quality and so on (PMI 2004, p 238). The objectives of project risk management are to increase the probability and impact of positive events, and decrease the probability and impact of events adverse to the project (PMI 2004, p 237). The general consensus from the project management institute (PMI 2004) and other risk management literature (Chapman and Ward 2003) and (Kerzner 2006) is that the risk management process can be divided in 4 basic processes. 1. Risk identification: the process of identifying events or conditions that may occur during project execution and could impact at least one project objective 2. Risk assessment: the process of identifying the likelihood of the event and the magnitude of its consequences on project objectives. This task is done in 2 stages, qualitative assessment followed by quantitative establishment of numerical rating to prioritized risks on one or several project objectives (PMI 2004, p 237). 3. Risk response planning: the process of identifying measures for dealing with risks. This includes the choice of proper strategy to avoid, transfer or mitigate risks in case of having risks that could be perceived as threats. Risk response also includes identifying the strategies to exploit, share or enhance risks if they contribute positively to the project (risk as opportunity). 4. Risk monitoring and control: the process of monitoring, evaluating and updating the risk register. Risk mentoring tools include, re-assessment of risks and re-examining of risk response measures.
Teaching project risk management (Martin 2000) indicates that some things cannot be easily be learned by reading, writing or thinking about them. You may have to do them as well. Using lectures, assignments and case studies therefore do not help students to develop an understanding of the difficulties involved in identifying, assessing, planning and monitoring risks. Developing an understanding to these processes requires different type(s) of instructional methods. (Taran 2007) questioned the effectiveness of using lecture-based teaching to provide students with enough confidence and ability to apply risk management concepts after finishing the course, and has specifically pointed out 2 major shortcomings: 1- Lectures do not provide possibility to try out and experiment with the material being taught. Specific exercises and activities help, but are not providing an entire “project picture” 2- It is difficult to provide students with a way to experience scenarios of following or ignoring risk management practices. Developing a design that is capable to capture the “project big picture” is a key word for the research effort presented in this paper. (Cano and Saenz 2003) points out that despite wide spread use of simulation games it is still not clear what conditions have to be provided in order to obtain optimal learning through simulation games. Most of the current games which are being used in education primarily falls under functional simulation games and usually insensitive to the dynamics of actual projects (Hussein 2007). The author believes therefore that building proper design should begin with identifying concrete challenges, solutions and tactics that are used by project managers in real projects and then mirror these issues into the design. In this way, students will be able to enhance their project risk management skills and develop deeper understanding of the overall process. Other reported shortcoming with the existing simulation games includes oversimplification (Hussein 2007). The oversimplification is manifested in the type, timing and range of events occurs during the game. The design presented in these paper seeks also to address this issue and our aim is to present a design that have adequate level of realism so that it can prompt appreciation for project complexity. The importance of using methods that prompt understanding and appreciation of project complexity was also stressed in (Thomas and Mengel 2008) Goal and scope of the paper The main goal of this paper therefore is: to develop a blended learning lesson that embodies important challenges faced by project mangers during managing project risks. The lesson should also prompts understanding and appreciation of project complexity by linking the design to a real life project case. The lesson should be executed in a gaming context in order to improve motivation and engagement. In order to achieve this goal, the following tasks are carried out: 1) Identification of requirements: Interviews are conducted with senior project managers from several management-consulting firms. The aim of these interviews is to identify the challenges and the corresponding tactics that are adopted in practice in order to manage project risks. These results are also presented in light of supporting literature. The challenges and associated tactics will then be mapped into a set of requirements (intended learning outcomes) for the lesson. The results are presented in section 2. 2) A proposal for a lesson design will be developed. Each requirement will be realized by one or several instructional aids and methods. All the instructional methods will be linked by a real life project case and executed in a gaming context. The design is fully explained in section 3. 3) The final task is to evaluate the design. The aim of the evaluation is restricted in this paper to investigate the impact of not implementing the full design. That is, not including some of the defined requirements. The evaluation strategy adopted in this paper is to test the design using 2 different classes. One class will be exposed to the complete design while the other will be exposed to parts of the design. Students will then be asked (from their own perception) to rate how the lesson improved their skills in each of the project risk management processes compared to
prior attending the lesson. The author assumes here that the students which are exposed to the complete design will rate the increase in their skills higher than the students which are exposed to only parts of the design. 2.
Identification of requirements
Semi-structured interviews were used in order to collect challenges and solutions associated with project risk management. After exhausting the questions, informants were offered the opportunity to provide other insights regarding managing project risks. Each project manager was asked to identify at least 2 major challenges or issues during each stage of the project risk management process. These interviews yielded a framework of general issues, challenges and specific tactics to be used by project managers to address these. The list of issues was then mapped into a list of 10 requirements (intended learning outcomes) the design must have. The interviews revealed the following results: Context All the informants have stressed the importance of identifying and understanding project context as perquisite for managing project risks. According to the informants, this is achieved by using proper methods for gathering and distributing information concerning project goals, objectives, constraints, conditions and limitations. That could include time, and budget constraints or organizational and resource constraints, laws, ethics, financial and pricing structure. It was stressed that project managers should make sure that the information is made available to, and understood by those who will be responsible for managing risks. Similar conclusions were also made by (Kendrick 2009). Mapping these issues into the design gives the following requirements: R1. Design should show that availability (with the right quantity and quality) of project information has an impact on the final outcome of the project risk management process. R2. Design should reflect the dynamic nature of projects, including changing constraints, stakeholder support and others. Risk identification The informants stressed the importance of having the right persons with the right experience in the group. They have also stressed the importance of including representatives of any stakeholder who has a stake in at least one of the project objectives. Work by (Maytorena, Winch et al. 2007) does not support this view and argues that the role of experience in the risk identification process is much less significant than it is commonly assumed to be. They confirm however that information search style, level of education and risk management training do play a significant role in risk identification performance. Lack of time has also been mentioned as a source of problems in this stage. The informants advised the use of structured and formal approach to risk identification stage. Informants have pointed that historical information and knowledge that has been accumulated from previous projects and from other sources of information are very helpful in speeding up the identification stage. In order to encounter these issues, the informants suggested using several identification techniques such as brainstorming, cause-effect relations and others. These techniques are described by (PMI 2004). (Lester and Lester 2007) gives the advantages and disadvantages of the usual risk identification methods such as brain-storming, prompt list, checklists and other methods. Mapping these issues into the design gives the following requirements: R3. Design should demonstrate the importance of competence and experience in the actual project domain. R4. Design should demonstrate the importance of using various formal techniques to identify risks that could occur in projects. R5. Design should take into account challenges in real project situations such as time limitation and human factors.
Risk Assessment The informants identified 2 main challenges associated with risk assessment stage: 1- Lack of experience with project risk assessment. This involves assessment of probability and the impact of risks on project objectives. The strategy identified by the informants to tackle this problem involves selecting proper compositions of persons with relevant experience in project domain as well as supporting the assessment with historical data from previous projects. 2- Failure to prioritize risks. Project risk management practice indicates that it is neither possible nor recommended to mitigate or eliminate all risks in the project. Monte- Carlo analysis is frequently used for assessing the probability of achieving project objectives such as cost and time in the presence of risks (Lester and Lester 2007). Risk prioritization is performed by grouping risk factors into categories depending on the magnitude of impact and probability of risks (Kendrick 2009). Informants have identified the causes for failing to prioritize risks appropriately, these includes: •
Inability to define proper boundary conditions for these categories
•
Trying to fit collected data into predefined models rather than adapting the models to collected data. Informants have stressed the importance of adaption of existing models in order to be able to prioritize risks. In addition to the know-how in using analytical techniques for assessing and prioritizing risks it is evident that former experience and familiarity with risk category is a precondition for completing this stage (Chapman and Ward 2003).
Mapping these issues into the design gives the following requirements: R6. Design should show that accurate risk assessment requires access to historical information and data from similar projects. R7. Design should help the learners to enhance their skills in project risk assessments R8. Design should help the learners to visualize the outcome of failing to prioritize risks probably on project objectives. Risk response planning Risk planning involves selecting proper measures in order to reduce or mitigate the probability of risk or to reduce its consequences. Risk planning could also include measures intended to remove the conditions that cause this type of risks. Informants have in the interviews stressed that all the agreed upon measures must have measurable results. Work by (Fan, Lin et al. 2008) confirmed that a proper risk-handling approach should take into account unique project characteristics, risk situation, and implications on project objectives. Acquiring information and improving communication are preconditions for developing proper risk response measures. Mapping these considerations into the design gives the following requirement: R9: Design should help learners to experience the impact of failing to select proper measures to deal with risks. Thus using the simulation as a forecasting tool to investigate possible risk response strategies for dealing with risks. Risk monitoring and control Informants have identified that the major challenge in this stage is information gathering and distribution, the availability of new information about changing project conditions, information about the results from risk response planning. Mapping these considerations into the design gave the following requirement: R10. Design should illustrate the importance of effective communication between the participants in order to be able to make informed decisions.
3.
Lesson Design
In this paper, we shall assume that all the above requirements are equally important. The author is aware that this assumption needs more testing. However, because of space limitation, the investigation of this assumption has not been done. This will be the subject of further research. The case chosen for the design is the construction of medium size house as a turnkey contract for a private owner by a construction company. The players in the design form project teams of 45 persons and are supposed to develop, assess, mitigate, and monitor a risk-register for the project according to the needs defined by the project owner. The project has 2 key objectives; 1) total costs should be kept within the budget, 2) project should be completed on time. Project teams must therefore strive to follow proper risk management practices in order to avoid any adverse consequences that might cause them to fall behind their opponents in the race to complete the project and satisfy all project objectives. The design proposed in this paper is a blended learning lesson and uses a combination of 4 instructional methods (briefing lecture, team-based assignment, computer game simulation, and a debriefing lecture). These methods are implemented in order to realize all the 10 requirements identified in the previous section. The author believes that trying to implement all the requirements by means of a computer game simulation only might lead to a complex game. This complexity will in turn affect learning in negative manner (Kiili 2005). In the following section, the paper will give a brief description of each method used in the design. 1. Briefing lecture. The aim of the briefing lecture is to introduce the underlying project risk management theory and processes. The concepts of risk register; risk matrix, brainstorming, qualitative and quantitative risk management, risk planning and risk monitoring techniques are explained. At the end of the briefing lecture, project teams (4-5 persons) are established and a Project Definition Document (PDD) is distributed to the teams. PDD includes information about project scope, product description, cost and time constraints, project success criteria defined by the project owner, and other project assumptions and requirements. Teams were also instructed to decide themselves the roles and responsibilities of each team member at each stage of the game. 2. Team-based assignment Based on the description given in PDD, teams are instructed to identify and develop a complete risk-register for each work package. The risk-register is a document that summarizes risk factors and assessment of the impact and likelihood of these factors. It summarizes also possible measures to mitigate these risks. A Leaflet containing typical problems, issues, solutions and statistics collected from similar previous projects were distributed to the teams during the assignment. At the end of the assignment, the instructor collects and reviews the completed risk-registers from each team. Risk factors are then categorized and duplicates are omitted. The final edited list is then fed into the database of a simulation environment where actual project execution will take place. The aim of this team-based assignment is two folds. Firstly, it provides students with hands-on training in identification, assessment and mitigation processes. Secondly, it prepares each team for the next phase of the game (computer simulation) when they will have to assess, prioritize and select mitigation measures for far more risk factors than what each team had managed to identify and assess on its own. 3. Computer game simulation This is a turn-based simulation environment that was programmed using Java script. The simulation environment contains an updateable risk-register database. The database contains a register of risk factors that may occur and could affect project objectives. At startup of the simulation, the simulation engine selects randomly 3-5 risk factors from the database for each work package. The list of risks includes 2 categories; 1) risk factors identified by the teams during the preceding assignment, 2) risk factors identified in previous experiments by other classes and other project mangers during developing the simulation. The database can therefore be seen as an incubator of risk factors for this class of projects. During simulation, teams should then use their own expe-
rience and information provided in the Leaflet and PDD in order to assess the likelihood and the consequences of each risk factor displayed before them. Alternative risk mitigation measures are also displayed next to each risk factor. Failing to select proper measure might trigger the occurrence of additional risk factors in later project stages. The price or time needed in order to implement each measure is also given. Failing to assess risks correctly will result ultimately in failing to respond to critical risks. This might result in severe delays, penalties by authorities, slow progression and so on. Visual effects such as video clips, sounds and images are used to illustrate the consequences of failing to assess risks. At the same time, teams should not mitigate all the risks listed before them. If they do so, actual costs will overrun the budget. Teams must therefore mitigate only those risk factors that are critical or significant. Information provided in the Leaflet are meant to assist the teams to prioritize risks and select measures appropriately. The simulation environment also contains 2 types of visual aids: •
Progression map. During project execution, teams will be able to get updates and information about the status of the project. Information shown in the map includes; work packages completed, percentage completion, number of days passed, money remaining, person-hours used, simulation time. Maximum allowed time to complete the project in the simulation was set to 60 minutes. After 60 minutes a fine will be imposed for each additional minute.
•
S-Curve. This is a graph showing the accumulated actual costs (AC), planned costs (PV), and the earned value (EV). S-curve is updated after each turn in the simulation. This graphical aid should help teams will to visualize the consequences of their decisions on time and money instantly. A reflective analysis of the information obtained by the S-curve help teams to think different during risk assessment and prioritization in the next work package.
4. Debriefing Lecture The debriefing lecture takes places at the end of computer simulation. It includes evaluation of the performance of each team, revisiting and discussing execution strategies, how teams distributed roles and responsibilities. Discussion with teams about efficiency of communication in the team, reflections about conformity, dominance, bias, and indifference are also investigated and explored with teams. Linking the results to the project management theory and finally to identify lessons learned. 4.
Evaluation of the design
The evaluation was done by running the lesson in 2 separate classes and then comparing the results. In the first class, the full design was used. We shall refer to the first class hereafter by Trondheim-class since the course was held in the city of Trondheim. In the second class, only parts of the design were used. We shall refer to the second class hereafter by Oslo-class since the course was held in the city of Oslo. The assumption here is that Trondheim-class (exposed to all the identified requirements in part 1) will assess the learning outcomes of the lesson higher than Oslo-class. Percentile and descriptive statistics will be used to support/refute the assumption. To examine this assumption each respondent was asked to assess to what degree the lesson has enhanced their overall knowledge in each of the project risk management processes and in overall. The scale given was from 1 to 6. Where 6 means considerably high, 5; high, 4, tend to high, 3 tend to low, 2 low and 1 means almost none. It is expected that there will be a tangible difference between the assessments of the two classes. Table 1 shows an overview of the identified requirements and how each requirement was embodied in the complete design. The table also shows which requirements (marked using red color) were not included in Oslo-class compared to Trondheim-class.
Requirement
Complete design (Trondheim-class). How the requirement was realized in the design?
Oslo-class
R1. Design should show that availability of project information impacts the final outcome of the project risk management process.
Briefing lecture / PDD is prepared in advance and distributed to all teams
Same as Trondheim-Class
R2. Design should show the dynamic nature of projects.
Simulation / some contextual risk factors are changed randomly.
Same as Trondheim-Class
R3. Design should illustrate the importance of knowledge in the actual project domain.
Briefing lecture / Teams formation is not random.
Team formation is random
R4. Design should demonstrate the importance of using various formal techniques to identify project risks.
Team-based assignment / teams were given time to try themselves to use formal techniques to identify, assess and mitigate project risks.
Was not used in Oslo-class. Team went directly from the briefing lecture to computer simulation
R5. Design should take into account challenges in real project situations such as time limitation and human factors.
Briefing / It was not permitted to work individually. Simulation was time limited. Debriefing /discussion with teams about human aspects encountered during simulation.
Same as Trondheim-Class
R6. Design should allow the learners to access information about similar projects.
Team-based assignment / Leaflets containing historical data and statics from similar projects are distributed.
Was not used in Oslo-class. Leaflet was NOT distributed
R7. Design should help the learners to enhance their skills in risk assessment.
Simulation / Teams are instructed to use their own experience and information Leaflet in order to assess risks accurately.
Leaflet was NOT distributed
R8. Design should give the learners the possibility to visualize the outcome of failing to prioritize risks correctly.
Simulation / S-curve and Progression map are built in the simulation to provide information to teams about progression and earned value.
Same as Trondheim-class
R9: Design should help the learners to experience the impact of failing to implement proper measures to deal with risks.
Simulation/ Failing to select the proper measure could trigger the occurrence of risks in a later project stage
This function was disabled in Oslo class
R10. Design should illustrate the importance of effective communication in order to be able to make informed decisions.
Briefing/ After forming the teams, students were instructed to decide themselves the roles and responsibilities of each team member.
Same as Trondheim-class
Table 1. Requirements and the method used to realize each requirement in both Trondheim and Oslo classes. Results Tabel 2 and 3 shows the results obtained from Trondheim- and Oslo-class respectively. The results indicat that 75% of the respondents in Trondheim-class rated the lesson’s ability to enhance their knowledge in risk identification process as (Tend to high) or better. While only 62.5 % of the respondents in Oslo-class rated the same process as (Tend to high) or better. This result gives support to the importance of including R4 (the design should give the students the possibil-
ity to experiment with different formal methods to identify risks) and R3 (importance of relevant competence in project domain). The tables show that 87,5% of the respondents in Trondheimclass rated the design ability to enhance their knowledge in risk monitoring process as (Tend to high) or better. While only 70.8 % of the respondents in Oslo-class rated their learning in the same process as (Tend to high) or better. This result gives support to the importance of including R9 (design should help the learners to experience the impact of failing to implement proper measures to deal with risks.). The results show small differences between Oslo and Trondheim concerening learning outcome in risk assesment process. We might therfore conclude that (R6) might not be very critical to imporove the learning outcome of risk assesment process.
Assessment scale Almost none Low Tend to Low Tend to high High Very high Sum (High) Mean
Risk Risk Risk Re-‐ Risk Identification Assessment sponse Monitoring 0.0 % 0.0 % 4.2 % 0.0 % 4.2 % 4.2 % 4.2 % 8.3 % 20.8 % 29.2 % 20.8 % 4.2 % 33.3 % 41.7 % 45.8 % 50.0 % 37.5 % 25.0 % 25.0 % 37.5 % 4.2 % 0.0 % 0.0 % 0.0 % 75.0 % 66.7 % 70.8 % 87.5 % 4,2
3,9
3,8
Overall risk mgmt. process 0.0 % 4.2 % 12.5 % 41.7 % 29.2 % 12.5 % 83.3 %
4,2
4,3
Risk Monitoring 0.0 % 25.0 % 16.7 % 50.0 % 20.8 % 0.0 % 70.8 % 3,6
Overall risk mgmt. process 0.0 % 12.5 % 25.0 % 45.8 % 29.2 % 0.0 % 75.0 % 3,8
Table 2. Results from Trondheim-class (N = 24 respondents)
Assessment scale Almost none Low Tend to Low Tend to high High Very high Sum (High) Mean
Risk Identification 0.0 % 16.7 % 33.3 % 29.2 % 33.3 % 0.0 % 62.5 % 3,7
Risk Assessment 4.2 % 16.7 % 29.2 % 41.7 % 20.8 % 0.0 % 62.5 % 3,5
Risk Response 4.2 % 25.0 % 12.5 % 33.3 % 33.3 % 4.2 % 70.8 % 3,7
Table 3. Results from Oslo-class (N = 27 respondents) The results obtained from these 2 experiments gave support to the importance of the following requirements; 1) Design must provide training in using formal methods to identify risks. 2) Design must take into account participants experience and competence in the project domain. 3) Design must provide training to the learners so that they can make informed decisions about the type of measures needed in order to mitigate project risks. (The design should show consequences of failing to mitigate the risks correctly). Evaluation have also shown that the availability of information about other similar projects might not be very critical if the project team members have relevant experience in the project domain. This result is not in accordance with data collected through interviews and therefore needs more investigation and analysis. Additional tests methods are required in order to examine the criticality of the other requirements identified. This will be the subject of future work. 5.
Conclusions
The major challenge of the research presented in this paper has been to select the right type and the timing of the different instructional aids and methods. It was indeed very tempting to let the computer game takes care of everything, but this would have resulted in a bulky computer game.
It will require substantial cognitive resources from the student in order to operate. This may in turn lead to reduced learning. Students who have participated in the complete design have indicated during debriefing session and casual discussions that the strengths of design are attributed to the following factors; 1) students were actively involved in the entire risk management processes, 2) students were able to simulate some of the risks they have identified themselves during the teamassignment. This gave students a feeling of ownership to the simulation, 3) students were able to experience in a very condensed time frame that decisions concerning risks must be based on adequate analysis of both risks and project context, 4) students have experienced that the main purpose of conducting project risk management is in fact to learn more about the project (a tool for learning, rather than decision making), and finally they appreciated the use of video clips, images, texts in order to show or visualize some of the consequences of failing to prioritize or mitigate risks correctly, they have reported that using this kind of aids gave the simulation a touch of reality. Acknowledgments: The author wishes to express his appreciation and gratitude to the reviewers for their valuable feedback and suggestions. References Cano, J. L. and M. J. Saenz (2003). "Project management simulation laboratory: Experimental learning and knowledge acquisition." Production Planning and Control 14(2): 166-173. Chapman, C. B. and S. Ward (2003). Project risk management: processes, techniques and insights. Chichester, Wiley. Fan, M., N.-P. Lin, et al. (2008). "Choosing a project risk-handling strategy: An analytical model." International Journal of Production Economics 112(2): 700-713. Hussein, B. A. (2007). On using simulation games as a research tool in project management. Organizing and learning through gaming and simulation, ISAGA 2007. the Netherland, Uitgeverij Eburon. Jaafari, A. (2001). "Management of risks, uncertainties and opportunities on projects: Time for a fundamental shift." International Journal of Project Management 19(2): 89-101. Kendrick, T. (2009). Identifying and managing project risk: essential tools for failure-proofing your project. New York, AMACOM. Kerzner, H. (2006). Project management: a systems approach to planning, scheduling, and controlling. Hoboken, N.J., Wiley. Kiili, K. (2005). "Digital game-based learning: Towards an experiential gaming model." Internet and Higher Education 8(1): 13-24. Lester, A. and A. Lester (2007). Project management, planning and control: managing engineering, construction and manufacturing projects to PMI, APMand BSI standards. Amsterdam, Elsevier/Butterworth-Heinemann. Martin, A. (2000). "A simulation engine for custom project management education." International Journal of Project Management 18(3): 201-213. Maytorena, E., G. M. Winch, et al. (2007). "The influence of experience and information search styles on project risk identification performance." IEEE Transactions on Engineering Management 54(2): 315-326. Perminova, O., M. Gustafsson, et al. (2008). "Defining uncertainty in projects - a new perspective." International Journal of Project Management 26(1): 73-79. PMI (2004). A Guide to the project management body of knowledge: (PMBOK guide). Newtown Square, Pa., Project Management Institute. Taran, G. (2007). Using games in software engineering education to teach risk management, Piscataway, NJ 08855-1331, United States, Institute of Electrical and Electronics Engineers Inc. Thomas, J. and T. Mengel (2008). "Preparing project managers to deal with complexity Advanced project management education." International Journal of Project Management 26(3): 304-315.
