Supporting risks in software project management - CiteSeerX

ARTICLE IN PRESS

The Journal of Systems and Software xxx (2003) xxx–xxx www.elsevier.com/locate/jss

Supporting risks in software project management M arcio de Oliveira Barros, Cl audia Maria Lima Werner, Guilherme Horta Travassos

*

Federal University of Rio de Janeiro, COPPE/UFRJ––Computer Science Department, Caixa Postal: 68511-CEP 21942-972 Rio de Janeiro, RJ, Brazil Received 13 August 2002; received in revised form 18 November 2002; accepted 23 November 2002

Abstract Complex software development is a risky job. The number of unsuccessful projects surpasses the number of successful developments, particularly when large projects are analyzed. This paper describes an approach to develop, retrieve, and reuse management knowledge and experience concerned with software development risks. Scenarios are used to model risk impact and resolution strategies efficacy within risk archetypes. A risk archetype is an information structure that holds knowledge about software development risks. A risk management process organizes the use of risk archetypes within an application development effort. The process resembles a reuse process framework, where two sub-processes are respectively responsible for identifying and reusing risk information. Simulating the impact of the expected risks can support some of the decisions throughout the software development process. The contribution of this paper is to show how risk archetypes and scenario models can represent reusable project management knowledge. An observational analysis of applying such an approach in an industrial environment and a feasibility study are also described. 2002 Published by Elsevier Inc. Keywords: Software project management; Risk analysis; Knowledge reuse and management; Empirical studies

1. Motivation Unsuccessful software stories can be found in several documented case studies and experiments over the last years (Charette, 1996). Although many improvements were achieved in software engineering, most software development projects still use more resources than planned, take more time to be concluded, provide less functionality and less quality than expected. Many studies relate such failures to inadequate project management, pointing to communication problems, wrong allocation of team skills, insufficient training, and manager inability to predict and adjust project behavior. Most of the techniques currently applied to software project management require that projects have clear and delimited objectives, there exists at least one solution to the problem at hand, development time and resources can be precisely stated before the project starts, the operational environment can be well defined, and * Corresponding author. Tel.: +55-21-2562-8712/8675; fax: +55-212562-8676. E-mail addresses: [email protected] (M.O. Barros), werner@ cos.ufrj.br (C.M.L. Werner), [email protected] (G.H. Travassos).

0164-1212/$ - see front matter 2002 Published by Elsevier Inc. doi:10.1016/S0164-1212(02)00155-3

quality metrics can be quantified for the project. These assumptions rarely occur in large projects. The development of projects for application domains with highrequirements volatility, the need for domain integration, ambiguity, complexity, discontinuity, diseconomy of scale, and complex feedback loops are characteristics of large projects. In practice, most projects are subject to uncertainty. These characteristics undermine the assumptions of traditional techniques, which can work in theory, but cannot be directly applied in practice. However, due to some successes resulted from using these techniques in the past, managers tend to take their underpinning assumptions for granted in every project. This misconception is very common and dangerous in software project management. Moreover, by using the same techniques, it seems that experienced managers are able to perceive some aspects of a software development process that remain invisible to novice managers. They account for these aspects in their decisions, raising their chances of developing a successful project. This management knowledge, which is embedded in expert managers’ mental base of experiences (Klein, 1998), is a valuable asset. Its importance

ARTICLE IN PRESS 2

M.O. Barros et al. / The Journal of Systems and Software xxx (2003) xxx–xxx

to software projects’ success suggests the need for a technique to represent this knowledge, allowing novice managers to share the experience of experts. This paper describes an approach to develop, retrieve, and reuse knowledge and experience concerned with software development risks. In this approach, a manager defines and documents the expected behavior for a project. Since uncertain events can affect this behavior, the manager analyzes project sensibility regarding several combinations of such events. This analysis provides feedback about risks that can challenge the project success. Scenarios are used to model risk impact and resolution strategies, conveying managerial knowledge that can be reused in several projects. A process is provided to support risk management through scenario analysis. The process resembles a reuse process framework, where two sub-processes are respectively responsible for identifying and reusing risk information. Two experimental studies were accomplished to support the proposed approach: an observational analysis in an industrial environment and a feasibility study in an academic environment. Their quantitative and qualitative data have been used to draw future research directions. We organized this paper in six sections. The first section comprises this motivation. Section 2 presents the fundamentals of the Scenario Based Project Management paradigm. Section 3 presents a risk management process based on documenting risks as scenarios. Section 4 presents two application studies of the proposed techniques. Section 5 compares the proposed approach to related works. Finally, Section 6 presents future perspectives of this work.

2. Scenario based project management The traditional view of software project management is deeply rooted in the assumptions taken by the currently applied software management techniques. It presumes that the project manager can provide a detailed strategy to guide the development team from early product specifications to the final software system, within predefined schedule, quality, and budget constraints. The accomplishment of such task requires complete knowledge about the project under development. This knowledge may not be available in innovative projects or in the early phases of the software development process. Since such a precise knowledge about project behavior is not common in several disciplines, including software development, the traditional management techniques were adapted to handle some level of uncertainty. For instance, activity networks were extended to allow a probability distribution, instead of a crisp number, to describe the time required to accomplish an activity. Resource allocation tables were adapted to

convey conditional statements that could change their resource distribution due to the occurrence of uncertain events. The project plan is therefore a dynamic artifact. It is susceptible to change throughout the development process while project uncertainties can flourish to problems or be contained. However, traditional management techniques cannot directly capture every kind of uncertainty. These techniques are unable to describe other sources of uncertainty than those elements to which they were originally developed, yet constrained by the information they hold for these elements. For instance, PERT diagrams can only convey uncertainties about activity duration, cost, and activities that are executed according to predefined conditions. Uncertainties about developers’ motivation, error generation, discovery, and correction rates due to the use of a particular software development method cannot be represented in a conventional activity network. In fact, we believe that no static model (such as an activity network or a resource allocation table) can support the evaluation of every kind of risk. However, an extensible model could describe the basic information about a project and allow other models to supply the complementary information needed to represent uncertainty events. The Scenario Based Project Management paradigm comprises a set of techniques that allow a manager to define the expected behavior for a project and several different scenarios that might occur along its development. Combinations of such scenarios are applied over the project and used to evaluate their impact upon its behavior. The architecture of the proposed approach is presented in Fig. 1. It is centered on two artifacts: the project model (represented by the ‘‘PM’’ blocks in Fig. 1) and scenario models. The project model defines the expected behavior for a project, while scenario models hold the representation of events, policies, management procedures, actions, and strategies. These cannot be considered part of a project, but practices imposed or

Fig. 1. Scenario based project management approach architecture.

ARTICLE IN PRESS M.O. Barros et al. / The Journal of Systems and Software xxx (2003) xxx–xxx

applied to the project and exceptional situations that may happen throughout its development. Scenarios can be used to modify a project model, which may impose project behavior deviation due to the events they represent. Since such assumptions are valid for several distinct projects, scenarios convey potentially reusable management knowledge. Highsmith (1992) observed that successful software projects convey a blend of careful objective evaluation, adequate preparation, continuous observation and assessment of the environment and progress, and adjusting tactics. Senior managers usually make this environmental assessment mentally. They construct mental models for the project, problems, and opportunities under investigation. Next, they scan their mental experience repository for adequate actions to be taken against problems or to explore opportunities. Scenario Based Project Management adopts a similar ‘‘anchoring-and-adjustment’’ approach to project management. Its main distinction from previous proposals is to provide a formal way to describe the project expected behavior and the uncertainties that may occur throughout its development. Also, it allows the manager to evaluate the impact of such uncertainties over project behavior. Both project and scenario models are formal software project models, described using system dynamics. System dynamics is a modeling technique and language to describe complex systems, focusing on their structural aspects (Forrester, 1961). This technique identifies and models cause-effect relationships and feedback loops. These relationships are described by using stock-andflow diagrams composed by four basic constructors: stocks, rates, flows and processes. Stocks represent elements that can be accumulated or consumed over time. A stock is described by its level, that is, the number of elements in the stock at a given moment. Rates describe stock variations, formulating their raises and depletions in an equation. Processes and flows complement complex rate equations and calculate intermediate model variables. Simulation is the technique that allows model behavior evaluation. Since system dynamics models are described by mathematical formulations, their equations can be evaluated to reveal the underlying project behavior. Models are evaluated through continuous simulation, where a simulation cycle is divided in several steps. Each step advances a clock by a constant and infinitesimal increment and evaluates model equations. Model results are obtained from the values assumed by its equations after a predefined number of simulation steps. System dynamics has been previously used in software engineering. Abdel-Hamid and Madnick (1991) developed the first system dynamics software project model. This model formalizes the effects of management policies and actions taken throughout a software de-

3

Fig. 2. An excerpt from a traditional system dynamics model.

velopment project. Later, other models extended the original model or focused on different aspects of software projects (Tvedt, 1996; Lin et al., 1997). Scenario Based Project Management’s project model differs from previous system dynamics software project models due to its support to develop and integrate reusable scenario models. This feature required major modifications in the former modeling approach: raising the abstraction level of the model, breaking the uniform treatment of elements pertaining to the same category, and separating facts from assumptions (such as theories and uncertain events) within the model. To raise the level of abstraction of project models is a strategy to allow the operational use of system dynamics models in software project management. Traditionally, such models are described through mathematical equations, which are very hard to understand and adapt, inhibiting the use of system dynamics and simulation to describe software projects in a practical manner. Fig. 2 presents an excerpt of a traditional system dynamics model. The proposed project model is described through a high-level modeling language, which can be translated to system dynamics in order to be simulated (Barros et al., 2001). Such language helps the interaction between the user and the model, without losing simulation capabilities. The project model representation is based on its activities, developers, artifacts, and resources. Activities represent work packages that must be accomplished to conclude the project. Developers accomplish activities using resources. Resources represent physical or logical elements that are used throughout the project development, such as computers, software packages, and components, among others. Activities create or evolve software artifacts, eventually using other artifacts as income. In the proposed high-level representation, project activities, developers, artifacts, and resources are described by classes. Each particular activity in a project is described as an instance of the activity class. Classes are presented in a domain model, while the instances related to a particular project are presented in a project model. The domain model is built once and reused by several projects. A distinct project model is built for each application to be developed based on the classes (high-level constructors) defined in the domain model.

ARTICLE IN PRESS 4


A class defines the properties and the behavior that describe a general element, such as an activity. A property is a relevant information about the class that can assume independent values for each class instance. The behavior is a mathematical formulation of how an instance responds to changes in other class instances or in the environment. Traditional system dynamics constructors describe a class behavior. These equations can depend on class properties, allowing distinct instances to react differently based on their particular characteristics or state. Also, by separating behavior formulations per class, the equations related to a particular element are clearly separated in the model. This feature helps model understanding and maintenance, in contrast to traditional system dynamics models, where it is difficult to identify the equations that describe an element in the maze of equations that compose the whole model. The associations among project elements (such as an activity and its developers) are captured through class relationships. A relationship represents a structural connection between two or more class instances. Such relationships can occur among instances of different classes or instances of the same class. The existence of a relationship among class instances allows the behavior equation of an instance to use properties and equations of its associated instances. The domain model presents the whole set of relationships that can be defined among class instances, while the project model describes the relationships that occur in a specific project. Fig. 3 presents an excerpt of a domain model for software development projects. The model conveys two classes (Developer and Activity) and two relationships (activity precedence and activity development team). The Developer class has two properties, which are used to calculate a developer’s hourly cost and productivity.

The behavior related to the Activity class is partially shown: a single equation estimates team productivity based on the developers which are associated to the activity. Usually, system dynamics models uniformly treat all elements pertaining to the same category. For instance, in a software project model all experienced developers are supposed to have the same productivity, activities are supposed to take the same time to develop, and so on. We assume that this model simplification is due to system dynamics inherent incapability to describe element’s particular features (Barros et al., 2001). However, we believe that, if such a model is to be used for operational management, it must capture the differences among its developers, activities, resources, and artifacts. By breaking this uniformity down, project managers are able to represent the particular features of each software project element, which allows the construction of richer project models. The proposed project model representation is able to capture differences among activities, developers, resources, and artifacts by providing distinct values for the properties that describe each instance of such elements. Since their behaviors depend on these properties, distinct elements may present different behavior. Fig. 4 presents a simple project model, conveying two developers and an activity. Finally, separation of facts from assumptions within a project model allows a manager to test project sensibility to different combinations of uncertain events and theories. Traditionally, system dynamics project models blend known facts about the real-world elements that compose a project with several assumptions upon their behavior and interaction. This framework allows a manager to analyze the implications of such assumptions upon project relevant variables, such as conclusion time and cost. However, it is generally difficult to test other assumptions than those provided by the model. By separating facts from assumptions, our approach supports the evaluation of several combinations of assumptions (scenarios) upon the provided facts (project

Fig. 3. An excerpt from a software project domain model.

Fig. 4. A simple project model.


5

3. Scenario based risk management

Fig. 5. Equations for a communication overhead scenario.

model). Scenarios are not limited to the characteristics defined by the project model. They can define new attributes to the elements that participate within a project and use such attributes to represent their behavior. The simulation of the isolated project model describes project expected behavior, without the influence of uncertain events and actions. The integration of scenarios to a project model demonstrates the potential impact of the scenario occurrence in the project behavior. This kind of simulation allows the manager to test project sensitivity to the events and strategies described by the scenario model. Scenarios are developed to adjust the equations of rates and processes defined in the project model. For instance, Fig. 5 presents the equations that compose a simple scenario. Its equations and assumptions were adapted from (Abdel-Hamid and Madnick, 1991). The scenario uses an extended system dynamics syntax, which is extensively described in (Barros et al., 1999). This scenario represents productivity losses due to communication overhead in a software development project where two or more developers work together in a single activity. The AFFECT clause indicates that the scenario modifies the DevProductivity equation, which was defined for the Activity class in the domain model (see Fig. 3). Since several scenarios can adjust the same equation, scenario integration ordering is relevant. Scenarios can be organized in an organization-wide knowledge base to project managers. They allow formal documentation of assumptions and proven information about managing software projects. Senior project managers can create scenarios, based on their personal experience, allowing less experienced managers to share their knowledge by retrieving and reusing such scenario models. While managing an application development project, when information about the application domain, technologies to be applied, developer roles, artifacts to be built, among others, become available, scenarios associated to these elements help the manager to explore project uncertainties. The manager retrieves those scenarios that he believes to affect his project and evaluates their impact upon the project behavior. Scenarios can be analyzed in isolation or in groups, representing their combined impact upon the project behavior.

The techniques presented in the previous section describe how scenarios are formulated, integrated within a project model, and how project behavior is affected by scenario combinations. In this section, we explore such techniques to support a risk management process that represents the risks faced by a project as scenarios. In the context of the Scenario Based Project Management paradigm, risk management is based on documenting risks through scenarios, reusing them along several projects, and simulating scenario combinations for risk evaluation within a particular project. These activities imply two different concerns in the risk management process. First, risks are identified and documented using scenarios. Next, risks are reused along several projects, allowing the simulation of their impact upon the project behavior. These different concerns resemble a generic reuse process framework. This framework is based on two sub-processes: one to develop reusable artifacts (i.e., development for reuse), identifying and documenting these software assets, and the other for application development based on reusing previously crafted artifacts (i.e., development with reuse), selecting and adapting them from a reusable information base. Applying the generic process framework to the Scenario Based Project Management paradigm, its risk management process is divided into two sub-processes: one for risk identification, and the other for risk reuse throughout application development. The main objective of the risk identification process is to describe common risks associated to a specific project element, such as an application domain, technologies, developer roles, software artifacts, and resources. This process organizes such risk information and allows its reuse in several application development processes. It is executed when an organization explores a new project element that can impose risks to a development effort. The main objective of the risk management process for application development is to identify and evaluate risks that can affect a project due to its specific project elements. This process reuses the risks associated to project elements, identified and documented by the former risk management process. It occurs in parallel with an application’s development process, tracking and handling its risks. The risk management processes inforce quantitative risk analysis, modeling risk impact and resolution strategies efficacy as scenario models. By forcing risk documentation through scenarios, we expect that the risk identification team is able to precisely state their assumptions about the effects of these risks upon the project. Without formalizing the risk documentation, this reasoning may be highly subjective, eventually unusable to future projects.

ARTICLE IN PRESS 6


The fundamental reusable artifact connecting the two processes is the information describing a risk, represented as risk archetypes. A risk archetype is a description of a potential recurring problem, which can cause some kind of loss to a software product or development process. It includes a description of the context where the problem can occur and provides solution strategies that can be applied before and after the problem occurrence. Risk archetypes serve as an awareness mechanism to the project manager and as containers for scenarios that model risk impact, contention and contingency strategies. Risk archetypes identify, document and evaluate potential events that can negatively affect software development projects. The experience of forecasting, identifying and resolving the risk is documented by the risk archetype, which can be effectively reused in future projects. Risk archetypes determine the information that must be stored with the potential problem description, presenting its characteristics in a structured and standard form. The use of a standard representation promotes risk communication, understanding, and a clear distinction among several risk events. The risk archetype is a standard information structure composed by five blocks, described in the following paragraphs: • Archetype identification: describes the potential problem represented by the risk archetype. It contains a textual and a scenario model description for the impact of the potential problem upon a software pro-

•

•

•

•

ject. Sources of information used to build the model are also provided. Identification mechanisms: describes the mechanisms used to identify the risk in a particular project. This block contains a textual description of the context in which the risk occurs, a checklist to identify the conditions that promote the risk in a project, and a list of known cases of its occurrences. Contention strategies: describes resolution strategies to inhibit or eliminate the potential problem described by the risk archetype before the risk occurs in a project. It provides a textual and a scenario model description of each contention strategy, associating these strategies with the conditions when they can be applied. As for the scenario model in the archetype identification block, information sources used to develop each scenario model are provided. Contingency strategies: describes resolution strategies to reduce the impact of the potential problem described by the risk archetype after its occurrence in a project. As in the contention strategies block, this block provides a textual and a scenario model description for each contingency strategy, along with information sources used in their development. Related archetypes: describes risk archetypes that occur in similar situations or that can replace the current archetype in software projects.

We expect that the structured form promotes information completeness, so that a risk archetype conveys

Fig. 6. Validation risk archetype.


7

all the information needed for future risk analysis. Information completeness is very important to reduce psychological barriers to risk analysis: risk is an abstract concept (Kontio and Basili, 1996), and an incomplete description of the risks to which a project is subjected inhibits the motivation for risk management. Fig. 6 presents an example of a risk archetype for the validation risk, that is, the risk of developing the wrong system. 3.1. The risk identification process An application domain may contain risks that occur in several projects developed within the domain. For instance, some domains are subjected to higher volatility of requirements, strict performance or reliability goals. Applications developed using a specific technology may share some common risks. If a software development organization decides to use C++ coding in a project, this project may be subjected to low-portability issues. Moreover, the adoption of a software process may also inflict some risks to the project. These examples show us that several project elements, such as application domain, selected technologies, and artifacts produced or consumed, impose risks to a software development project. We argue that this is a natural consequence of the benefits and disadvantages brought by these elements to a project. However, these positive and negative forces generated by them may not be clear to a project manager when planning or accomplishing a development effort. There must be some documentation describing them and providing means for their consideration during project planning. Since every project element may be related to uncertainty events that cause prejudice to a software project, development teams must reuse knowledge about risks that should occur in applications developed with these elements. The formalization and reuse of these risks reduces the effort spent by every project in risk identification. It also prevents risks from passing unnoticed through particular application risk identification activities. The objectives of the risk identification process are the determination of risks that can occur in applications related to a project element, specification of the conditions when such risks are relevant to an application, definition of mechanisms for risk identification in applications, and definition of guidelines for contention and contingency plans. The process produces several risk archetypes. Risk reuse has been previously reported (Kontio and Basili, 1996; Garvey et al., 1997; Hall, 1998). The main differences of our approach are: • The use of scenario models to represent risk, contention and contingency strategies impact upon project

Fig. 7. The risk identification process.

behavior, along with the use of simulation to evaluate this impact. • The association among risks and project elements, which allows the selection of the risks to which an application is subjected, based on its project elements. Fig. 7 presents the activities that compose the risk management process for risk identification. Due to its inherent refinement structure, where each activity refines the artifact created by the previous one, the process architecture resembles a waterfall life cycle. However, we expect the process to be applied in several iterations, each one discovering new risk archetypes and adding new information to previously documented archetypes. The first activity of the risk management process for project element risk identification is the risk archetypes identification. It aims to list risks that can frequently occur in applications related to the project element under analysis, highlighting the context that promotes these risks occurrence. This activity uses uncertainties, difficulties, and existent knowledge about the project element. Within this activity, the risk management team produces risk statements, represented as partial risk archetypes, where only the identification blocks are fulfilled. The risk archetype identification activity also defines the risk impact scenario, that is, a scenario model that describes the impact promoted by the risk occurrence upon a project expected behavior. During the risk management process for application development, a project manager is recommended to simulate project behavior sensitivity to several combinations of risk impact models in distinct time frames. This is accomplished by integrating the risk impact models to the project model of the particular application. The next activity (risk checklist formulation) assesses questions that help to identify the conditions that promote each risk archetype occurrence in an application. Each risk archetype is independently analyzed, and its

ARTICLE IN PRESS 8


checklist formulated. This activity results in refined risk archetypes, where risk statements and uncertain factors that inforce their occurrence can be sought. Since the fundamental uncertainties that promote risks are expressed in the risk checklist, they guide the selection of contention plan strategies. These plans define actions that can control the uncertainty expressed in checklist questions, reducing the probability or the impact of the potential problem occurrence. The contention plan formulation activity studies each checklist question to identify actions that can be taken to reduce the uncertainty that it expresses. It is important to remember that no project can prevent from every risk, even if there are contention plans for every uncertainty. However, if there are prepared plans for every situation and risks are periodically evaluated; the most relevant contention plans can be activated throughout the project life cycle. A scenario model is developed for each contention plan identified at the contention plan formulation activity. Each model shows a contention plan impact upon the project behavior. As it occurs with risk impact scenarios, a project manager simulates project sensitivity for contention scenarios during the risk management process for application development. Multiple contention scenarios can be combined with risk impact scenarios, allowing the manager to evaluate their expected efficacy to resolve the risks. The contention plan formulation activity results in even more refined risk archetypes, where risk statements, uncertainty factors, and uncertainty controlling action can be sought. Likewise, the contingency plans formulation activity studies the risk archetype, identifying actions that reduce risk occurrence impact in application development efforts. In this activity contingency plan scenarios are developed, which are then simulated in the contention plan and impact model scenarios fashion. This activity results in partial risk archetypes, which can be used to identify risks and prepare their resolution strategies. The identification of induced risks activity studies the relationships among the current risk archetype and other previously defined archetypes. In this activity, the risk management team identifies similar risks to each risk archetype. Also, risks that are induced by each archetype occurrence are identified. The activity results in complete risk archetypes that can be retrieved from the project element under study by application development projects.

oped for the application determines its project elements. The objectives of the risk management process for application development are: • to determine the risks that may affect an application, according to its project elements; • to evaluate risk impact; • to select the most important risks; • to develop risk handling plans; • to monitor risk and project status; and • to execute contention and contingency plans. Fig. 8 presents the activities that compose the risk management process for application development. The process first activities resemble a waterfall life cycle, while further activities are executed in several iterations along the development process. The waterfall like process prepares a first version of the application project model, selects its project elements risks, and organizes their risk handling plans. The iterative activities continuously update project model with status information, evaluating project risks and action triggers along the development process. The first activity of the risk management process for application development is risk identification, which assesses the project elements used by the application project model to determine risks that may affect its development process course. These risks were previously documented as risk archetypes and associated to project elements by the risk management process for risk identification. In the risk identification activity, the risk management team consults the project elements presented in the application project model, retrieving their risk archetypes. Induced risks, indicated in the risk archetypes of the project element, are also retrieved. The project manager can also navigate through similar risk archetypes and their occurrences, selecting other archetypes.

3.2. The application risk management process The application risk management process reuses previously created risk archetypes during an application development. The process explores the project elements used by the application, tracing to potential problems associated with these elements. A project model devel-

Fig. 8. The application risk management process.


Still within the risk identification activity, the project manager answers the retrieved risk archetypes’ identification checklist. These answers are registered in risk handling plans, which are application level documents that convey project specific risk archetype information. For instance, risk archetypes offer checklists for risk identification, but risk-handling plans contain the answers given by an application project manager to that checklist. Beyond risk identification, risk-handling plans convey the precise strategy selected to contain or resolve a risk, based on contention and contingency plans defined in risk archetypes. Also, risk-handling plans are linked to risk archetypes in the later known cases list, where archetypes point navigation links to risk handling plans, indicating previously developed applications that have faced the risk they describe. The activities following risk identification in the risk management process for application development fulfill risk handling plans information. Within the risk evaluation activity, the risk management team estimates risk impact over project behavior using risk impact scenarios presented in the selected risk archetypes. The team can evaluate the impact of several risk impact scenario combinations. This feature allows the process to capture diversification effects, where one risk may compensate or intensify other ones, reducing or raising the overall project risk level when both risks are taken together. Each perceived risk impact over a relevant project variable is annotated in the risk handling plans. After every risk impact evaluation, risk-handling plans are ordered by their impact. Finally, the major risks are selected, while the remaining risk handling plans are discarded. This selection reduces the resources needed to plan, monitor, and track risks throughout the development process, focusing the existent resources to the project major risks. The number of risk handling plans taken to the next activity depends on project resources dedicated to risk management. In the risk planning activity the team selects strategies to counter these risks and defines triggers to indicate when tactical actions are to be executed. The team analyses the answers to the risk archetypes checklist, selecting the contention plans provided by the archetypes according to these answers. Two triggers are defined for each risk: an early-warning and a late-warning threshold. Both triggers are based on project model variables, which are monitored throughout the development process. The risk tracking and risk resolve activities form the iterative part of the risk management process for application development. In the risk tracking activity, the risk management team inserts new project status information into the application project model, updating its behavior. As the behavior changes, risk evaluation and contention triggers threshold verification are required. Risk plan triggers are recalculated, and, when early or

9

late warning thresholds are reached, risk impact is reevaluated and contention plans are activated. The risk resolution activity is executed when a risk occurs in the project. The impact of contingency plans over project behavior is evaluated within this activity. Adequate contingency strategies are also selected to be activated in the project in risk occurrences. Finally, the update archetypes activity enriches the risk knowledge base, providing risk archetypes for problems that were not perceived in previously developed applications. Within this activity, the development team refines existing risk archetypes, indicating innovative contention and contingency plans, and extending the archetype known cases list with the current application. The activity is executed in a post-mortem fashion, after the application development process.

4. Experimental analysis We have conducted two empirical studies regarding the proposed techniques. The first one was an observational analysis of applying the risk management framework while developing projects for a specific domain in an industrial setting. The second, executed in an academic environment, was a feasibility study to evaluate the use of scenario models and project simulation to support decisions while managing a software project. The following sections describe these experimental studies and summarize their results. 4.1. Analysis of the risk management framework This section presents an example of how the proposed risk management processes can be applied in real world projects. The results presented here derive from a study using both risk management processes in two projects developed within the same application domain. The selected domain was the financial market risk management. The selected projects convey the specialization of a risk management software system for the investment, debt, and operational portfolios of two large-size Brazilian enterprises. Before the accomplishment of the first project, there was little information about the risks related to the development of a risk management software system for a Brazilian enterprise. The team selected for the project had previous experience in implementing a financial market risk management software system for an investment bank. However, the specialization of such software for an enterprise presented several differences: • Risk horizons are very different: in general, investment banks are interested in short-term market fluctuations, such as the variations of exchange rates

ARTICLE IN PRESS 10


and equity prices in a single day. They want to know what will be their potential loss due to market variations from today to tomorrow. On the other hand, enterprises have their medium and long-range debt portfolios in foreign countries. Such portfolios extend for multiple years and are dependent on several future exchange rates. Enterprise are more interested in what will be their profit variation due to market fluctuations along one fiscal year or semester. • Commodities: by the time of the first project, the investment bank had little interest in commodities, such as coffee, gold, petroleum, or steel. However, these commodities have great importance for enterprises that sell such products in an open, global market. Since the market dictates commodities prices, the company profit or loss is subjected to market prices variations. Even if the company sells its products only in local markets, it may need to import incomes for its production lines. Future markets, such as London or Chicago Mercantile Exchange, can also regulate those incomes. These are only two examples to show the different visions of risk management for an investment bank and an enterprise. The first project in an enterprise was susceptible to some risks related to these characteristics, along with some generic software development risks. The domain-dependent risks were documented as risk archetypes and associated to the financial market risk management domain. Although a complete description of their risk archetypes is beyond the scope of this paper, short examples of such risks are shown below: • Commodities modeling: due to their sazionality and dependence on external factors, such as weather conditions, provider organizations, plagues, and diseases, the uncertainties associated to specific commodities are hard to model. Complex stochastic processes describe the behavior of some commodities, such as petroleum and gold. Measuring risks depending on such processes is also hard, specially in commodities derivatives. • Monte-Carlo based risk evaluation: Monte-Carlo simulations are frequently used to measure risks when financial market instruments cannot be described by closed-form equations. While they represent a very flexible and reliable risk evaluation framework, Monte-Carlo simulations require a lot of computing resources, slowing down risk evaluation. The dependence on such simulations for large portfolios can reduce the risk management software quality, since the user might wait too long for every analysis he decides to perform. • Portfolio composition volatility: denoted here by the frequent inclusion of new types of financial market instruments in an enterprise portfolio. Portfolio com-

position volatility is a domain-dependent facet of the requirement volatility risk. Every time a new instrument is inserted to the risk management software, its risk mapping must be defined (designed and coded), and maybe new risk factors and historical information needs to be added. Even without a formal assessment of its risks, the first project was rated as a success, being concluded only one week behind its four-month schedule. The project was divided in four major phases: portfolio elicitation, statistical analysis, portfolio mapping to risk factors, and risk mapping evaluation. Risk archetypes documentation, accomplished by executing the risk management process for risk identification, was performed in a post-mortem fashion, after the first project conclusion. Three (3) risk archetypes were created by using this process, one for each item presented in the preceding list. The second project executed the risk management process for application development, using the three risk archetypes created after the first project as risk warnings. The second project was a bigger one because the company had several operational portfolios, distributed along its subsidiaries and differing on the products it sells, buys or just distributes. It was scheduled to take six months. Again, the project was concluded one week later, but we believe it would be much more behind schedule if it were not the risk archetypes and the experience gained from the first project. Since just one developer took part in both projects, risk archetypes represented the major source of knowledge about potential problems and solutions to minimize the effects of their occurrence. For instance, when the portfolio elicitation team (mainly composed by financial engineers) came to the risk mapping team (mainly composed by designers and programmers) with a list of market instruments, the risk archetypes helped the developers to pinpoint instruments that could represent risks to the project. These risks were due to the need of sophisticated evaluation techniques (such as Monte Carlo simulations or dynamic programming) to precisely model such instruments. These time-consuming techniques could reduce project usability, since the user would have to wait a long time for each analysis, thus reducing his willingness to test several different portfolio allocation strategies. One risk archetype developed in the first project suggested that the portfolio analysts should decompose such instruments, replicating their behavior from basic market instruments where possible. Monte-Carlo simulations were provided for fine-grained analysis as a separate tool, while good closed-form approximations were developed for day-today risk monitoring. We believe that, without the advice provided by the risk archetypes, portfolio analysts would insist in a single, precise and slow evaluation


technique (reducing project usability), instead of a scalable refinement approach. The second project created five (5) new risk archetypes, which again were documented in a post-mortem fashion. The present study has shown us the usefulness of risk archetypes to document project management experience and the need for tools to improve the proposed techniques applicability. Both the first and the second projects have not used scenario models to evaluate risk impact over project behavior. However, both projects were tracked and activity duration, error generation, and portfolio volatility information are available for scenario development. We intend to accomplish another project in the same domain, this time using dynamic models. 4.2. Analysis of applying scenarios to management After establishing the basis for scenario and project model development, integration and simulation, we chose to execute an empirical feasibility analysis before spending effort to refine the proposed techniques. In this section, we summarize the results of this study. In a feasibility study, data is collected according to some experimental design, but full control over all possible variables is not achieved (Shull et al., 2001). This kind of empirical study usually aims to provide researchers with enough information to justify continued improvement of the techniques under analysis. In the current study, we emphasized functionality and usefulness, setting usability as a secondary goal. Also, we focused on applying scenarios for project management, instead of building new scenarios, because we wanted some evidence of scenario models efficacy before investing effort in better scenario and project model development tools. The feasibility study reported below was accomplished during a graduate software engineering course in the winter 2001 at the Systems Engineering and Computer Science Department of the Federal University of Rio de Janeiro (COPPE/UFRJ). 4.2.1. Subjects The subjects were students of the referred course, students from a graduate software engineering program and students from an undergraduate computer science department. Out of the 18 subjects, 13 were master degree students, three were doctoral students, one holds a recently achieved master degree, and one was an undergraduate student. Eight subjects had been project leaders in industrial projects, while three subjects had been leaders in academic projects, and seven subjects had not been project leaders neither in academy nor in industry. From the last seven subjects, three had participated in industrial projects, three had developed

11

software as part of coursework, and one had developed software only for personnel uses. 4.2.2. Objective To observe whether managers who applied the integration and simulation techniques to support their decision would perform better than managers who relied only on their experience and other techniques. The selected performance criterion was time to conclude a project (TCP): managers were asked to conclude their project in the least time possible. It is important to acknowledge that time to conclude a project may not be the most important aspect for a software project. However, it was selected as performance criteria because we needed a quantitative metric to compare the participant’s performance (others could have been chosen, such as project cost or project quality). Our null hypothesis states that the average TCP of subjects using the proposed techniques is equal to the average TCP of subjects not using these techniques. Our alternative hypothesis states that, provided that correct scenarios are considered, the average TCP of subjects using the proposed techniques is below the average TCP of subjects not using these techniques. Also, we intended to qualitatively evaluate the feasibility and usefulness of the proposed techniques. 4.2.3. Organization The study was planned as a single object, multi-test experiment (Wohlin et al., 2000). The subjects were randomly assigned into two groups: one to apply the proposed techniques and one to serve as a control group, accomplishing the study without the aid provided by the techniques. Two doctoral students and seven master degree students composed the first group. The remaining subjects were assigned to the second group. Five subjects from the first group had leadership experience in academic or industrial projects. Six subjects from the second group also had such experience. 4.2.4. Procedure Each subject was asked to manage a software project, without the intervention of other subjects. The project to be managed aimed to develop part of the Systems Engineering and Computer Science Department’s academic control system (CtrlPESC). The CtrlPESC system manages information about professors, students, research areas, disciplines, and course registrations. It is a small system, comprising about 67 adjusted function points and six major entities in its high-level design. Subjects from the first group were trained to use the proposed techniques and the simulation environment, while subjects from the second group only received a project debriefing. Since we intended to run a feasibility analysis, the cost of executing a real project with similar characteristics (team formation, team domain knowledge,

ARTICLE IN PRESS 12


activities to be performed, among others) for each subject was prohibitive. So, a project emulator was used. The emulator is a software system that controls and presents to its user a project’s behavior, adjusting it according to decisions taken by the users. The emulator dictates project behavior according to a model and a random generation engine, which defines the duration for each project activity (durations are represented stochastically in the model). Due to the engine’s stochastic features, each subject observed distinct activity durations. Decision points where subjects could act included: determining which developer to assign for each project activity, deciding how much time to spend in quality control activities, and defining the number of hours each developer should work per day. The subjects from the first group could test the project sensibility for their decisions in the simulation environment before applying them to the project emulator. Time was not a constraint on decisionmaking for both groups: the subjects spent as much time as they desired in analysis before applying their decisions to the project emulator. After applying their decisions, the subjects could instruct the emulator to advance project time by several days, observing the effectiveness of their decisions upon the project behavior. 4.2.5. Materials All subjects received the project emulator, conveying a process for the CtrlPESC project and the descriptions of a set of developers that could be arranged to take part on the software project team. All subjects received a brief description of the project. Subjects applying the techniques under study also received the simulation environment, a project model, and a set of scenarios models, built from knowledge presented in the technical literature, mostly from (Abdel-Hamid and Madnick, 1991) and (Jones, 2000). 4.2.6. Data collection Questionnaires were used to collect qualitative data that characterized each distinct subject and directly addressed the questions of feasibility and usefulness. The qualitative data included open-ended questions concerned with opinions about the techniques’ effectiveness, measured by confidence on whether subjects could conclude the project in less time without their aid, and a subjective evaluation about the techniques’ usefulness and the difficulties regarding the interpretation of their results. The project emulator gathered quantitative information about each subject performance, that is, the time that each subject took to conclude the CtrlPESC project, measured in days. 4.2.7. Outlier elimination After collecting quantitative performance data, we have conducted a two-phased outlier elimination pro-

Table 1 Time taken by each subject to accomplish the example project (after extreme value elimination) Subjects applying the techniques 25 days 30 days

38 days 25 days

Subjects not applying the techniques 25 days 27 days 27 days

42 days 35 days

28 days 31 days

37 days 33 days

Table 2 Analysis results for the experimental study Time to conclude the project

Subjects applying the techniques

Subjects not applying the techniques

Average Maximum Minimum Standard deviation

28.1 days 38.0 days 25.0 days 4.7 days

34.3 days 42.0 days 28.0 days 4.9 days

cedure. First, a qualitative analysis eliminated one subject who applied the proposed techniques (46 days) because the subject declared he committed errors during the study. Then, a T distribution-based cut was used to quantitatively eliminate extreme values (Freund and Perles, 1998). This type of extreme value elimination procedure is useful when we have small samples and we cannot assume that the population is normally distributed. Table 1 presents data points after extreme value eliminations. Eliminated values are crossed and italiced. 4.2.8. Analysis Since we are interested in verifying differences among averages, we selected average evaluation statistical techniques to analyze the results from the experimental study. The data in Table 1, except for the outliers, was submitted to a 95% T -test 1 (Freund and Perles, 1998) that showed that the average time to conclude the project for subjects applying the proposed techniques is less than the average time taken by the ones who did not apply the techniques. A Mann-Whitney (Wohlin et al., 2000) rankbased statistical analysis asserted the T -test results. Table 2 summarizes the analysis results. 4.2.9. Result and lessons learned The quantitative data from this study provides some indications that, given correct scenario models, subjects using the proposed techniques performed better than subjects not using them. Also, all subjects from the first group agreed that the techniques were helpful. Thus, the data and positive results drawn from it show some indications that the techniques are feasible and that they 1 A T -test is a statistical analysis procedure based on the T distribution that compares if the averages of two groups are the same (to a degree of certainty, such as 90% or 95%) in the lights of their variances.


can provide help for project managers. The qualitative data indicates that the research upon the techniques should continue following well defined paths. The questionnaires returned by subjects from the first group indicate that three subjects had difficulties interpreting the results produced by the simulator. So, the simulation environment’s user interface should be improved. Also, though the experimental study yielded positive results, we found some points where it could be improved, including: • Allowing subjects from the first group to use the simulator in a ‘‘toy’’ project during the training session. The current training session included only a brief presentation of the simulator. Four subjects from the first group required more training to effectively use the proposed techniques and the simulator environment. • Providing training on how to use the project emulator for subjects from both groups. Since support from the study organizers was required by several subjects, a training section that would show subjects how to use the emulator would be beneficial. • Presenting the dynamics that govern the models used in the experimental study during the training sessions. One subject from the first group was confused with some technical terms used in the study, while two subjects from the same group were concerned about quality assurance activities. • Perfecting the mechanisms that present simulation results to the user. Three subjects from the first group proposed improvements for the simulation environment, mostly involving enhancements in simulation results visualization.

5. Related works Our approach to risk management has some innovations when compared to previous software risk management processes. First, it considers the relationships between risk management and particular project elements, such as application domains, technologies, software artifacts, resources and developer roles. These relationships are built upon risk archetypes, which are linked to project elements when created and retrieved when an application using such project elements is developed. Kontio and Basili (1996) propose risk information reuse. Althoff et al. (1999) propose mechanisms to organize, retrieve, and reuse software development experience, although they do not describe a risk management process. Managers can use such experience to predict problems early in the software development life cycle. In our approach, the risk management processes naturally embed software reuse, following a generic reuse frame-

13

work. The first process allows a project manager to document a particular risk or to analyze a specific project element to document these risks. The second process helps a project manager to identify the risks within an application, reusing the available knowledge from previously documented risks. Garvey et al. (1997) and Jones (1994) present standard information structures to document risks, but their information structures do not allow risk impact simulation within a specific project. Hall (1998) proposes several risk forms or databases to store static risk information. Again, this static information does not allow an integrated analysis of risk impact. We use risk archetypes to offer a structured way to document risk information. We expect that this structured documentation helps to break through psychological barriers to risk analysis, due to incomplete information about risks. Besides, they allow the integration and simulation of risks regarding a project model. Kontio and Basili (1996) quantifies the effects of risk scenarios using utility functions. However, these functions become very complicated when several process aspects are taken into account, due to nonlinear relationships among scenario elements and feedback loops inherent to the software development process. In our approach risk impact, contention and contingency plan effects can be aggregated to project models in a way that they can be evaluated. Such evaluation captures the possibly nonlinear risk and plan impact over the process. For instance, consider a risk whose outcome is to reduce a project team by a predefined amount of developers. This team reduction effect can be directly evaluated from the project model, capturing all the relationships with project current status and other model variables. Risk impact evaluation through simulation offers a better way to understand the cascade of impacts imposed by the risk upon project behavior. Scenario models can also be compared to other research efforts focusing on the creation of reusable artifacts for project management. Most notably, Henninger (1996) proposes an evolutionary organizational learning approach where domain knowledge is built from cases that document project experiences and design artifacts. Recurrent cases are progressively refined and translated to domain knowledge as they appear in projects. Like risk archetypes, cases are documented by standard structures that resemble design patterns. Though Henninger’s work is close to our proposal, it does not provide automatic interpretation for cases, since they are described in natural language. Scenario models can be formally evaluated through simulation when integrated to a project model. Such analysis provides means for interpreting the effects of applying the reusable knowledge (scenarios) upon a specific context (project model).

ARTICLE IN PRESS 14


Althoff et al. (1999) propose mechanisms to organize, retrieve, and reuse software development experience. Managers can use such experience to predict problems early in the software development life cycle. Williamson et al. (2000) present a formal representation for engineer’s knowledge and design rationale. The proposed representation is used to document software components assertions and constraints, being tied to the components stored in a reuse library. Although the representation is formal and unambiguous, the authors present no mechanisms to integrate it to an incomplete project design in order to evaluate component fitness to the project.

supporting project model definition, scenario development, scenario integration, and a simulation environment.

Acknowledgements The authors would like to thank CNPq and CAPES, for their financial supporting, Dr. Shari Lawrence Pfleeger for her insights, Dr. Victor Basili for his comments and the subjects of the experimental study for their valuable contribution.

References 6. Final considerations This paper described an approach to develop, retrieve, and reuse project management knowledge and experience. Such knowledge is represented in scenario models, which are formal models of management actions and theories. We used this scenario-based approach to model risk impact and resolution strategies efficacy within risk archetypes when applied to software development projects. A risk archetype is an information structure that holds knowledge about software development risks allowing its reuse. The main contribution of this paper is to show how risk archetypes and scenario models can represent reusable project management knowledge. Scenario Based Project Management paradigm raises model abstraction level, breaks down treatment uniformity of same category elements, and separates facts from assumptions within the model. These features improve previous system dynamics software project models. We also present a risk management process that uses risk archetypes and scenario models to identify and evaluate software development risks. The proposed process resembles a reuse process framework, where two sub-processes are respectively responsible for identifying and reusing risk information. Two experimental studies concerning the proposed techniques were discussed. Scenario Based Project Management is an on going work. We have made some preliminary studies in two similar projects to understand the relationship between risks and application domain. However, since these studies were limited to small projects and a single domain, they are not conclusive, but motivating to continue the work towards a more general evaluation. Also, the second study evaluated scenario application to project management, but did not address the difficulties related to scenario development. Next steps regard the definition of more complex project models and to apply them to operational project management situations. After understanding the process, we also intend to finish the implementation of tools

Abdel-Hamid, T., Madnick, S.E., 1991. Software Project Dynamics: an Integrated Approach. Prentice-Hall Software Series, Englewood Cliffs, NJ. Althoff, K., Birk, A., Hartkopf, S., M€ uller, W., Nick, M., Surmann, D., Tautz, C., 1999. Managing Software Engineering Experience for Comprehensive Reuse. In: Proceedings of the 11th International Conference on Software Engineering and Knowledge Engineering, Heidelberg, GR. Barros, M.O., Werner, C.M.L., Travassos, G.H., 1999. Applying System Dynamics to Scenario Based Software Project Management. In: Proceedings of the 18th Conference of the System Dynamics Society, Bergen, NW. Barros, M.O., Werner, C.M.L., Travassos, G.H., 2001. From Models to Metamodels: Organizing and Reusing Domain Knowledge in System Dynamics Model Development. In: Proceedings of the 19th Conference of the System Dynamics Society, Atlanta, USA. Charette, R.N., 1996. Large-scale project management is risk management. IEEE Software 4 (13), 110–117. Forrester, J.W., 1961. Industrial Dynamics. The MIT Press, Cambridge, MA. Freund, J.E., Perles, B.M., 1998. Statistics: A First Course, seventh ed. Prentice-Hall, Englewood Cliffs, NJ. Garvey, P.R., Phair, D.J., Wilson, J.A., 1997. An information architecture for risk assessment and management. IEEE Software 3 (4), 25–34. Hall, E.M., 1998. Managing Risk: Methods for Software Systems Development. SEI Series in Software Engineering, Addison-Wesley Longman Inc. Henninger, S., 1996. Accelerating the Successful Reuse of Problem Solving Knowledge Through the Domain Lifecycle. In: Proceedings of the Fourth International Conference on Software Reuse, Orlando, USA, pp. 124–133. Highsmith, J., 1992. Software Ascents. American Programmer Magazine (June). Jones, C., 1994. Control and Assessment of Software Risks. AddisonWesley Publishing Company, Reading, MA. Jones, C., 2000. Software Assessments, Benchmarks, and Best Practices Addison-Wesley Information Technology Series. AddisonWesley Publishing Co, Reading, MA. Klein, G., 1998. Sources of Power. MIT Press, Cambridge, MA. Kontio, J., Basili, V.R., 1996. Risk Knowledge Capture in the Riskit Method. In: 21st SEW Proceedings, SEL-96-002, NASA/Goddard Space Flight Center, MD. Available from http://sel.gsfc.nasa.gov/ website/sew/1996/topics/kontio_ p.pdf. Lin, C.Y., Abdel-Hamid, T., Sherif, J.S., 1997. Software-Engineering Process Simulation Model (SEPS). Journal of Systems and Software 38, 263–277.

ARTICLE IN PRESS M.O. Barros et al. / The Journal of Systems and Software xxx (2003) xxx–xxx Shull, F., Carver, J., Travassos, G. H., 2001. An Empirical Methodology for Introducing Software Processes. In: Proceedings of the Joint 8th European Software Engineering Symposium and 9th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Vienna, Austria. Tvedt, J.D., 1996. An Extensible Model for Evaluating the Impact of Process Improvements on Software Development Cycle Time. D.Sc. Dissertation, Arizona State University, Tempe, AZ. Williamson, K.E., Healy, M.J., Barker, R.A., 2000. Reuse of Knowledge at an Appropriate Level of Abstraction––Case Studies Using Specware. In: Software Reuse: Advances in Software Reusability. Proceedings of the 6th International Conference on Software Reuse, Vienna, Austria, pp. 58–73. Wohlin, C., Runeson, P., H€ ost, M., Ohlsson, M.C., Regnell, B., Wesslen, A., 2000. Experimentation in Software Engineering: an Introduction. Kluwer Academic Publishers, Norwell, MA.

15

Marcio O. Barros received the D.Sc. degree from the Federal University of Rio de Janeiro. He is a research assistant on software engineering at the Computer Science Department in the same university. His main research interest are modeling, simulation, and their application to project management, quality control, and risk management. Currently, he is working on how to apply these techniques to train software project managers. Claudia Maria Lima Werner received the D.Sc. degree from the Federal University of Rio de Janeiro. She is an associate professor of Computer Science (software engineering) at the System Engineering and Computer Science Department at COPPE/UFRJ. Her main research interests are software reuse and component based software development. Guilherme Horta Travassos received the M.Sc. and D.Sc. degrees from Federal University of Rio de Janeiro. He has spent two years as a visiting associate professor at the Department of Computer Science, University of Maryland. He is an associate professor of Computer Science (software engineering) at the System Engineering and Computer Science Department at COPPE/UFRJ. His main research interests are experimental software engineering, software engineering environments and OO software inspections.