have been few systematic attempts to create such measures. Complicating ... Disaster or extreme event situations combine many elements thatâby definitionâ.
Chapter X Measures of Resilient Performance David Mendonça
Introduction In order to theorize, manage – even engineer – resilience, it is necessary that the factors that contribute to resilience be identified, and that measures of these factors be validated and exercised. Yet to date there have been few systematic attempts to create such measures. Complicating the matter is the fact that the resilience of safety-critical systems may only be manifested during actual operations. As a result, opportunities for controlled study (even systematic observations) on resilient organizations are severely limited. There is therefore a clear need to identify the factors that contribute to resilience, develop measures for these factors, and validate instruments for estimating the values of these factors. This chapter takes as a starting point a set of factors developed in prior research on organizational resilience. It then discusses an approach to refining and measuring these factors. The framework is then applied to the development and assessment of a candidate set of measures for the factors, using data drawn from observation of infrastructure restoration in New York City, New York, following the 11 September 2001 World Trade Center attack.
Defining and Measuring Resilience Among the definitions of resilience are an ability to resist disorder (Fiksel, 2003), as well as an ability to retain control, to continue and to rebuild (Hollnagel & Woods, 2006). Indeed, despite its relevance to the maintenance and restoration of system safety and operability, resilience may be a difficult concept to measure. For example, during system operation it may be possible only to measure its potential for resilience, rather than its resilience per se (Woods, 2006). The following factors are thought to contribute to resilience (Woods, 2006):
•
buffering capacity: size or kind of disruption that can be absorbed/adapted to without fundamental breakdown in system performance/structure
•
flexibility/stiffness: system’s ability to restructure itself in response to external changes/pressure
•
margin: performance relative to some boundary
•
tolerance: behavior in proximity to some boundary
•
cross-scale interactions: how context leads to (local) problem solving; how local adaptations can influence strategic goals/interactions Resilience engineering is “concerned with monitoring and managing performance at the boundaries of competence under changing demands” (Hollnagel & Woods, 2006). In seeking to engineer resilience, it is therefore appropriate to consider how these factors may be measured. Resilient performance (or the lack thereof) may arise out of need or opportunity, though the latter case is very rarely studied. In the former case, there are numerous studies of how organizations have dealt with situations that push them to the boundaries of competence. Disaster or extreme event situations combine many elements that—by definition— challenge capabilities for planning and response . Opportunities for examining resilient performance in response to extreme events are limited. First, there may be high costs associated with large-scale and nearly continuous observation of pre-event conditions. Second, the consequence of extreme events can include destruction of established data collection instruments, as occurred with the emergency operations center and, later, at the New York Fire Department command post as a consequence of the World Trade Center attack. Third, new processes, technologies and personnel brought in to aid the response may not be measurable with any instruments that remain available, as commonly occurs when the victims of an extreme event act as first responders, or when ad hoc communication networks are formed. A very real challenge in engineering resilience is therefore fundamentally methodological: how can organizational theorists and designers develop and implement measurement instruments for “experiments” which are essentially undesignable?
Broadly speaking, measurement may be defined as the “process of linking abstract concepts to empirical indicants” (Carmines & Zeller, 1979). It is worth emphasizing that this definition does not presupposed that measures are quantitative, merely that the linkage between abstract concepts and their instantiation in the real world be provided empirically. Intimately bound in any discussion of measurement in science and engineering are the notions of reliability and validity. Reliability refers to “the tendency toward consistency found in repeated measures of the same phenomenon” (Carmines & Zeller, 1979). In other words, a measurement instrument is reliable to the extent that it provides the same value when applied to the same phenomenon. On the other hand, an indicator of some abstract concept is valid to the extent that it measures what it purports to measure (Carmines & Zeller, 1979). In other words, a valid measurement is one that is capable of accessing a phenomenon and placing its value along some scale. Two types of validity are commonly investigated. Content validity “depends on the extent to which an empirical measurement reflects a specific domain of content. For example, a test in arithmetical operations would not be content valid if the test problems focused only on addition, thus neglecting subtraction, multiplication and division” (Carmines & Zeller, 1979). Construct validity is “the extent to which an operationalization measures the concepts it purports to measure” (Boudreau, Gefen, & Straub, 2001). More precisely, construct validity “is concerned with the extent to which a particular measure relates to other measures, consistent with theoretically-derived hypotheses concerning the concepts (or constructs) that are being measured” (Carmines & Zeller, 1979). Construct validation involves determining the theoretical relation between the concepts themselves, examining the empirical relationship between the measures and the concepts, then interpreting the empirical evidence to determine the extent of construct validity. With a few prominent exceptions, the path to instrument development is seldom discussed, thus providing little insight for researchers and practitioners about the validity and reliability of the measurements produced by these instruments. A common exception is the survey instrument, which is often to access attitudes and other psychological states that might be difficult to measure directly. Yet for quite some
time, a number of researchers have argued for increasing the portfolio of measures used in social science. For example, numerous unobtrusively collected measures may be of use in advancing understanding of organizations (e.g., Weick, 1985). In the early days of the development of theory for a new (or newly discovered) class of phenomena, the need for discussions of instrument development is particularly great. Without adequate attention to the assumptions underlying instrument development, the theory may become too narrow (or too diffuse) too quickly, leading either to an unnecessarily narrow view or to a hopelessly broad one. Resilience engineering is clearly a field in the midst of defining itself and its relationship to other fields, and this includes identifying and defining the phenomena which researchers in the field intend to investigate. Research in resilience engineering has been predominantly informed by field observations and not, for example, by laboratory studies. More to the point, research in the field has been strongly interpretive, focusing primarily on case studies. A study may be said to be interpretive “if it is assumed that our knowledge of reality is gained only through social constructions such as language, consciousness, shared meanings, documents, tools and other artifacts” (Klein & Myers, 1999). The types of generalizations that may be drawn from interpretive case studies are the development of concepts, the generation of theory, drawing of specific implications and the contribution of rich insights (Walsham, 1995). Principles for evaluating interpretive case studies may be used for establishing their reliability and validity, though the methods for doing so differ from those used in positivistic studies. Klein and Meyers (1999) provide a set of principles for conducting interpretive field studies, as follows. The fundamental principle—that of the hermeneutic circle—suggests that “we come to understand a complex whole from preconceptions about the meanings of its arts and their relationships.” Other principles emphasize the need to reflect critically on the social and historical background of the research setting (contextualization) and how research materials were socially constructed through interaction between researchers and participants (interaction). The principle of dialogical reasoning requires sensitivity to possible contradictions between theory and findings. Similarly, the principle of multiple interpretations requires sensitivity to differences in participants views, while the principle of suspicion requires sensitivity
to possible biases and distortions in those views. Application of the principles of the hermeneutic circle and contextualization yield interpretations of data collected in the field. The principle of abstraction and generalization requires relating these interpretation to theoretical, general concepts concerning human understanding and social action. In contrast to interpretive studies are positivist studies. A research study may be said to be positivist “if there is evidence of formal propositions, quantifiable measures of variables, hypothesis testing, and the drawing of inferences about a phenomenon from a representative sample to a stated population” (Orlikowski & Baroudi, 1991). There are some obvious challenges associating with a positivist approach to research in resilience engineering at this stage. For example, there are still very many contrasting definitions of resilience itself, as well as of the factors that are associated with it. Combining interpretive and positivist approaches seems a reasonable way to make progress in developing this new area of research, but few studies—at least in the social sciences—seek to do so, and indeed there are very few guidelines to lead the way. One approach is triangulation which may be defined as “the combination of methodologies in the study of the same phenomenon” (Denzin, 1978). There are two main approaches to triangulation: between (or across) methods, and within method (Denzin, 1978). Within-method triangulation “essentially involves cross-checking for internal consistency or reliability while ‘between-method’ triangulation tests the degree of external validity” (Jick, 1979). Triangulation can provide “a more complete, holistic, and contextual portrayal of the units under study,” though it is important to keep in mind that “effectiveness of triangulation rests on the premise that the weaknesses in each single method will be compensated by the counter-balancing strengths of another” (Jick, 1979). The remainder of this paper discusses a combined interpretive and positive approach to the measurement of factors associated with resilience, with a particular focus on the use of triangulation for improving measurement reliability and validity.
Identifying and Measuring Factors Affecting Resilience in Extreme Events Extreme events may be regarded as events which are rare, uncertainty and potentially high and broad consequences (Stewart & Bostrom, 2002). There are some immediately obvious reasons to study resilience in the context of the response to extreme events. Performance of organizations in situations is often at the boundary of their experience. It is conducted by skilled individuals and organizations, who must make high-stakes decisions under time constraint (Mendonça & Wallace, 2007a). On the other hand, the boundaries of experience may be difficult to identify a priori (i.e., before the event has occurred) and perhaps even afterwards. It is very likely that unskilled individuals and organizations will participate in the response. The decisions taken during the response may be very difficult to evaluate, even after the event. Finally, the long lag times between events—coupled with the difficulties involved in predicting the location of events—can make preevent monitoring impractical and perhaps impossible. When a disaster is sufficiently consequential (e.g., Category IV or V hurricanes, so-called strong earthquakes), public institutions may provide essentially unlimited buffering capacity in the form of personnel, supplies or cost coverage. On the other hand, non-extreme events that nonetheless test organizational resilience (i.e., those typically called crises) require that this buffering capacity reside within the impacted organization. In the extreme event situation, then, buffering capacity is essentially unlimited. The remainder of this section therefore offers preliminary thoughts on the measurement of margin, tolerance, and flexibility/stiffness (cross-scale interactions will be discussed briefly in the context of flexibility/stiffness).
Margin System boundaries may be said to represent both limits of performance (e.g., person-hours available for assignment to a task within the system) and the borders that separate one organization from the outside world (e.g., entry and exit points for the products associated with the system). For all but the simplest systems, multiple boundaries will be present for both types, requiring organizations to reckon their performance along multiple (sometimes conflicting) dimensions. Measuring the margin of a system, then, requires an approach that acknowledges these dimensions,
along with possible trade-offs among them. Given the nature of extreme events, as well as their ability to impact system performance, the dimensionality of this assessment problem poses considerable challenge to the measurement of margin.
Tolerance Like margin, tolerance refers to boundary conditions of the system. In this case, however, the concept describes not the performance of the system but rather how that performance is achieved: that is, how the people, technologies and processes of the system function. In measuring margin, a chief problem is paucity of data; in measuring tolerance, the challenge is to develop process-level descriptions of organizational behavior. For example, this might entail pre- and postevent comparisons of communication and decision making processes at the individual, group and organizational levels. Given the rarity of extreme events, cross-organizational comparisons may not be valid beyond a very limited number of organizations.
Flexibility/Stiffness The literature on organized response to disaster has shown the importance of planning (Drabek, 1985; Perry, 1991) to organizational capacity to respond to extreme events, but it has also shown that flexibility and an ability to improvise remain crucial in mitigating losses during response (Kreps, 1991; Turner, 1995). Indeed, the literature on emergency response is replete with examples of how response personnel have improvised social interaction (Kreps & Bosworth, 1993), behavior (Webb & Chevreau, 2006)and cognition (Vidaillet, 2001; Mendonça & Wallace, 2003; Mendonça, 2007) in seeking to meet response goals. Yet the measurement of flexibility and improvisation has been concentrated on product-related constructs, such as the perceived degree of effectiveness and creativity in the response. Only recently have there been attempts to develop process-related measures, and these are limited to cognitive and behavioral constructs. The final factor thought to contribute to resilience is cross-scale interactions, which relates closely to decision making and communication strategies, and therefore to the cognitive processes that underlie these strategies. Cross-scale interactions generally refer to within-organization interactions, though it may be possible that cross-organization
interactions are also relevant to resilient performance. Both of these latter factors are complementary. Flexibility/stiffness refers to organizational restructuring, while cross-scale interactions may perhaps be seen as a particular type of organizational restructuring, one in which new processes emerge during response, perhaps as part of organizational restructuring. Consequently, cross-scale interactions will be discussed in the context of flexibility/stiffness.
Resilient Performance in Practice As suggested above, development of concepts concerning the factors that contribute to resilience has progressed to the point where it is now appropriate to consider how these factors may be measured. This section reports on the development of measures for margin, tolerance and flexibility/stiffness as manifested during the response to the 2001 attack on the World Trade Center (WTC). As a result of the attack, there were extensive disruptions to critical infrastructure systems in New York City, leading to local, national and international impacts. Some disruptions were isolated to single systems, while others cascaded across systems, clearly demonstrating interdependencies that existed either by design (e.g., power needed to run subway system controls) or that emerged during and after the event itself (e.g., conflicting demands for common resources) (Mendonça & Wallace, 2007b). A number of studies have detailed the impact of the attack on critical infrastructure systems (O'Rourke, Lembo, & Nozick, 2003), as well as described some of the restoration activities of subsequent months. Damage to the electric power system was considerable, certainly beyond what had been experienced in prior events. This included the loss of 400 mega-watts (MW) of capacity from two substations which were destroyed following the collapse of World Trade Center building 7, and severe damage to five of the feeders that distributed power to the power networks. Indeed, five of the eight total electric power distribution networks in Manhattan were left without power. In total, about 13,000 customers were left without power as a result of this damage. Restoration of this power was an immediate high priority for the city and, in the case of the New York Stock Exchange, the nation. Within the telecommunications infrastructure, the loss of power impacted a major switching station, backup emergency 911 call routing and consumer telephone service, all located within the building housing
the center. The task of the company was to restore power to the building and recommence telecommunications services as quickly as possible. Taken together, these studies provide a means for understanding the link from initiating incidents (e.g., power outages), to disruptions (e.g., loss of subway service due to lack of power for signaling devices) and finally to restoration (e.g., the use of trailer-mounted generators for providing power to individual subway stations). The human side of both impacts and restoration, on the other hand, has not been nearly as well explored. Since resilience encompasses both human and technological factors, it is appropriate to consider how measures for both sets of factors may be defined and estimated in order to clarify the concept of resilience.
Method Data collection activities associated with both studies may be characterized as initially opportunistic, followed by stages of focused attention to salient sources. An initial concern was simply how to gain access to these organizations. Existing contacts within both industries, combined with the support of the National Science Foundation, were instrumental in providing initial entree. The briefing for the project was to study organized response in the restoration of interdependent critical infrastructure systems. At the time the data were being collected (beginning in late 2001), few studies had addressed the role of the human managers of these systems, instead concentrating on technical considerations of design and management. There were therefore few exemplar studies – and very little direct methodological guidance – on how to proceed in the study. Both studies therefore adopted a strongly interpretive approach to the evolving design of the study. Initial consultations with the companies responsible for the power and telecommunications infrastructures being studied were done in order to identify critical incidents, particularly those which involved highly nonroutine responses. Direct consultations were held with upper management-level personnel, who then contacted individuals involved with the candidate incidents in order to assess whether they would be able (or available) to take part in the study. This occasionally led to additional, clarifying discussions with management, usually to investigate expanding the respondent pool. A considerable amount of
time went into developing a respondent pool that spanned those levels in the organization that were involved in the incident. For example, study participants ranged from senior vice presidents to line workers (e.g., those who conducted the physical work of repairing the infrastructures). For the power company, the initial consultations led to a set of eight incidents. For the telecommunications company, various incidents were discussed, but only one could be investigated given the time commitments of interview subjects, many of whom were still deeply involved in other restoration activities. In hindsight, timely data collection was paramount to the success of both studies. From a practical perspective, data collected so soon after the fact were fresh – something particularly desirable for data drawn from human subjects. It also provided an opportunity for the study team to demonstrate that it could collect data without causing unreasonable perturbations in the work patterns of study participants. Data collection methods reflected both the goals of the project and the perspectives of the four investigators, two of whom were involved in the study of human-machine systems, and two of whom were involved in the technical design aspects of infrastructure systems. Discussions amongst the investigators produced agreement on the salience of core concepts from “systems engineering” (e.g., component and system reliability, time to restoration) as well as human psychology (e.g., planning, decision making, feedback) to the study. Given the range of core concepts, the points of contact at the companies were asked to request study participants to come prepared to discuss the incident, and to bring with them any necessary supplementary materials (e.g., maps, drawings). Suggestions on supplementary materials to bring were sometimes made by the points of contact and the investigators. A detailed protocol for the interviews was provided to these points of contact for review and comment. With a few exceptions, the Critical Decision Method (Flanagan, 1954; Klein, Calderwood, & MacGregor, 1989) was used for all interviews, with two interviewers and one or two respondents. One interviewer asked the probe questions (Klein et al., 1989); a second took notes (with two exceptions, it was not possible to audio- or video-record the interviews). The critical decision method (CDM) is a modified version of the critical incident technique (Flanagan, 1954) and, like other cognitive task analysis methods, is intended to reveal information about
human knowledge and thinking processes during decision making, particularly during non-routine decision making (Klein et al., 1989). It has been used in a wide variety of studies (see (Hoffman, Crandall, & Shadbolt, 1998) for a review). The five stages of the procedure were completed in all interviews (i.e., incident identification and selection; incident recall; incident retelling; time line verification and decision point identification; progressive deepening and the story behind the story). However, not all interviews were equally detailed. In practice— and following guidance in the use of this method—the choice of probe questions asked of respondents was determined mainly by study objectives, but also by exigency. For example, all respondents were asked whether the incident fit a standard or typical scenario, since the study was strongly informed by work on organizational improvisation, and plans may be highly relevant as referents for improvised action. On the other hand, probe questions concerning mental modeling were never asked, since the investigators had decided early on that formal modeling of the reasoning processes of respondents would not be feasible for the project. At other times, respondents simply did not have the time to commit to a full-scale interview. Most respondents appeared highly cooperative. The investigators emphasized throughout their discussions with points of contact and interview participants that logs of system behavior were vital to the study design, since these provided the closest possible approximation of the behavior of technical systems during response and recovery activities. Materials brought to interviews included system maps, engineering drawings, photos, field notes and meeting minutes. These materials were sometimes extensively used. In fact, interviews which did not include these materials tended to be less illuminating than those where they were used. When these materials were present, it was far easier to keep the interviews grounded in the lived experiences of participants. Finally, it should be noted that other logs were collected with the help of the points of contact. These were reviewed with company personnel for completeness and accuracy, and any identified deficiencies were noted. At the conclusion of each interview, participants filled out a brief questionnaire on their background and experience. A second questionnaire, adapted from work by Moorman and colleagues on improvisation by organizations (Moorman & Miner, 1998; Miner,
Bassoff, & Moorman, 2001), was used to measure organizational improvisation, organizational memory and the evaluation of the response. Finally, it should be noted that a small number of supplementary materials, such as newspaper and other reports from the popular press, were sometimes used by the investigators to provide context on the activities of the two companies. The distribution of these different types of data across the two studies is given in Table 1.
Results Data collection activities took place beginning in late 2001 and continued throughout 2002. In total, eleven in-dept interviews were conducted (10 for electric power, one for telecommunications), along with approximately 20 shorter sessions with approximately one interview subject per session. Other data sources are described below. Infrastructure
Organizational Units of Interview Participants
Data Sources
Electric Power
engineering emergency management electric operations energy services distribution engineering
interviews log data meeting notes after-action reports photographs drawings questionnaires
Telecommunications network operations
interview questionnaire after-action reports photographs
Table 1: Summary of organizational units and data sources in the studies
Overview of Restoration Activities The power company engaged in two inter-related strategies for restoring that power: connecting trailer-mounted portable generators to provide spot power; and installing temporary feeder lines – called shunts – to connect live networks to dead ones. The telecommunications company also relied upon trailer-mounted portable generators. An overview of these three critical decisions is presented
before illustrating the development and implementation of measures of margin, tolerance and flexibility/stiffness. For the power company, the loss of distribution capacity was far beyond the scale of previous incidents. Soon after the attack, the company began attempting to procure trailer-mounted generators in order to provide spot power to critical customers. By 12 September, it was clear that the amount of time and effort required to secure, install and operate these generators would be considerable. As a result, the company decided to create a Generator Group, comprised of individuals from various parts of the organization, which would have primary responsibility for work in this area. The second part of the company’s strategy was the use of shunts – cables with 13 kilovolt (kv) capacity – which were used to make connections between dead networks and live ones. This task was handled by existing units in the organization (such as Distribution Engineering and Electric Operations). Procedures executed by these units included determining shunt routes through the city and coordinating pick-ups (i.e., the actual connecting of the shunts to the networks). For the telecommunications company, the loss of power to the building would have triggered a standard operating procedure to connect a generator to the building via hookups in the basement. However, water and debris in the basement made this procedure unexecutable. A decision was then made to connect cable from the diesel generators directly to the floors which they were to power, but, according to interview respondent, “there’s no good way of doing that, because it’s all hard wired in an elaborate system of switches.” The solution required cutting riser cables above the basement and attaching them to portable generators with between one and 2.5 megawatt capacity. (Risers are cables within the building that are normally used to transmit power throughout the building.) The task of connecting the cables required considerable care in order to ensure that cables were properly matched. Generators were running by Friday, 14 September. A gradual transition was then made to commercial power, essentially resolving the incident (though generators remained on stand-by and were periodically tested). (See Mendonça, 2007 for a complete discussion of the case.)
Measuring Resilience Examining the initial definitions for margin and tolerance, it is clear that – in order to estimate these factors – it is necessary to identify system boundaries. In the electric power study, study participants and members of the research team offered numerous suggestions to comprise a set of boundaries. For example, staff utilization represents the extent to which personnel in an organization are utilized. Proxy measures for this construct had been monitored (e.g., sign-in sheets for on-duty employees), but ultimately could not be made available to the research team for reasons of employee confidentiality. Other examples include transmission system capacity, which represents the amount of power that could be delivered over existing infrastructure. A complete picture of transmission system capacity was not available, however. It was, however, possible to estimate the incremental contributions made to transmission capacity via the installation of generators shunts. A more sophisticated measured might combine capacity measures with estimates of anticipated load from customers. In the telecommunication study, system boundaries were considerably more difficult to discern. The main reason was the reliance of the study on a limited range of participants, but also on the highly localized nature of the incident: the case study concerned the restoration of power to a single building. It should also be mentioned that restoration activities were still being conducted during site visits by the research team, and therefore there were limits on the amount of time that participants could devote to supporting data collection. Resource utilization was discussed in terms of managing demand, since there was sufficient slack in the system to allow services that were normally provided through the facility to be provided through other facilities. The amount of load on the network was also discussed in this context. Factor
Power
Telecommunications
Margin/ Tolerance
Transmission capacity Network stability Network load Resource utilization
Resource utilization network load
Flexibility/Stiffness
Restructuring of organizational units Development of new procedures
Development of new procedures
Recognition of unplanned-for contingencies Identification of opportunities for renewal
Table 2: Candidate measures for factors contributing to resilience So, as with other extreme events, both margin and tolerance are difficult to evaluate since organizational boundaries are difficult to identify. In the power restoration case, a key observation is that the magnitude of the restoration problem far exceeded that of previous experience. Indeed, while generators had been part of previous restorations, the company had never before needed this quantity in such a short time. Using the available data for the generator strategy, it does appear that the path to restoration – as indicated by the cumulative number of generators restored to the network – followed an S-shape, similar to that of a prototypical learning curve. In the case of the shunt strategy, the number of feeder connections made per day suggests a straight path to achieving sufficient interim capacity. The nature of flexibility/stiffness in the power restoration case is suggested by its decision to create a new organizational structure—the Generator Group—almost immediately after the attack in order to manage generator procurement and use. The group was dissolved once the generators ceased to be a crucial part of the restoration plan. In other interviews (not discussed here), respondents stated that some existing organizational units improvised their roles, undertaking tasks that were within the capability of the organization but which were not in the usual range of activities for the units themselves. This phenomenon has been amply demonstrated in the response to many other events (Webb, 2004). Flexibility/stiffness is also reflected in the major restructuring of physical network, resulting in a new design for the distribution system (i.e., one using three larger networks instead of eight smaller ones). In contrast to the generator situation, there was no major restructuring of organizational units. In the case of telecommunications restoration, evidence of flexibility is found in the development of new procedures. For example, during the interview the manager emphasized the limited usefulness of plans during the response. He stated “If I’d had to go to anything other than my head or someone else’s it wouldn’t have worked. You don’t pull a
binder off the shelf on this one. You certainly wouldn’t grab a laptop and go into something.” Indeed, “no one to my knowledge when into a system quote unquote that gave them an answer in terms of what to do.” Yet on the other hand, he stated earlier in the interview that the decision to use diesel generators was made in a “split-second.” Similarly, the decision to connect the generators to the risers was “one of those decisions that truly took milliseconds. I said, OK we have to get the building risers – meaning the hard-wired cables – cut them and splice cables from the street into the riser.”
Discussion By considering these studies with respect to the principles evaluating interpretive field studies, recommendations may be made for how to proceed in future studies of power and telecom restoration. Many of the principles for evaluating interpretive studies given by Klein and Myers (1999) speak directly to some of the challenges involved in researching organized response to extreme events. For example, there were close interactions between researchers and subjects throughout the project. Indeed, some individuals in the organizations were both subjects and points of contact, and it was through interactions with these individuals that potential data sources and subjects were identified. Both companies were clearly interested in seeing the results of this work and looking for ways to apply them to their organizations. The principle of suspicion applied here to both sides of the relationship between researchers and subjects. For example, interim reports to key points of contact enabled both groups to look for evidence of bias or distortion. In practice, the principle of multiple interpretations can be difficult to follow in situations such as these, since there is a natural tendency in after-action reporting to construct coherent, even logical or linear narratives to explain the observed sequence of events. Finally, numerous personnel – particularly those who had been with the company for extended periods of time – discussed the relevance of prior experience in their efforts towards restoration. While the inclusion of these observations in the study results may improve the assessment of the study, in practice it was difficult to apply the principle of suspicion to these observations, since they drew upon incidents that had occurred tens of years earlier.
A variety of approaches to measuring the factors is evident from the cases and subsequent discussion. In order of decreasing granularity, they may be described as follows: •
Output measures that describe the resilient performance (e.g., mean time to restoration). These offer limited insights.
•
Measures that describe the impact of contextual factors on resilient performance. This approach is only useful to the extent that these contextual factors can be measured. It does not unveil process-level phenomena. Process measures that show the observed relationship between inputs and outputs, perhaps including explanations of the impact of contextual factors. •
Model-based explanations, which make ongoing predictions about the processes that translate (observed) inputs into (observed) outputs. Associated with any of these approaches are two threats to validity. First, post-event reports by response personnel are notoriously unreliable and potentially invalid, particularly when cognitive demands are unusually high. To achieve consistency (i.e., internal validity), it is often necessary to triangulate the observations of numerous participants. and almost certainly to give considerable weight to data collected concurrent with the occurrence of the phenomena. Second, the external validity is necessarily limited for all but a few cases. To achieve generalizability (i.e., some measure of external validity), we probably need to measure phenomena associated with these factors at a much lower level and then aggregate the results. As an example, begin with the study of individual processes and aggregate these – rather than beginning with looking for results at the group or organizational level.
Concluding Comments A number of observations from the conduct of this study may be used to improve the quality of further research into measuring factors thought to contribute to resilience. In the case of power restoration, margin and tolerance have been assessed according to the behavior of the physical system. Yet even with such assessments, it is difficult – perhaps even impossible – to evaluate performance on this case against some theoretical optimum. Even post-event assessments are
challenging, leading to the use of measures of relative performance or efficiency. Engineering estimates of anticipated system performance tend to be heavily informed by expert judgment rather than historical data (National Institute for Building Sciences, 2001). Evidence of flexibility in the power case is found in the company’s efforts at revising organizational structure, but also its activities and which people are assigned to the activities. The design of the physical system may have helped determine the organizational structure, a question that might be further investigated through other studies of physical change in infrastructure systems. Given the practical difficulties of developing and using measures that can be used in assessing – and eventually engineering – organizational resilience in response to extreme events, it is reasonable to plan on achieving understanding through the use of multiple methods. Given the importance of field studies to these efforts, the evaluation of results may benefit from an assessment of the methods of these studies to principles for evaluating interpretive field studies. This study has illustrated the possible benefits (and complications) of triangulating observations using both quantitative and qualitative methods. Yet it is certainly the case that a broader and more comprehensive range of observation techniques and analytic methods will be necessary in order to inform theory about how to engineer resilience in power and telecommunications infrastructures. Access to a wide variety of preand post-event data sources may facilitate this process, but this access must be negotiated through organizational gatekeepers – further emphasizing the need to embrace the principles discussed here. There are certainly opportunities for the development of technologies that may support measurement concerning organizational boundaries. Concurrently, better approaches must be developed for capturing data associated with low-level phenomena in order to support analysis of organizational resilience at a broad level.
References Boudreau, M. C., Gefen, D., & Straub, D. W. (2001). Validation in Information Systems Research: A State-of-the-Art Assessment. MIS Quarterly, 25(1), 1-16. Carmines, E. G., & Zeller, R. A. (1979). Reliability and Validity Assessment. Newbury Park, CA: Sage Publications.
Denzin, N. K. (1978). The Research Act: A Theoretical Introduction to Sociological Methods. New York: McGraw-Hill. Drabek, T. (1985). Managing the Emergency Response. Public Administration Review, 45, 85–92. Fiksel, J. (2003). Designing Resilient, Sustainable Systems. Environmental Science and Technology, 37, 5330–5339. Flanagan, J. C. (1954). The Critical Incident Technique. Psychological Bulletin, 51, 327-358. Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis. Human Factors, 40(2), 254-276. Hollnagel, E., & Woods, D. (2006). Epilogue: Resilience. Engineering Precepts. In E. Hollnagel, D. Woods, & N. Leveson (Eds.), Resilience Engineering: Concepts and Precepts. Aldershot, UK: Ashgate. Jick, T. D. (1979). Mixing Qualitative and Quantitative Methods: Triangulation in Action. Administrative Science Quarterly, 24(Dec.), 602–659. Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical Decision Method for Eliciting Knowledge. IEEE Transactions on Systems, Man and Cybernetics, 19, 462-472. Klein, H. K., & Myers, M. D. (1999). A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Quarterly, 23(1), 67–94. Kreps, G. A. (1991). Organizing for Emergency Management. In T. E. Drabek, & G. J. Hoetmer (Eds.), Emergency Management: Principles and Practice for Local Governments. Washington, D.C.: International City Management Association, 30-54. Kreps, G. A., & Bosworth, S. L. (1993). Disaster, Organizing and Role Enactment: A Structural Approach. American Journal of Sociology, 99(2), 428-463. Mendonça, D. (2007). Decision Support for Improvisation in Response to Extreme Events. Decision Support Systems, 43(3), 952–967. Mendonça, D., & Wallace, W. A. (2003). Studying Organizationallysituated Improvisation in Response to Extreme Events. Newark, NJ: New Jersey Institute of Technology. Mendonça, D., & Wallace, W. A. (2007a). A Cognitive Model of Improvisation in Emergency Management. IEEE Transactions on Systems, Man, and Cybernetics: Part A, 37(4), 547–561.
Mendonça, D., & Wallace, W. A. (2007b). Impacts of the 2001 World Trade Center Attack on New York City Critical Infrastructures. Journal of Infrastructure Systems, 12(4), 260-270. Miner, A., Bassoff, P., & Moorman, C. (2001). Organizational Improvisation and Learning: A Field Study. Administrative Science Quarterly, 46(June), 304-337. Moorman, C., & Miner, A. S. (1998). Organizational Improvisation and Organizational Memory. Academy of Management Review, 23(4), 698723. National Institute for Building Sciences (2001). Earthquake Loss Estimation Methodology HAZUS99 SR2, Technical Manuals IIII. Washington, DC: National Institute for Building Sciences. O'Rourke, T. D., Lembo, A. J., & Nozick, L. K. (2003). Lessons Learned from the World Trade Center Disaster about Critical Utility Systems. In J. L. Monday (Ed.), Beyond September 11th: An Account of Post-Disaster Research. Boulder, CO: Natural Hazards Research and Applications Information Center, 269-290. Orlikowski, W. J., & Baroudi, J. J. (1991). Studying Information Technology in Organizations: Research Approaches and Assumptions. Information Systems Research, 2(1), 1–28. Perry, R. (1991). Managing Disaster Response Operations. In T. Drabek, & G. Hoetmer (Eds.), Emergency Management: Principles and Practice for Local Government. Washington: International City Management Association, 201-224. Stewart, T. R., & Bostrom, A. (2002). Extreme Event Decision Making: Workshop Report. Albany, NY: University at Albany. Turner, B. A. (1995). The Role of Flexibility and Improvisation in Emergency Response. In T. Horlick-Jones, A. Amendola, & R. Casale (Eds.), Natural Risk and Civil Protection. London: E.&F. Spon, 463-475. Vidaillet, B. (2001). Cognitive Processes and Decision Making in a Crisis Situation: A Case Study. In T. K. Lant, & Z. Shapira (Eds.), Organizational Cognition: Computation and Interpretation. Mahwah, NJ: Lawrence Erlbaum Associates, 241-263. Walsham, G. (1995). Interpretive Case Studies in IS Research: Nature and Method. European Journal of Information Systems, 4(2), 74–81. Webb, G. R. (2004). Role Improvising during Crisis Situations. International Journal of Emergency Management, 2(1-2), 47-61. Webb, G. R., & Chevreau, F.-R. (2006). Planning to Improvise: The
Importance of Creativity and Flexibility in Crisis Response. International Journal of Emergency Management, 3(1), 66–72. Weick, K.E. (1985). Systematic Observational Methods, in G. Lindzey and E. Aronson (Eds.), The Handbook of Social Psychology, 567-634, Random House, New York. Woods, D. (2006). Essential Characteristics of Resilience. In E. Hollnagel, D. Woods, & N. Leveson (Eds.), Resilience Engineering: Concepts and Precepts. Aldershot, UK: Ashagate.
