Evaluating Support for Improvisation in Simulated Emergency Scenarios D. Mendonça New Jersey Institute of Technology
[email protected]
G.E.G. Beroggi Spring Analytica
[email protected]
Abstract Technological systems involving hazards are typically managed by experienced personnel guided by wellformulated, pre-determined procedures. These procedures are designed to ensure that operations proceed in a safe and cost-effective manner. Yet normal operations in these systems are exposed to unexpected contingencies that can require personnel to develop and deploy new procedures in real-time. Creative thinking in such situations is therefore necessary in order to prevent degradation of operations, particularly when there is potential for personal injury, economic loss or environmental damage. One approach to addressing these situations is improvisation. The research described here discusses a series of studies conducted to evaluate the efficacy of a computer-based system for supporting improvisation in simulated crisis situations. The design and implementation of the system are first discussed, drawing upon prior work in blackboard-based systems. The experimental design is then reviewed, followed by a discussion of how the studies were run using groups of emergency response personnel from the Port of Rotterdam in The Netherlands. The group task was to address unexpected contingencies in a timely fashion. A number of measures of group decision effectiveness and uniqueness are presented. Results of the studies suggest that availability of decision support may have had an uneven influence on solution effectiveness and no influence on solution uniqueness. Possible implications for the design of group decision support systems for improvisation are then discussed, along with a number of observations on conducting experimentally-based research on group improvisation.
1. Introduction Emergency response organizations have benefited from technologies for information storage and retrieval (e.g., global positioning systems for locating equipment) and for communications (e.g., cellular communications networks for management of field operations). Newer information technologies and techniques are being assessed for their potential to aid in emergency response decision making. However, in order to make effective use of these technologies, reasoning systems to support human cognition are required. The exposure of these systems to unplanned-for contingencies can require
personnel to develop and deploy new procedures in realtime. Creative thinking in such situations is therefore necessary in order to prevent degradation of operations, particularly when there is potential for personal injury, economic loss or environmental damage. One approach to real-time development and deployment of new procedures is improvisation. This paper discusses the design and evaluation of a computer-based system for supporting improvised decision making in simulated emergency response situations. The system’s design is informed by research on cognition in improvisation as implemented in blackboard-based computer systems. The impact of the system on group decision making in simulated emergency response scenarios is assessed by considering both subjective and objective measures of the effectiveness and uniqueness (or creativity) of the decisions made by groups. The paper proceeds as follows. Related work in supporting improvisation is first reviewed, followed by a description of the design of a system to support group improvisation in simulated emergency response situation. A series of experiments designed to evaluate the impact of availability of this system on group performance is then described, and the results of these experiments presented. The work contributes to ongoing research on improvisation by considering how group improvisation may be supported and evaluated in simulated emergency response situations.
2. Background and Related Work Emergency response relies on successful execution of one or more contingency plans, often managed by a command and control center. A commander at the scene coordinates the activities of the units fighting responding to the emergency. The on-scene commander and support staff gather and analyze data, make decisions, and monitor their implementation and consequences. The activities required to respond to an incident are often dangerous and must be performed under time pressure. The decision to activate emergency plans is based upon assessment of the potential impacts of an accident and the courses of action needed to eliminate or at least mitigate this impact. These contingency plans can rarely be executed as expected, as the case of the Exxon Valdez accident showed [1]. Flexible approaches to emergency management are therefore required. Any such approach
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
W.A. Wallace Rensselaer Polytechnic Institute
[email protected]
must be able to deal with an uncertain and changing environment and allow for revision of planned courses of action. Moreover, the approach must be able to support emergency managers in real-time development and deployment of new courses of action when no standard operating procedure can prevent or even mitigate catastrophe. One approach to real-time development and deployment of new procedures is improvisation. Improvisation has provided a rich set of ideas and orienting principles for practical and theoretical work in a variety of domains for over twenty years [2]. Moorman and Miner ([3], [4]) offer a descriptive model of the influence of certain organizational factors, such as organizational memory, on the incidence and effectiveness of improvisation in new product development. Kreps and Bosworth [5] and Weick [6, 7] have described improvised decision making during emergency response operations. Much of this previous work on improvisation in organizations has agreed that, in order to understand how improvisation may succeed, researchers should study decision making in situations which require, or at least permit, some degree of improvisation (e.g., [8], [6]).
2.1.
Blackboard-based Systems
Hayes-Roth [9] and other researchers have created computational models of improvisation which have been used for supporting improvised decision making by humans. In a series of early experiments in human planning, Hayes-Roth and Hayes-Roth [10] observed that human planing may be multidirectional and opportunistic. It is multidirectional in that humans seem to plan at various levels of abstraction and seem to able to move between those levels, displacing some goals in favor of others. Planning is opportunistic in that "goals that fit into a developing plan are integrated, and goals that belong together are clustered into subplans, often without regard for how the subplans will integrate with the overall plan" [11]. Opportunistic planning is closely akin to "coordination by feedback" which takes place in Emergency Operations Centers [12], where decision makers must be prepared to make decisions based on feedback they receive from the field. Hayes-Roth and colleagues developed a series of blackboard-style architectures [13] based on these experiments [14]. There are three elements in a prototypical blackboard-style system [13]. The first is a set of knowledge sources that act as specialists in subdomains relevant to the solution of some problem. The problem definition, data available for use in solving it and proposed solutions to it are presented on the blackboard. Knowledge sources are triggered by blackboard contents and propose contributions to the solution of the problem.
Proposed solutions are then evaluated by a controller, which decides whether or not to post a solution to the blackboard [15]. Once a solution (or solution component) is posted, it becomes accessible to other knowledge sources. The controller may facilitate assembly of the solution. These elements of the system are depicted in Fig. 1 (adapted from [14]):
KS1 KS2
KSn
Blackboard
Figure 1. Prototypical Blackboard System The blackboard system architecture has been used to explore directed improvisation [16]— that is, the “simultaneous invention and performance of a new ‘work’ under the constraints of user-specified directions”—by software agents. The Virtual Theater, a collection of BB1-based software characters embodied as agents, is the testbed for the directed improvisation paradigm. The Guardian system has been tested on the tasks of monitoring and diagnosing emergency room patients in real-time and been favorably evaluated [11]. A blackboard-based system can be comprised of a team of humans and software agents operating to change and respond to the status of the domain. In the Virtual Theater application [17], a human provides high level directions to software agents (“characters”) whose task is to improvise a action while abiding by those directions. Directions may be abstract, such as “go to x.” A character considers behaviors that are consistent with the user’s directions and with the character’s perceptions of other characters’ actions. A realization of the preceding direction might be “hop to pedestal.” A human team member communicates via the blackboard system with other team members, whether they are real people or software agents. When communication is from software agent to human operator, the simplest type of operator control occurs when an action recommended by the system is rejected or approved. Higher levels of communication allow for provision of more general commands. In either type of communication, the blackboard performs the function of a group mental model since it holds decision makers’ representations and is shared by group members.
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
C O N T R O L L E R
2.2.
Group Decision Support in Simulated Emergency Response Scenarios
Evaluation studies of group decision support systems may be introduced into existing emergency response training programs to provide a pool of skilled participants to researchers and new sources of feedback to program administrators. Training is an integral part of maintaining vigilance among emergency response personnel and there are numerous programs throughout the world where training exercises are conducted regularly. Participants in these programs are typically emergency preparedness personnel from offices throughout the country. Gaming simulations are an educational technology frequently used in these training programs. The research discussed here was conducted at one such facility, located and administered by the Port of Rotterdam in The Netherlands. The Port of Rotterdam is one of the largest cargo and container ports in the world [18]. In addition to intense cargo handling activities, a large number of processing facilities and storage sites for hazardous materials are located at the port, including storage places for ammonia, chlorine, liquefied natural gas, and propylene. The area of the port falling within the hazard area of port activities is about 600 square kilometers and contains about one million people. The Rotterdam harbor area has developed a Regional Operational Base-Plan (ROB) to protect the physical and social health of the population. The two most important decision making authorities for ROB are the Command Place Incident (CoPI) and the Regional Operational Team (RegOT). CoPI members are commanders of the fire brigades, police, ambulance and hazardous materials squads (a spokesperson is also sometimes a member). One individual, usually the fire brigade commander, serves as group leader. CoPI members meet at a centralized location, usually a specially-equipped vehicle. The decision-making regime for emergency management is defined in the Coordinated Regional IncidentManagement Procedure (CRIP). CRIP has four alarm levels, corresponding to increased levels of severity in risk, and becomes active for incidents involving hazardous materials, large-scale technical emergency response, or any other accident where at least one of the CoPI members calls for a coordinated response to an incident. Port management is currently investigating alternative methods for supporting communication and decision processes. To this end, port management has acquired various advanced communications and information technologies, including a group decision support facility (GDSF), a multimedia authoring tool to design exercise scenarios, a large flat panel display for digital video conferencing, and several digital cameras to record incident scenarios and monitor the exercises. The incident
description will be embedded in a multimedia system and will include databases, animation, video, audio, and textual and graphical instructions. Feasible courses of action would be ranked according to decision makers' preferences under normal operating conditions. Unanticipated events could be generated at any time by the exercise staff, forcing the commanders at CoPI to assess the impact of these events and, if necessary, to alter the current courses of action. The port has also identified a need to train response teams in recognizing and adapting to unplanned-for contingencies. The port has therefore broadened its focus from training for SOPfollowing to training for skill in improvising.
3. System Design The research described here has involved collaborating with Rotterdam port management to design a series of field exercises using new information and communications technologies and training approaches. The focus of the current paper is on the evaluation of blackboard-based decision support for improvisation in simulated emergency response scenarios. The design of the system is intended to capture essential elements of the blackboard system. These are (i) knowledge which has been partitioned into knowledge sources, (ii) a controller that selects and assembles posted actions, and (iii) computer-specified partial solutions tailored to each knowledge source. These respective elements are provided by (i) populating each group with a representative from each emergency service and by designing the information so that collaboration is encouraged; (ii) ensuring that each group has a commander who make a final choice on which of the various proposed courses of action to execute; and (iii) working with domain experts or drawing upon other sources to develop solution components which can serve as the bases for improvisations. The system described here has been built using Macromedia Director software and implemented for use in a networked environment.
4. Experiment Design This stage of the research is intended to assess the impact of the availability of decision support in the form of recommended actions on both the effectiveness and uniqueness of group decisions in simulated emergency response situations. This section presents the design of the experiments and defines the specific measures of solution effectiveness and uniqueness, as well as qualifications of participants.. It should be noted that, in order to conduct the experiments properly, all material and communications were translated from English into Dutch (the participants’
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
native language). Doing so required close and frequent collaboration with port management and with numerous research assistants to ensure accuracy of the translations. Upon completion of the experiments, all Dutch-language data (i.e., audio and video recordings, computer logs, questionnaire responses and meetings notes) were professionally translated into English and transcribed before being analyzed.
4.1.
Participants
A vital component of this research is the inclusion of experienced emergency response practitioners as participants. Voluntary participation was solicited by port management officials through an internal memo sent to individuals from key emergency management services within the port. Five roles were to be filled in each group: commanding officer (CO) and representatives of fire brigade (FD), police department (PD), chemical advisor/hazardous materials (CA), medical office (MO) and port management (PM). A large number of personnel expressed interest in participating; thirty-nine actually participated. With some exceptions, each role was filled by an individual from the respective service. These exceptions arose in last-minute, sometimes harrowing, situations. At least two scheduled individuals could not participate since they were involved in responding to an actual emergency. Similarly, during another session, one of the participants had to leave the group and communicate by phone with field personnel responding to an actual emergency. When less than six participants were present, the group decided amongst themselves who was to take on an additional or substitute role. An example of taking on an additional role occurred in session one, when the CO took on the roles of CO and CA. An example of taking on a substitute role occurred only once, in session four, when a member of Port Management acted only as CA and not as PM. In total, there were nine sessions, six of which are included in this analysis. Sessions 3 and 6 had too few participants; in session 7 a miscommunication between experiment personnel and participants resulted in a non-participant being present in the experiment room and influencing participants’ decision making. Data from session 3, 6 and 7 were therefore discarded. In questionnaires submitted at the conclusion of the experiments, participants described their qualifications. The results are summarized in Table 1.
Table 1: Summary of Participants’ Qualifications, by Group Question Age Number of years in present position Number of years with present organization Number of years of experience working with computers Number of training exercises participated in Number of actual emergency responses involved in
26 20 21 24
15 11 20
8
12 7
10
9
13 10
9
9
8
3
9
7
130 3
2
41
5
46 54 7
Participants therefore had, on average, 20 years of experience at the port. Experience in actual emergency responses was fairly high (groups two and four each had a member who reported having participated in over 100 emergency responses).
4.2.
Procedure
The room were the experiments were conducted was designed to resemble the portable field office used during emergency responses that require coordination among numerous port services. At the port, the portable field office is known as the Command Center at the Scene (CCAS) and is typically staffed with one representative from each of the services involved, as well as a commander responsible for coordinating these services. The CCAS is usually equipped with portable phones, radios and supporting information such as chemical hazards manuals and phone books. For the experiments, participants convened in the round and had access to a flip chart, scratch paper and individual laptop computers (the introduction of laptop computers is seen as the next phase in the development of the CCAS). In addition to containing software for decision support, these computers were used to send messages to participants and to store their responses on the network server. During the experiments, only participants and one audio/visual technician were present in the experiment room. All experiment personnel were in a separate room out of participants’ view. The layout of the experiment room is shown in Figure 1.
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
Group 1 2 3 4 5 6 Avg. 51 44 52 46 41 46 46 6 15 6 10 3 4 7
50 minutes. It was therefore essential that participants account simultaneously for planning and execution times during phase two. A scale on the map allowed participants to estimate the travel time between points on the map. Participants worked on each phase until it was completed or time ran out. If participants completed a case before time ran out, the audio/visual technician signaled the coordinator to enter the room. An interface for Phase Two of Case One is given in the Figure 2. Figure 1: Layout of Experiment Room Participation in the experiments was entirely voluntary and was solicited through a letter sent from port management to all management-level personnel. Upon their arrival, participants were welcomed to the training facility, thanked for their participation and shown to seats in the experiment room. They were addressed first by a trainer from port services and then by a member of the research team. The trainer emphasized that the format of the experiment would differ from that of the training sessions to which all were accustomed. This was necessary since the trainer anticipated that participants would expect their customary training exercise format. The trainer then left the room and the coordinator discussed the elements of the computer interface, allowing participants to manipulate the interface and then answering questions. The introduction ended once participants seemed to be comfortable with the interface and could perform the requisite tasks for manipulating it (i.e., typing and mouse-clicking). Typically, this introductory step lasted about ten minutes. Participants were then told that they had been called in as consultants to another port on an emergency situation, that all information regarding the emergency would be presented on-screen, but that they might occasionally receive additional messages. They were shown a countdown clock on the wall (also shown on their laptop screens) and told to monitor the time remaining. The countdown commenced and the experiment began when each screen was set to the case description. The coordinator left the room at this point. Each group solved two cases and each case had two phases. In phase one, participants were told to plan for the activities necessary to address the emergency and were given 10 minutes to do so (in other words, they were not asked to consider the time to execute the activities they recommended). Phase one concluded once all activities had been submitted by all members of the group. Phase two began when participants were notified that certain resources available in phase one were not longer available, but that other alternative resources were available. Also, updated messages on the screen notified them that the situation threatened to escalate within 50 minutes. Finally, participants were told that activities in phase two had to be planned-for and completable within
Figure 2: Case One, Phase Two Cases One and Two are meant to differ in terms of the magnitude and type of event. While Case One concerns a cargo ship fire with oil spill, Case Two concerns a collision between two ships with resulting chemical emission. Similar mixes of resources are available in both cases. The cases are also similar with regard to measures such as the total number of sites (i.e., locations of resources), the distance from the incident location ("Z") to the most distant site, the number of resources available and the number of goals which participants were told to address. Certain elements of the cases are designed to disable the activation of standard operating procedures (SOPs). As an example, constraints on the use of many resources are present but are combined with requests for capabilities which could be best addressed by those resources. In other words, some of the resources thought to be best-suited to the emergency response were not available. These elements have been introduced in order to create a situation in which participants must improvise. Supported groups had recommendations available during phase two of each case. As discussed previously, recommendations were in the form of procedures which had to be assembled to form a solution. Participants elected either to accept, reject or modify these procedures. Unsupported groups received no assistance on either case. At the conclusion of phase two of each case, participants evaluated their decisions [4]. Once both cases had been solved, participants filled out a final questionnaire concerning their qualifications, the design
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
of the system and their satisfaction with group processes [19], [20]. Finally, the trainer and the coordinator conducted an informal debriefing with the group. The debriefing was intended to elicit participants’ opinions in a less structured way.
5. Results The principal results below concern both the effectiveness and degree of improvisation in group decisions between supported and unsupported groups. Additionally, communication patterns are also considered, along with participants’ self-assessments of the degree to which their prior experiences contributed to their decision making.
5.1.
Table 2: Excerpt from a Session Transcript Line Participa nt 1 CO 2 PD 3 PM 4 5
PD CO
Application of the Blackboard Model
The blackboard model posits a controller acting as a filterer and organizer of information [21]. As mentioned previously, one aspect of the model’s implementation in the experiments is in the designation of the commander as the controller of the system. Both supported and unsupported groups were similarly structured since availability of decision support was not intended to influence communication patterns. To determine whether this was the case, communication patterns are here investigated by considering CO participation in the group conversations. Conversation turn-taking data [22] for supported versus unsupported groups are for this purpose. The intention is to discover whether any significant differences in communication patterns, as measured by conversational turn-taking, arose between supported and unsupported groups. A conversational turn is said to occur when there is a change in speaker. Table 2 is an excerpt from a conversation in one session. The boundaries of a sequence are located at points where the CO speaks. The sequence in this excerpt therefore begins with PD at line 2 and ends with PD at line 4. The length of the sequence is 3; the number of participants in the sequence is 2.
Text No, we cannot see that here. Okay, I will just do that, fictionally. I would block the access to the terminal, and only allow people out. I have ten extra people for that. Then I will check the different locations with you. I have a number of points on my screen which are called O, P, J, K, L, M, N. These are not connected to a specific service. At location O there are oil booms, 500 meters, no vehicle or vessel, so you can use it.
Table 3 summarizes turn-taking data for supported versus unsupported groups on Cases One and Two. By considering length of sequences (denoted LS), it is possible to ascertain the extent to which the commander is involved in group conversations. Shorter sequences suggest greater involvement by the CO in conversations. The number of participants in a sequence (denoted NP) in a sequence is one measure of degree of participation by group members. Higher values of NP suggest that conversations involved more members of the group. Table 3: Sequences of Turn-taking Unsupported
Supported
Case 1 LS NP
3.21 1.58
2.72 1.38
Case 2 LS NP
3.53 1.44
2.81 1.43
LS for unsupported groups is significantly greater (p = 0.04) for unsupported than for supported groups. Differences in NP are significant only at p = 0.10. So, while approximately the same number of participants communicated to the group in unsupported and supported conditions, the CO took more turns (i.e., was more involved) in discussions among supported groups than unsupported ones. Differences in satisfaction with group processes [19] between supported and unsupported groups do not seem to explain the previous result. Responses to question 1 in Table 4 suggest that supported groups believed less
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
strongly than unsupported groups that time spent in problem solving was efficiently used. Table 4: Satisfaction with Group Processes Quest . 1
Unsup.
Sup.
1.8
3.3
2
1.5
2.7
Question Topic (1=agree — 7= disagree) The time spent in the exercise was efficiently used. Members worked together as a team.
These responses suggest that supported groups may have had reduced feelings of working as a team. This finding is quite counterintuitive unless it is considered that the presence of available solutions actually reduced cooperation by reducing the need to discuss possible actions.
5.2.
Group Performance
The results for group performance concern the effectiveness and uniqueness of group solutions. Because improvisation involves satisfying constraints as well as developing solutions that are unique or novel, both the uniqueness and effectiveness of solutions are considered. Performance of supported versus unsupported groups during phase two of each case is considered here, since this phase required groups to consider planning and executing times simultaneously. Solution Uniqueness To assess the uniqueness or novelty of individual solutions, participants were asked to assess the perceived level of improvisation for each case. Improvisation questions (taken from [4]) asked group members to assess the extent to which their actions were improvised. Table 5 summarizes responses to these questions. Table 5: Self-evaluation of Improvisation with Respect to Phase Two Actions Question Topic Rate the Action 1. Figured out the action as we went along — 7. Action followed a strict plan as it was taken 1. Improvised in carrying out this action — 7. Strictly followed our plan in carrying out this action
Case 1 Unsup Sup 3.4
3.9
4.0
3.4
Case 2 Unsup Sup 3.2
3.0
3.8
4.1
Differences between supported and unsupported groups were not statistically significant (α=0.10). However, the responses to these questions suggest that further investigation may be warranted in determining whether supported groups feel less strongly than unsupported groups that they were improvising. Solution Effectiveness A number of measures of solution effectiveness are used here. The first measures of effectiveness relate to planning and execution times. Planning time is defined here as the time between when a group receives the problem definition and when the CO submits the solution. At the start of phase two, each group was given 50 minutes in which planning and execution activities could occur. Consider a group that submitted its plan at the end of the tenth minute. Approximately 40 minutes would be available for executing this plan. Planning time for the group (that is, the amount of time spent by the group in developing the actions before submitting them to field commanders) would be 10 minutes. If all actions could be completed within 34 minutes (i.e., the max time used below), then the slack in the plan would be 6 minutes. The time to complete the shortest task is denoted min: this is the time until the first resource arrives at the incident location. Total time refers to the utilization of resources in executing the plan (i.e., the total amount of minutes). So, if the group proposed only two activities—a fire truck traveling 10 minutes from its station to the incident location, and a police car traveling 34 minutes from its location to the incident—then the value of total for this group would be 44 minutes. It is assumed that groups strive to solve the problem as quickly as possible (that is, to minimize planning time) and to bring resources to the scene as quickly as possible (that is, to minimize execution time). Table 6 summarizes these measures for unsupported and supported groups. Table 6: Planning and Execution Times Case 1, Phase 2 time allotted = 50 min.
Case 1, Phase 2 time allotted = 50 min.
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
max min
Unsup 34.0 14.5
Sup 33.5 15.5
total 94.5 planning 16.5 slack -0.5
159.3 22.8 -6.3
max min
29.5 6.5
24.5 7.5
total 82.0 planning 26.0 slack -0.5
115.3 19.8 0.75
where max = duration of longest course of action (minutes), min = duration of shortest course of action (minutes), total = sum of duration of all courses of action (minutes), planning = time spent on developing the plan (minutes) and slack = time allotted – (max +planning) (minutes). In Case One, groups two through five had negative slack times; in Case Two, groups two, four and six had negative slack times. However, the results suggest that supported groups required less time to plan for each minute of activity, as indicated by differences in total time between supported and unsupported groups (the difference is significant only for Case Two, with p=0.09).
6. Discussion and Conclusions This research builds upon previous, principally descriptive work in improvisation by proposing a system to support improvised decision making in simulated emergency response situations, and by applying existing and new measures to assess the effectiveness and uniqueness of improvised decisions taken either with or without access to this system. Availability of recommendations from the system may have influenced conversational turn-taking, suggesting that provision of recommendations may have helped increase participation by the coordinator. In a blackboard system, the blackboard contains a type of working memory for the group. Participants’ remarks in post-experiment debriefings are consistent with the notion that communications do not persist indefinitely on the blackboard. Nearly every group expressed a desire to be able to preserve communications from group members. For example, members of group six mentioned that they wanted a white board of some kind, perhaps even a view of the CO’s screen, to facilitate their discussions. Provision of such a capability might help researchers better model group working memory since, given the finite space of the board, not all communications posted to it would persist over time. It would of course be important in future work to characterize more fully both group improvisation and the contents of working memory in studies of this type. The present research suggests that such work is needed. Some post-experiment comments implied that availability of decision support in the form of partial solutions may overly focus creative thinking. As one participant from group 3 noted in speaking about the recommendations, “I think one automatically is inclined to adopt those things. ... So you have to be careful that the computer does not make all the decisions for you. You must continue to use your common sense, even if the inclination exists to deliver yourself to the technology.
This is how I experienced it, I believe that this happens in practice.” Similar comments were made by some members of other groups. A caution for designers of systems intended to support improvisation is that, by providing solutions (even partial ones), the incentive to think creatively may be reduced. This view is consistent with the observation that decision makers often need to trade-off effort and accuracy (see [23], [24]). Taken with the results of post-experiment questionnaires, reviewed previously, this result suggests that greater attention should be given to developing recommendations which stimulate creative thinking (see [25] and [26] for two recent approaches). Both the uniqueness and effectiveness of improvised decisions have been discussed. Results for uniqueness of the actions do not suggest differences which vary with the availability of support. Members of supported groups may have regarded their solutions as less improvised than did unsupported groups, perhaps suggesting that the mere presence of recommended activities might create the impression that solutions were not novel. Results for the effectiveness of blackboard-based decision support for improvisation were mixed. Some differences in allocations of resources were detected. With respect to planning, groups tended to consume nearly all (or more than) the available time regardless of whether or not they were supported. Availability of decision support may therefore have enabled more extensive planning rather than more immediate execution. The research presented here expands the scope of recent work on improvisation by considering the performance of groups of experienced personnel solving time-constrained tasks for which their expertise is relevant. The methods employed to analyze performance pertain both to participants’ views of solution novelty and to objective measures of solution effectiveness. It will likely be useful in future work to explore the impact— both in terms of solution effectiveness and uniqueness— of the availability of decision support for groups of differing levels of expertise. Acknowledgements This research was supported by the Port of Rotterdam and U.S. National Science Foundation Grant 9872699. We especially thank Daan van Gent and Vincent Bouter for their help in preparing the experiments. William A. Wallace and David Mendonça acknowledge the support they received from the School of Technology, Policy, and Management, Delft University of Technology, The Netherlands, where they were Visiting Professor and Research Fellow, respectively, when this research was conducted.
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
7. References [1]
[2]
[3]
[4]
[5]
[6] [7]
[8] [9] [10] [11]
[12]
[13]
[14]
J. Harrald, H. Marcus, and W. A. Wallace, "The Exxon Valdez: An Assessment of Crisis Prevention and Management Systems," Interfaces, vol. 25, pp. 14-30, 1990. M. Pina e Cunha, J. Vieira da Cunha, and K. Kamoche, "Organizational Improvisation: What, When, How and Why," Int. Journal of Management Reviews, vol. 1, pp. 299-341, 1999. C. Moorman and A. S. Miner, "Organizational Improvisation and Organizational Memory," Academy of Management Review, vol. 23, pp. 698-723, 1998. C. Moorman and A. S. Miner, "The Convergence of Planning and Execution: Improvisation in New Product Development," Journal of Marketing, vol. 62, pp. 1-20, 1998. G. A. Kreps and S. L. Bosworth, "Disaster, Organizing and Role Enactment: A Structural Approach," American Journal of Sociology, vol. 99, pp. 428-463, 1993. K. E. Weick, "Improvisation as a Mindset for Organizational Analysis," Organization Science, vol. 9, pp. 543-555, 1997. K. E. Weick, "The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster," Administrative Science Quarterly, pp. 628-652, 1993. M. Crossan and M. Sorrenti, "Making Sense of Improvisation," Advances in Strategic Management, vol. 14, pp. 155-180, 1997. B. Hayes-Roth, "A Blackboard Architecture for Control," Artificial Intelligence, vol. 26, pp. 251321, 1985. B. Hayes-Roth and F. Hayes-Roth, "A Cognitive Model of Planning," Cognitive Science, vol. 3, pp. 275-310, 1979. B. Hayes-Roth, "Architectural Foundations for Real-Time Performance in Intelligent Agents," The Journal of Real-Time Systems, vol. 2, pp. 99-125, 1990. R. R. Dynes and E. L. Quarantelli, "Organizational Communications and Decision Making in Crises," Disaster Research Center, University of Delaware, Columbus, OH, Report Series 17, 1977. H. P. Nii, "Blackboard Systems," in The Handbook of Artificial Intelligence, vol. 4, A. Barr, P. R. Cohen, and E. A. Feigenbaum, Eds. Boston: Addison-Wesley, 1989, pp. chapter 16. K. Pfleger and B. Hayes-Roth, "An Introduction to Blackboard-style Systems Organization," Knowledge Systems Laboratory, Stanford University, Stanford KSL-98-03, 29 July 1997.
[15] [16]
[17]
[18] [19]
[20] [21]
[22]
[23] [24]
[25]
[26]
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)
0-7695-1874-5/03 $17.00 © 2002 IEEE
B. Pomeroy and R. Irving, "A Blackboard Approach for Diagnosis in Pilot's Associate," IEEE Expert, vol. 5, pp. 39-45, 1990. B. Hayes-Roth, L. Brownston, and E. Sincoff, "Directed Improvisation by Computer Characters," Stanford University, Knowledge Systems Laboratory KSL-95-04, 1995. B. Hayes-Roth, L. Brownston, and R. van Gent, "Multiagent Collaboration in Directed Improvisation," presented at First International Conference on Multi-Agent Systems, San Francisco, CA, 1995. Port Information Center, "Port Statistics," Rotterdam Municipal Port Management,, pamphlet 1996. J. E. Bailey and S. W. Pearson, "Development of a Tool for Measuring and Analyzing Computer User Satisfaction," Management Science, vol. 29, pp. 530-545, 1983. R. M. Davison, "An Instrument for Measuring Meeting Success," Information and Management, vol. 32, pp. 163-176, 1997. N. Carver and V. Lesser, "Evolution of Blackboard Control Architectures," Expert Systems with Applications, vol. 7, pp. 1-30, 1994. H. Sacks, E. A. Schegloff, and G. Jefferson, "A Simplest Systematics for the Organization of Turn-taking for Conversation," Language, vol. 50, pp. 696-735, 1974. E. J. Johnson and J. W. Payne, "Effort and Accuracy in Choice," Management Science, vol. 31, pp. 395-414, 1985. G. E. G. Beroggi and W. A. Wallace, "Operational Control of the Transportation of Hazardous Materials: An Assessment of Alternative Decision Models," Management Science, vol. 41, pp. 1962-1977, 1995. E. L. Santanen, R. O. Briggs, and G.-J. d. Vreede, "The Cognitive Network Model of Creativity: A New Causal Model of Creativity and a New Brainstorming Technique," presented at Proceedings of the 33rd Hawaii International Conference on System Sciences, Maui, 2000. D. Mendonça and W. A. Wallace, "Development of a Decision Logic to Support Group Improvisation: An Application to Emergency Response," presented at Hawaii International Conference on System Sciences (HICSS-35), Big Island, HI, 2002.
