Towards a framework for command post exercises

Heumüller et al.

Towards a framework for command post exercises

Towards a framework for command post exercises Erich Heumüller Sebastian Richter Universität der Bundeswehr München Universität der Bundeswehr München [email protected] [email protected] Ulrike Lechner Universität der Bundeswehr München [email protected] ABSTRACT

The paper describes work-in-progress of an action research approach investigating command post exercises in disaster response organizations. The empirical basis of our approach is three command post exercises in Germany. The main contribution of this paper is the framework for the conceptualization of command post exercises and an analysis of challenges in the conceptualization of command post exercises.

Keywords

Command post exercise, Exercise conceptualization

INTRODUCTION

Today’s disaster response missions require intensive collaboration, integration and synchronization of all organizations and agencies involved in every level of management (Hewlett et al., 2001). To coordinate forces and resources disaster organizations develop emergency plans and organize staffs. To validate plans and to prepare their staffs for mission they conduct exercises. In our research, we identify limitations of existing approaches in theory and practice and develop a framework to support the conceptualization of command post exercises. Especially as most exercises (at least these we observed and took part) are conceptualized by voluntary members of disaster response organizations and exercise planning is challenging and labour-intensive, this process should be assisted. Also, command post exercises are found to be difficult to plan and evaluate as staff processes are less tangible and unstructured – compared to traditional exercises at operational level. We develop an approach for practitioners to support the conceptualization of command post exercises. Our research question is: “How can a scenario-based command post exercise be conceptualized goal-oriented and evaluation-driven to cover demands of relevant stakeholders and to increase the benefit of exercises for the organization?” We propose as a solution a framework to guide command post exercises. This framework includes a systematic way to develop scenarios and to evaluate staff performance. It guides the process to design the exercise according to an exercise goal. Subsequently we describe theoretical background, method and introduce our framework.

THEORY

To allocate, plan, manage and supervise disaster response capabilities in disasters all different actors of the German disaster response system organize staff elements to command and control missions (Richter et al. 2010, Perry, 1995, 2003). Staff elements command and control on the basis of emergency plans, which are developed by disaster response organizations to address disaster mission preparedness (Payne, 1999, Hewett et al., 2001). Although these plans are designed and tested, they can never face every line of action possible during disaster development and are thus never complete (Perry, 2004). Staffs enable with timely decisions and situation based planning the whole disaster response system to be flexible and adaptive. Staff work requires a large and complex set of skills, functions and processes which are neither completely explored nor which are completely predictable in advance (Crichton et al., 2005, Perry, 2004). To help staffs to be prepared for their challenging task in disaster missions they are trained in staff exercises. Exercises facilitate several key functions crucial for disaster response as the validation and adaption of emergency plans (Payne, 1999, Hewett et al., 2001, Bartley et al., 2006, Callan, 2009, Klein et al., 2005, UNDHA 1993, Peterson & Perry, 1999), the evaluation of equipment and preparedness measures, practicing of processes, Proceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 1

Heumüller et al.


training of staffs and demonstration of disaster response skills (Callan, 2009, USDT, 2004; ODP, 2004, Peterson & Perry, 1999, Klein et al., 2005). To train staff work skills in a near realistic crisis environment, so-called scenario-based command post exercises are widely used instruments to exercise staffs and C2-systems (e.g. Friesen et al., 2010, Alexander, 2000, Hosseini & Izadkhah, 2010, Oregon, 1997). Through scenarios action is triggered. For effectiveness of training, it is crucial, to conceptualize scenarios appropriately (Alexander, 2000). It is crucial, to evaluate such exercises and to understand staff processes, to evaluate key functions, to analyze output of staffs and to conceptualize and measure input-output relations for staffs. Exercise design has to go hand in hand with scenario conceptualization and exercise evaluation. It is challenging to conceptualize scenario-based exercises suited for goal-oriented exercise evaluation (Potter & Farry, 2009). Guidelines for exercises existing in literature (BBK, 2011, BMI, 2008, Glass, 2006, Renner, 2001, Callan, 2009, Oregon, 1997, Ohio, 1999) typically focus on organizational aspects of exercises as media involvement, organizational arrangements or logistics or on testing and validation of emergency plans (Hewett et al., 2001, Bartley et al., 2006, Payne, 1999, Peterson & Perry, 1999). We could hardly identify support for the creative process of scenario development and for systematic exercise evaluation. Potter & Farry (2009) and Friesen et al. (2010) emphasize the need for an approach for scenario development for such exercises. We further identify a lack of evaluation instruments (see also Klein et al., 2005) and criteria to identify how beneficial an exercise is.

RESEARCH DESIGN

We follow in our method Action Research (Baskerville, 1999). Action Research is a dual method, which contributes to theory and strives to solve problems of an organization. Researchers collaborate and work in the field. We follow this method, because our collaboration partners asked us to help them with exercise conceptualization and evaluation and they wanted a scientific approach and scientific guidelines. The empirical basis of our research are three command post exercises (see Figure 1). We describe partners, actions and our involvement and the topic of the exercises below. Exercise GERETSRIED

Action/Involvement Scientific observer, Evaluation

GROßER KREIS

Scientific observer and adviser, evaluation

FEUERBALL

Methodological driver of exercise concept, scientific observer, evaluation

Data Sources Evaluation of existing evaluation reports, observations, discussions, one workshop, questionnaire on exercise organisation, checklists on performance observations, discussions, workshops, questionnaire on exercise organisation/success of exercise/own assessment of performance, checklists on performance observations, discussions, workshops, questionnaire on exercise organisation/success of exercise/own assessment of performance, checklists on performance, survey on own framework, interviews

Figure 1: Exercises, Actions and Data Sources

We observed and evaluated the THW exercise “GERETSRIED” (October 22nd-24th, 2010) to analyze the staffs’ performance. THW of Bavaria and its branch of the coordination and communication units are our research partner. The THW is a German federal technical relief organization under superior of the Federal Ministry of the Interior with the mandate to support technically in cases of civil defense or disasters. A coordination and communication unit is assigned to coordinate THW forces or forces of other disaster response organizations (Teuber, 2007). It provides disaster response forces with communication tools and sets up staff elements responsible for C2 functions. These staff elements are called “Führungsstelle”. We were involved in the exercise shortly before exercise and evaluated it. Topic of exercise E1 was the exchange of a staff on a Führungsstelle in an ongoing mission. We did exercise questionnaires and observations to collect data. Exercise “GROßER KREIS“ (July 15th/16th, 2011) was an exercise of Kreisverbindungskommandos in Mittelfranken, i.e., liaison units at the county level. Our research partner is a part of the disaster response organization of the Bundeswehr at the lower tactical level. These liaison units support cooperation of Bundeswehr and civilian crisis headquarters. They advise and coordinate forces (VDR, 2011). Liaison units are staff-organized and consist of reservists, usually officers. We observed and to some extent guided conceptualization, preparation, conducting and were responsible for the evaluation and wrote the evaluation report. We provided some guidance in the development of scenarios, the exercising concept and the planning of the exercise. Data collection was done via observation, analysis of artifacts and various post-exercise surveys (for exercise participants, referees, observers). The exercise topic of E2 was the training of general processes in a respective staff.

Proceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 2

Heumüller et al.


At the third exercise “FEUERBALL” (October 7th-9th, 2011), we were actively involved in conceptualization, conducting and evaluation for the first time. The topic of FEUERBALL was the management of an assembly area. Partners in this exercise were again the coordination and command units of the THW in Bavaria.

RESULTS

We develop our framework for exercise and scenario development on the empirical basis of three exercises. We argue that – in different organizations with different backgrounds – staff structure and staff work as well as the problems of the exercise planning teams are quite comparable (see for staff structure e.g. FwDV-100, 1999). The organizations aim to test the performance of their staffs but had difficulties to conceptualize adequate exercise settings and approaches to evaluations. Especially the scenario development process in exercise planning is laborious and hardly structured. The planning teams have problems to develop measurements to find out how well the different staffs performed and to implement these measurements in the scenarios of the exercise. In exercise GERETSRIED we identified problems in evaluation in general as exercise evaluation was not considered in exercise planning. During the exercise planning process neither measurements for observers (or referees) were developed nor criteria to assess staffs’ work results. We missed logical linkages between theory (regulations, standards, policies) and exercise injects. During the exercise participants developed adhoc new ways to solve challenges (triggered by injects). These new processes have not been documented by observers or exercise planning team and are lost for the organization. We analyzed exercise evaluation reports and found them rather narrative and to lack goal definitions, evaluation criteria and the link between goal and evaluation. In GROßER KREIS our findings are similar. Scenario development was laborious and the exercise planning team developed scenarios and injects until the exercise started. Criteria to assess whether scenarios and exercise conceptualization meet the goal of the exercise needed to be developed and this added to work load and also insecurities in the whole process. Evaluation methods and scales previously used are found to be very general (three color codes with red, yellow and green) with little systematic approach to evaluate the specifics of staff. In FEUERBALL our task was to support the exercise planning team to conceptualize the exercise and this was the first exercise to test our framework for staff exercises given below. The exercise planning team of FEUERBALL utilized our framework and we discussed their experiences in a workshop. We then refined our framework on the basis of the planning team’s feedback and literature reviews. This publication refers to the current version of the framework after this review.

Framework for Exercise Development Exercise Conception Definition of exercise goal and topic

Evaluation Instruments Exercise screenplay & Analysis Plan

Exercise Course

Exercise Evaluation

Scenarios

Process-related Evaluation

Evaluation criteria for observer checklists

Situational Evaluation

Sample solutions for injects and further single measurements

Situational evaluation

Process-related evaluation

Injects

Scenarios

Injects

Evaluation findings of exercise

Exercise Evaluation

products Guideline for Guideline and Template for Template for creation Template for exercise scenario specification framework for inject documentation of of observer checklists screenplay design and structured development executed injects description Template for Template for sample structured inject solutions for injects description

Figure 2: Framework for exercise conceptualization

Our solution to address the problems in staff exercise conceptualization is a framework for the conceptualization of staff exercises (see Figure 2). A staff exercise conceptualization is to be guided by exercise goal and topic. We distinguish three levels of exercise conceptualization and evaluation (big blue box in Figure 2) Each level contributes to the exercise design (left side) and evaluation (right side) with respective evaluation instruments (white box). Based on the exercise goal and topic, the “exercise course” specifies its “scenarios” and “scenarios” specify their “injects”. The “Exercise Course” represents the top level of contentual planning and includes e.g. the overall screenplay. “Exercise Evaluation” represents the top level of evaluation planning including an analyProceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 3

Heumüller et al.


sis plan. Our framework comprises templates and guidelines to support planning teams in developing exercise screenplay and analysis plan. It provides guidance for the three levels of conceptualization. A “scenario” is a subtopic of the exercise topic. The core of our approach is that scenarios logically follow the overall exercise goal and topic but also define specifications for injects for that scenario. The specification of a scenario results from requirements defined in the exercise course and consequently the exercise goal. A guideline with check questions and checklists supports scenario specification (see Figure 3, Translated excerpt from a guideline, individualized for THW and the exercise FEUERBALL). Scenario integration in exercise context and training background

Check questions:

Relation to exercise goal Relation to skills and tasks of a Führungsstelle Relation to exercise topic Mapping to frame scenario Reference documents / regulations

Why to exercise this scenario? Why is this scenario appropriate for exercise goal and topic?

Goals and core contents of scenario What is the goal of this scenario? Why to exercise the respective areas as planned? Which processes and tasks should be exercised in this scenario?

Goal, purpose Scenario content

Assessment criteria Evaluator

criteria

Notes

Referee/Observer

What are features of a successful task accomplishment? What are characteristics of a good staff performance in this scenario? How can these features be measured and distinguished?

Exercise staff

Tasks / expected behaviour of participants Sphere of responsibility

Activities / expected behavior

Who is responsible to work on these tasks? What has this staff element to do, to solve these tasks successfully?

Figure 3: Excerpt from scenario specification

The planning team integrates the scenarios into the exercise goal and topic and frame scenario. This needs to consider skills or functions that are to be exercised according to the exercise goal. Check questions have to be answered: “Why to exercise this scenario?” and “Why is this scenario appropriate for exercise goal and topic?”. Scenarios exercise subgoals through a collection of injects and each scenario has a scenario goal. Performance criteria have to be defined on which staff performance can be assessed in this scenario (as part of the process evaluation). These criteria are the basis for requirements for injects to trigger specific actions. The planning team has to develop an evaluation plan that details who evaluates which criteria and which instruments are used to evaluate which criteria. E.g. an observer can check if respective forms are used correctly or observe coordination processes and the exercise staff can assess status reports or demands of the staff. The performance of the staff is evaluated at two levels: for each inject the action of the staff can be evaluated e.g., by observers and at scenario level actions and internal processes can be evaluated. Our observer-checklists comprise criteria to evaluate general and exercise specific staff work processes. Scenario criteria contribute to assessment of work processes depending on topic and goal (e.g. processes of fuel supply in an assembly area). Criteria of general work processes follow our understanding of staff work, which we gained during our research. The emphasis lies here on concepts describing work processes, namely staff work, knowledge & experience, order-specific measure and work organization. E.g. we analyze staff work with criteria about work on overviews (e.g. “Force tables were always up-to-date.”) or on situational map (“Tactical signs on situational map were always correct.”). Assessment is implemented through checkboxes, rating scales and free text. We validated our understanding through discussions with experienced staff members. The framework for staff exercises has several products. They are depicted in Figure 2. There are templates, guidelines and check questions for all relevant parts of exercise conceptualization like inject development or creation of observer-checklists for staff performance assessment. The framework comprises also templates for exercise conducting and evaluation. E.g., there is a template supporting the exercise staff in providing injects to the exercise participants. The template refers to relevant points of an inject (e.g. sender, addressee, channel, content, purpose, etc.) and can so provide all necessary information to the “injector”. There is a template for injectreaction documentation to capture reactions, their effects or new ways of solutions.

FUTURE DEVELOPMENTS

We introduced our framework for staff exercises. Our objective is to support exercising of staffs, assessment of training levels and exercise success as well as staff performance analysis. The conceptual “heart” of our frameProceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 4

Heumüller et al.


work is a model of staff work. It relates goals to evaluation results and guides both evaluation and design of exercises. Due to space limitations it could not be published in the paper at hand. We continue to enhance our framework with its products and on the validation of our framework with more partners. The main contribution of our research is a framework to support command post exercises based on a systematic and scientific way. We consider our approach as well-grounded in practice and relevant. The usual limitations of action research namely generalizability of our findings apply.

REFERENCES

1.

Alexander, D. (2000): Scenario methodology for teaching principles of emergency management. International Journal of Disaster Prevention and Management, 9(2), 89-97.

2.

Bartley, B.H., Stella J.B., Walsh, L.D. (2006): What is a disaster? Assessing utility of simulated disaster exercise and educational process for improving hospital preparedness. Prehospital and Disaster Medicine 21(4): 249-255.

3.

Baskerville, R. L. (1999): Investigating Information Systems with Action Research, Communications of the Association for Information Systems, 2(19).

4.

BBK (2011): Leitfaden für strategische Krisenmanagement-Übungen. Bevölkerungsschutz und Katastrophenhilfe, Bonn.

5.

BMI (2008): Anlagen zum Konzept für Notfall- und Krisenübungen in Kritischen Infrastrukturen, Bundesministerium des Innern, Berlin.

6.

Callan, T. 2009. So, you want to run an exercise? Australian Journal of Emergency Management 24(2): 5962.

7.

Crichton M.T., Lauche, K., Flin, R. (2005): IC Skills in the Management of an Oil Industry Drilling Incident: a Case Study, in Journal of Contingencies and Crisis Management, 13(3).

8.

Friesen Sh. K., Hales, D., Chuka, N., Morrisey, Ch.C. (2010): Covering the Bases: Development of a Framework for Defence Force planning scenarios; 15 th ICCRTS.

9.

FwDV-100 (1999): Führung und Leitung im Einsatz, Feuerwehr Dienstvorschrift 100, NRW.

10. Glass, W. (2006): Übungshandbuch für Katastrophenschutzeinheiten, Walhalla-Fachverlag, Berlin. 11. Hewett, P.L. Jr, Mitrani, J.E., Metz, W.C., Vercellone, J.J. (2001): Coordinating, integrating, and synchronizing disaster response: use of an emergency response synchronization matrix in emergency planning exercises and operations. International Journal of Mass Emergencies and Disasters 19(3): 329-348. 12. Hosseini, M., Izadkhah, Y.O., (2010): Training emergency managers for earthquake response: challenges and opportunities. Disaster Prevention and Management, 19(2): 185-198. 13. Klein, K.R., Brandenburg, D.C., Atas, J.G., Maher, A. (2005): The use of trained observers as an evaluation tool for a multi-hospital bioterrorism exercise. Prehospital and Disaster Medicine 20(3): 159-163. 14. ODP (2004): Office of Domestic Preparedness: Homeland security exercise and evaluation program, Volume I: Overview and doctrine. Available at http://www.oregon.gov/OMD/OEM/docs/dom_prep/fed_support/hseep_volume_1.pdf?ga=t. on 06.02.2012. 15. Ohio (1999): The Ohio Hazardous Materials Planning and Exercise Guidance Booklet. Ohio Emergency Management Agency. 16. Oregon (1997): Oregon Version: Exercise Design and Evaluation Course. State of Oregon. 17. Payne, C.F. (1999): Contingency plan exercises. Disaster Prevention and Management, 8(2): 111-117. 18. Perry, R.W. (1995): The Structure and function of community emergency operations centres. Disaster Prevention and Management, 4(5): 37-41 19. Perry, R.W. (2003): Incident Management Systems in disaster management. Disaster Prevention and Management, 12(5): 405-412. 20. Perry, R.W. (2004): Disaster Exercise Outcomes for Professional Emergency Personnel and Citizen Volunteers, Journal of Contingencies and Crisis Management, 12(2). 21. Peterson, D.M. & Perry, R.W. (1999): The Impacts of Disaster Exercises upon Participants, Journal of Disaster Prevention and Management, 8(4). 22. Potter S., Farry, M., (2009): Supporting the C2 of Training Exercise Management: The design of a Decision-Centered Scenario Management Tool, 14th ICCRTS. Proceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 5

Heumüller et al.


23. Renner, S. (2001): Emergency exercise and training techniques. Australian Journal of Emergency Management, 16(2). 24. Richter, S., Heumüller, E., Lechner, U. (2010): Concepts for Command & Control Effectiveness in German Disaster Response, BledConference, Slovenia. 25. Teuber, M. (2007): Lernunterlage, Einheiten des THW – Grundsätze Einsatzoptionen, THW Bundesschule Hoya. 26. UNDHA (1993): Strategic Aspects of Geological and Seismic Disaster Management and Disaster Scenario Planning, United Nations Department of Humanitarian Affairs, New York. 27. USDT (2004): US Department of Transportation: Developing a hazardous materials exercise program: A handbook for state and local officials. Available at http://www.ntl.bts.gov/DOCS/254.html. on 06.02.2012 28. VDR (2011): Verband der Reservisten Bayern: Zivil-Militärische Zusammenarbeit in Bayern. http://www.vdrbw-bayern.de/LdsKdo_BY/LdsKdo_BY_003_ZMZ_uebersicht.htm on 28.11.2011

Proceedings of the 9th International ISCRAM Conference – Vancouver, Canada, April 2012 L. Rothkrantz, J. Ristvej and Z. Franco, eds. 6