A Cognitive Modeling Approach to Decision Support Tool Design for ...

This article was downloaded by: [North Carolina State University] On: 07 January 2013, At: 06:12 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Human-Computer Interaction Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hihc20

A Cognitive Modeling Approach to Decision Support Tool Design for Anesthesia Provider Crisis Management a

b

a

Noa Segall , David B. Kaber , Jeffrey M. Taekman & Melanie C. Wright a

c

Department of Anesthesiology, Duke University Medical Center, Durham, North Carolina

b

Edward P. Fitts Department of Industrial & Systems Engineering, North Carolina State University, Raleigh, North Carolina c

Patient Safety Research, Trinity Health and Saint Alphonsus Health System, Boise, Idaho Accepted author version posted online: 11 Apr 2012.Version of record first published: 03 Jan 2013.

To cite this article: Noa Segall , David B. Kaber , Jeffrey M. Taekman & Melanie C. Wright (2013): A Cognitive Modeling Approach to Decision Support Tool Design for Anesthesia Provider Crisis Management, International Journal of HumanComputer Interaction, 29:2, 55-66 To link to this article: http://dx.doi.org/10.1080/10447318.2012.681220

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Intl. Journal of Human–Computer Interaction, 29: 55–66, 2013 Copyright © Taylor & Francis Group, LLC ISSN: 1044-7318 print / 1532-7590 online DOI: 10.1080/10447318.2012.681220

A Cognitive Modeling Approach to Decision Support Tool Design for Anesthesia Provider Crisis Management Noa Segall1, David B. Kaber2, Jeffrey M. Taekman1, and Melanie C. Wright3 1

Department of Anesthesiology, Duke University Medical Center, Durham, North Carolina Edward P. Fitts Department of Industrial & Systems Engineering, North Carolina State University, Raleigh, North Carolina 3 Patient Safety Research, Trinity Health and Saint Alphonsus Health System, Boise, Idaho

Downloaded by [North Carolina State University] at 06:12 07 January 2013

2

changes in patient state, administering drugs, and so on, within a short period. This combination of task complexity coupled with the dynamics of the OR (multiple team members with multiple objectives under high task demands) can be conducive to errors in managing critical incidents. Studies have attributed up to 82% of anesthesia mishaps to human error (Weinger, 1999). Crisis management skills are important in several work domains. In aviation, for example, aircrews are trained in the use of checklists and problem-solving strategies for emergency conditions, such as on-board fires, engine failures, and so on (Burian, 2004). In anesthesiology, however, Gaba et al. (1994) observed that crisis management is not adequately taught and not easily learned during clinical practice. The use of cognitive aids, such as checklists that can guide anesthesia providers during crises, is also limited. Aids have been developed for preparing to recognize and manage crises, debriefing after a crisis, and training purposes (Gaba et al., 1994). However, there has been a historical emphasis on relying on memory to handle both routine and crisis situations in real time (Veterans Health Administration, 2003), which can lead to errors due to inaccurate recall of procedures (Weed, 1997). There is a need to develop electronic aids that can assist anesthesia providers in confirming diagnoses and in recalling critical treatment steps and the order of those steps. The patient vital signs monitor is one of the anesthesia provider’s main sources of information on changes in patient physiological state. However, currently available monitor technology does not provide sufficient support for prompt and accurate decision making during crisis management (e.g., Krol & Reich, 1999; Mylrea, Orr, & Westenskow, 1993; Weinger, 1999; Zhang et al., 2002). Several anesthesia decision support tools (DSTs) have been developed to address this problem, though few are in clinical operation (Liu, Wyatt, & Altman, 2006; Uckun, 1994). Most common among these are intelligent alarms aimed at reducing the frequency of false alarms in the OR. An intelligent alarm system monitors multiple patient physiological variables in real time and synthesizes this information to produce a status assessment and to warn of possible problems when deviations from normal conditions are detected

Prior research has revealed existing operating room (OR) patient monitors to provide limited support for prompt and accurate decision making by anesthesia providers during crises. Decision support tools (DSTs) developed for this purpose typically alert the anesthesia provider to existence of a problem but do not recommend a treatment plan. There is a need for a human-centered approach to the design and development of a crisis management DST. A hierarchical task analysis was conducted to identify anesthesia provider procedures in detecting, diagnosing, and treating a critical incident and a cognitive task analysis to elicit goals, decisions, and information requirements. This information was coded in a computational cognitive model using GOMS (Goals, Operators, Methods, Selection rules) Language. An OR monitor interface was prototyped to present output from the cognitive model following ecological interface design principles. A preliminary assessment of the DST was performed with anesthesiology and usability experts. The anesthesiologists indicated they would use the tool in the perioperative environment and would recommend its use by junior anesthesia providers. Future research will focus on formal validation of the DST design approach and comparison of tool output to actual anesthesia provider decisions in real or simulated crises.

1. INTRODUCTION 1.1. Crisis Management in the Operating Room To anticipate and remedy critical incidents in the operating room (OR), anesthesia providers must continuously monitor the patient, physiological variables and other displays, and the surgical procedure. Patient crises can develop in a matter of seconds, and swift action is necessary to prevent permanent injury or death. At least 20% of anesthesia cases involve some type of perioperative problem, and approximately 5% of cases develop into crises (Gaba, Fish, & Howard, 1994). The anesthesia provider must be able to quickly perform multiple complex tasks, such as identifying the source of problems, monitoring Address correspondence to Noa Segall, Department of Anesthesiology, Duke University Medical Center, Box 3094, Durham, NC 27710. E-mail: [email protected]

55


56

N. SEGALL ET AL.

(Mylrea et al., 1993). For instance, Becker, Käsmacher, Rau, Kalff, and Zimmermann (1994) employed a fuzzy inference approach to the design of an intelligent alarm for cardiac anesthesia. Rules for estimation of five state (derived) variables such as depth of anesthesia, based on heart rate and other physiological variables, were developed through consultation with domain experts. An interactive display showed deviations of the state variables from normal ranges. This system was installed in an OR and evaluated by anesthesia providers during surgical procedures. Its sensitivity, specificity, and predictability were found to be high (Becker et al., 1997). Some intelligent alarms go beyond diagnosis of abnormal events, suggesting therapeutic actions to correct them. For example, Schecke et al. (1988) developed the AES-2 system for decision support in aortocoronary bypass surgery (after termination of the extracorporeal circulation). AES-2 is an extension of an anesthesia information system that records patient variables and manual data inputs (e.g., drug administration). Anesthesia provider knowledge was elicited to create fuzzy rules that determine whether derived variables deviate from normal ranges. When such a deviation is detected, AES-2 alerts the anesthesia provider and recommends therapeutic action—which symptom to treat first, what drugs to administer (based on side effects), and in what dosage (based on patient data and results of previous dosages). Initial evaluations of this tool have been carried out, but no results have been reported. There is a need for additional training and decision support technologies for anesthesia providers in order to reduce critical incident rates. Unfortunately, the majority of systems developed for this purpose remain in the research stage. Furthermore, these technologies are primarily for alerting providers to potential problems and often do not extend to treatment recommendations.

1.2. Modeling Methods for DST Development With the aforementioned need in mind, we hypothesized that a model of expert anesthesia provider cognitive performance would serve as an effective basis for DST development with the capability to support treatment recommendations in crisis situations. Prior research on medical expert systems for physicians (see Shortliffe, 1986, for a seminal study) demonstrated that expert knowledge acquisition and representation combined with model-based reasoning were key to progress in the development of effective DSTs. In general, three approaches to modeling have been identified for the purpose of automating human tasks, including task, cognition, and knowledge modeling (Merkelbach & Schraagen, 1994). Task modeling, including methods such as hierarchical task analysis (HTA) and cognitive task analysis (CTA), focuses on goals to be accomplished through task performance, independent of the agent performing the task. DST development in the anesthesia domain typically has not applied structured knowledge elicitation techniques, including CTAs, to support real-time crisis management. Cognitive modeling emphasizes

representation of expert strategies in tasks that can be broken down into procedures, which are dynamically organized for performance considering human information-processing limitations such as the finite capacity of working memory. The Goals, Operators, Methods, Selection rules (GOMS) modeling approach, originally developed by Card, Moran, and Newell (1983) and formalized as a computational language by Kieras (1999, 2006), is a good example of this type of approach. In knowledge modeling, methods such as KADS (Knowledge Analysis and Documentation System; Wielinga, Schreiber, & Breuker, 1992) are used to represent the knowledge structures and strategies of an ideal user, who has access to all required information and is able to execute all strategies under any circumstance. Merkelbach and Schraagen (1994) viewed these three modeling approaches as complementary, advocating shared implementation in the analysis of cognitive tasks. On this basis, we formulated a human-centered approach to the design and prototyping of a DST for anesthesia crisis management by integrating task and cognitive modeling. As a starting point, we focused the modeling work on describing crisis management, specifically myocardial infarction (MI; a heart attack) accompanied by hypotension and tachycardia (rapid heart rate), both from a task perspective and the perspective of the anesthesia provider. We present the use of computational GOMS Language (GOMSL) for modeling management of perioperative MI. To our knowledge, no previous study has applied cognitive modeling as a basis for DST engine development in the anesthesiology domain. The resulting DST is also novel relative to existing intelligent alarms in that it outputs treatment steps for anesthesia providers in real time, based on changes in patient state. More important, it provides explanations as to how the suggestions were derived. Because healthcare is an open system, events that a DST does not anticipate are bound to occur (Vicente, 2003). When a tool suggests an uncertain course of action, operators tend to simply accept its imperfect advice, even when the necessary information to make a decision is available (Vicente, 2003). Therefore, it is useful for a DST to explain its underlying logic so the anesthesia provider can evaluate its suggestions before accepting (or rejecting) them and determine whether the system is attempting to address an event outside its normal operating range. Such a justification could also promote user acceptance of the tool (Huang & Endsley, 1997). The specific steps and contributions of each method to the resulting DST development are described in the following sections. In addition, we present a preliminary assessment of the approach to DST development in terms of anesthesiologists’ perceptions of the utility of the tool for the OR as well as the usability of the interface prototype.

2. METHODS Although experience in high-risk, safety critical domains such as aviation and nuclear power plant management has

DECISION SUPPORT TOOL FOR ANESTHESIA PROVIDERS


demonstrated the use of human factors methods to reduce errors (Lin et al., 1998), the anesthesia DSTs described in the literature have not been developed with a human-centered approach. The steps in our approach to developing the anesthesia DST included the following: 1. Performing a HTA of anesthesia provider procedures in managing MI to identify critical cues and resources. 2. Conducting a CTA to describe anesthesia provider knowledge requirements for detecting, diagnosing, and treating MI. 3. Coding the cognitive model in GOMSL on the basis of the task analyses. 4. Prototyping an interface for presenting output from the compiled cognitive model, based on the ecological interface design (EID) framework. 5. Simulating operation of the DST using a GOMSL compiler and model analysis tool.

2.1. Hierarchical Task Analysis HTA is a method for analyzing complex tasks in order to better understand the procedures, cues, and actions required to accomplish the task. HTA has been applied to a wide range of problems, from printer cartridge replacement to air traffic control tasks (Shepherd, 2001), as a basis for designing new interfaces (or modifying existing ones), and to compare the complexity of different system designs. The process of HTA starts with data acquisition through behavior observation, review of process documents (e.g., standard operating procedures), interviews, and simulations (Annett, 2003). The task is then described as a hierarchy of tasks and subtasks using goals, tasks, operations (unit behaviors), and plans. An HTA of the task of detecting, diagnosing, and treating MI was conducted following the approach laid out by Annett (2003). In the data acquisition phase, anesthesia medical documents were reviewed for an overview of the procedures. Due to the low incidence of perioperative MI, it was not practical to conduct OR observations to study management of the crisis. Instead, we carried out observations and interviews of anesthesiologists at the Duke University Human Simulation and Patient Safety Center. At this facility, patient simulators are programmed to train medical and nursing students in crisis management (cf. Morgan & Cleave-Hogg, 2005). Information about how to detect the onset of MI, the consequences of correct and incorrect diagnoses, and the different possible treatments and complications were gathered while watching video recordings of anesthesiology residents managing the simulated crisis. In addition, three experienced anesthesiologists participated in five semistructured interviews (1–2.5 hr each), which revolved around MI diagnosis and treatment as personally witnessed and treated by the anesthesiologists or as taught to anesthesiology residents. Example questions asked during the interviews (adapted from Nielsen, 1993) included the following:

57

• Describe an exception from the routine treatment of MI. • Describe a notable success or failure in treating MI. • Describe problems that may be encountered with the MI treatment procedure. Experts were also presented with MI treatment algorithms (Gaba et al., 1994; Ludbrook, Webb, Currie, & Watterson, 2005) and asked to adapt them to their own treatment plans and to provide criteria for quantifying patient states. This step as part of the HTA was utilized to identify plans or strategies the expert anesthesia provider may use, as well as the task environment and system states that trigger the use of specific strategies. 2.2. Cognitive Task Analysis A CTA focuses on the knowledge, thought processes, and goal structures of operators performing cognitive tasks (Hollnagel, 2003). Its objective is to identify and describe mental strategies, critical decisions, and situation awareness requirements for performing a particular cognitive task. CTA has been used to design new system interfaces, evaluate existing interfaces, develop expert systems, and provide a basis for operator training (Wei & Salvendy, 2004). Goal-directed task analysis (GDTA), a form of CTA suited for critical decision and information requirements assessment (Endsley, 1993), was used as part of this work. Task goals, key decisions, and information needs are elicited from domain experts in interviews. Analysts then create a goal tree describing this information, independent of technology that may be used to perform tasks (Usher & Kaber, 2000). Unlike HTA, this analysis is based on operator goals in a scenario and not on specific states of the task environment. The analysis does not require that goals be addressed in a specific order or specified in terms of the tools with which operator may be working. In the present study, three expert anesthesiologists were interviewed (1–2 hr each) to create a MI treatment GDTA. A partial goal tree was prepared based on the process documents, observations, and HTA, and the anesthesiologists were asked to modify and add to it as necessary. Following each interview, it was updated to reflect the interviewee’s suggestions and the revised goal tree was presented to the next interviewee in an iterative refinement process. 2.3. Cognitive (GOMSL) Modeling Once information about the expert decision-making processes and environmental cues was obtained through the task analyses, a GOMSL (Kieras, 1999) cognitive model of MI management was developed. The goal of such models is to describe and predict how users will interact with existing and proposed system designs (John & Kieras, 1996b). These models can be used to predict the amount of learning required to perform a task (Kieras, 1999), task complexity (Dix, Finlay, Abowd, & Beale, 2004), and imprecision in human judgment (Karwowski, Kosiba, Benabdallah, & Salvendy, 1990). This capability has


58

N. SEGALL ET AL.

been found to expedite and reduce costs of user testing during the initial phases of interface design (John & Kieras, 1996b; Kieras, 1999). GOMS has been successfully used to model human–computer interaction in many real-world applications, from document preparation with a word processor to a command and control database system for space operations (Card et al., 1983; John & Kieras, 1996b; Kaber, Green, Kim, & Segall, 2011). With respect to GOMS model data structures, a Goal is the state of affairs to be achieved. Its dynamic role is to provide a memory point to which the system can return on failure and from which information can be obtained (e.g., about what has already been tried). Operators are perceptual, motor, or cognitive acts of a user. A Method is a list of steps necessary to accomplish a goal; it is a conditional sequence of goals and operators. Selection rules route control to appropriate methods using if-then statements. GOMSL is a computational variant of GOMS for cognitive behavior modeling based on a simple serial stage human information-processing architecture (John & Kieras, 1996a). The architecture includes auditory, visual, vocal, manual, and cognitive processors, each with its own working memory store, as well as a shared long-term memory store (Kieras, 2006). GOMSL has a structured notation in which methods contain both external (keystroke-level) operators and internal operators that can, for example, add content to working memory. John and Kieras (1996a) empirically validated GOMSL models for predicting keystroke-level behaviors. GOMSL model code can be compiled on a computer to simulate user interaction with a system interface using the GOMS Language Evaluation and Analysis Tool (GLEAN). GLEAN assigns execution times to each data structure in a model based on estimates from engineering psychology research (Card et al., 1983; Kieras, 1999; Kieras & Meyer, 1998). On this basis, the compiler generates several outputs for a GOMSL model, including total performance time, level of learning difficulty based on the number of methods in a model, and task complexity estimated based on the length of model methods. Automation of the behavioral analysis process using the GLEAN compiler standardizes model development through a defined syntax, yields a common set of outcome measures, and facilitates objective and accurate evaluation of interface designs. In the present study, a GOMSL model was coded following the approach laid out by Kieras (1999). The model was based on the task analyses and included goals corresponding to goals and subgoals in the GDTA, methods corresponding to tasks in the HTA, operators corresponding to operations in the HTA, and selection rules and decisions corresponding to plans in the HTA and decisions and information requirements in the GDTA. These types of associations of components of task analyses with elements of GOMSL models have not previously been made and provide a highly structured approach to development of computational cognitive models. Here, it is also important to note several limitations of GOMS that have been identified in the literature. GOMS is intended to

represent expert, error-free performance. Thus, it is not applicable for modeling novice behavior, which may rely more heavily on problem solving than on plan retrieval and execution. The GOMS method cannot account for errors in performance, which even skilled users may commit. To analyze the nature of errors in user interaction with a system design, erroneous action sequences must be coded in a model. GOMS was originally developed for describing serial behavior, whereas many tasks involve cognitive and psychomotor processes that occur in parallel (Olson & Olson, 1990). However, in the most recent version of GOMSL and GLEAN, Kieras (2006) integrated multiple thread execution capability at the method level; that is, the operators or steps in multiple methods can be executed simultaneously.

2.4. Ecological Interface Design EID is a theoretical framework for the design of interfaces for human–machine systems intended to support skilled users in coping with unanticipated events. The framework focuses on presenting constraints of a work domain to users so they can make decisions within these boundaries that are appropriate for the unfolding situation. In this way, EID encourages skill- and rule-based behavior, which involve fast, effortless processing that is less error prone while also supporting knowledge-based behavior that is crucial for novice users and for managing unexpected problems (Vicente & Rasmussen, 1992). Following the approach to EID presented by Burns and Hajdukiewicz (2004) and using an anesthesiology work domain model (i.e., a model of the human body) developed by Hajdukiewicz, Vicente, Doyle, Milgram, and Burns (2001), an ecological interface was prototyped to display patient variables and constraints as part of the DST. Variables identified in the work domain model were used as a basis for defining constraints and to guide the display design process. For example, single variable constraints usually place upper and lower bounds on a variable. Information on such constraints can be used to display ranges of scales, determine alarm limits, and so on. Multivariate constraints are relationships between two or more variables that can be expressed as equations. Displaying these constraints in a way that is understandable to users can enhance performance and reduce mental workload. Finally, means–end relationships describe the implication of one variable in the value of another. These relationships can be presented through display organization, for example, by grouping related graphics so a user can see the effect of the setting of one variable on the outcome of another.

2.5. GOMSL Model and Interface Integration Wood (2000) developed the error-extended GOMS language evaluation and analysis tool (EGLEAN). This tool extends Kieras’s GLEAN by providing the capability to interconnect a GOMSL model with an actual software (Java) application


Eclipse Scenario.txt

EGLEAN

Human.gomsl

scenario events commands

Java

Interface.java


Output: - Time to task completion - Working memory

FIG. 1.

EGLEAN system architecture (color figure available online).

interface (Soar Technology, 2005). We used EGLEAN to compile and execute the GOMSL model representing anesthesia provider behavior and to connect the running model with the DST interface. To simulate human behavior at the application interface based on the model code, EGLEAN makes use of three files (also see Figure 1 for an architecture of the system). A .java file contains the interface, including visual objects the modeled user can see and interact with. Any underlying functionality (e.g., interface behavior when a button is pressed) is also coded in this file. A .gomsl file contains the GOMSL model of human behavior in interacting with the interface. The model receives as input states of visual objects in the interface and outputs interactions with these objects (e.g., pressing a button). A .txt file contains scenario information that populates the Java interface in real time. In the DST, the scenario file included time-dependent patient physiological data, which served as input to the .java file. Using EGLEAN, the running GOMSL model could “see” patient variables displayed in the Java version of the interface and “react” to changes in their values. These reactions were output (through the same interface) as DST advice that could be used by real anesthesia providers. In this way, the cognitive model was used as an engine for the DST. 2.6. Preliminary Assessment of the DST The DST was evaluated for applicability to the OR by expert anesthesiologists. Usability of the interface design was assessed by human factors professionals as well as the clinicians. For the purpose of these evaluations, two scenario files were developed to drive the cognitive model simulation. The ultimate purpose of the DST is to run in real time during surgical procedures in the OR, receiving real-time physiological data as input. However, the prototype DST was not directly connected to actual OR sensors. Thus, virtual scenarios were scripted, each

59

comprising rows of values for simulated patient physiological variables, including heart rate, end-tidal CO2 , and so forth. The DST read a new row of variable values every 5 s. In the first scenario, a simulated patient suffered MI, accompanied at later stages by hypotension and tachycardia. In the second scenario, the patient hemorrhaged extensively and became hypotensive. The treatment protocol for this scenario was adapted from the hypotension management steps in the MI treatment algorithm and was meant to demonstrate the capabilities of the DST under conditions less extreme than cardiac ischemia. The cognitive model diagnosed these problems as they developed and output recommended treatment steps for the anesthesia provider to consider. The goal of the applicability assessment was to determine the usefulness of the DST to anesthesia providers in managing perioperative crises like MI. Three anesthesiologists were recruited to evaluate the DST. They were provided a brief user manual describing DST features and patient information sheets, which described the simulated patient’s baseline physiological state and the surgical procedure for both scenarios. The anesthesiologists viewed the DST output on a laptop computer for the first scenario (scenario presentation order was randomized) and completed a survey requesting ratings of agreement with seven statements (e.g., “the explanations provided by the tool supporting its suggestions were useful”) and soliciting comments regarding these statements. They then watched the second scenario and completed a similar survey. They also rated five general statements (e.g., “essential features could be added to this interface”) after watching both scenarios. For a usability inspection, it is recommended that at least three to five evaluators examine an interface for usability problems (Nielsen, 1993). The evaluators do not need to be usability experts (Virzi, 1997); in fact, one study has shown that the evaluation results of work domain experts and usability experts complement each other: Although domain experts found fewer problems, the problems they identified were more severe (Følstad, 2007). For the usability inspection of the DST, two human factors experts (who also watched the DST step through the two scenarios) and the same three anesthesiologists (domain experts) evaluated the DST interface for compliance with six usability heuristics (from Nielsen, 1993), including preventing errors and providing prompt feedback. The evaluators prepared a list of heuristics that were violated and detailed descriptions of each problem they identified. This was a preliminary assessment of the technology, serving as a basis for design enhancements prior to any formal validation of the DST in a simulated or real OR environment. 3. RESULTS AND DISCUSSION 3.1. Hierarchical Task Analysis Figure 2 presents the high-level HTA diagram for the MI treatment task. The overall goal is to treat MI. Tasks for achieving this goal include verifying the manifestations of


60

N. SEGALL ET AL.

FIG. 2.

High-level hierarchical task analysis diagram for myocardial infarction treatment task (color figure available online).

myocardial ischemia, considering precipitating factors, and so on. Operations include “Evaluate hemodynamic status” and “Obtain a 12-lead ECG [electrocardiography].” Finally, plans (such as those shown to the right and below the diagram) are used to specify task strategies when certain conditions apply. The complete HTA included 11 high-level tasks, 28 secondlevel tasks, 45 third-level tasks, and 48 fourth-level tasks. Of these, 103 were operations, which could not be further broken down. The HTA also included 18 plans. Although this analysis provided a clear sense of the sequence and tasks as part of anesthesiologist management of MI, the plans did not provide complete information on critical decisions on patient states or the information requirements an anesthesia provider might have for addressing low-level tasks. The GDTA was used to elicit this information from anesthesiologists.

3.2. Goal-Directed Task Analysis The GDTA provided a comprehensive description of the critical decisions and information requirements of the anesthesia provider in treating MI. It included 11 goals, two subgoals, and 21 tasks. The high-level goals of the GDTA were similar to

those of the HTA. There were 83 decisions and 206 information requirements associated with these goals. As an example, Figure 3 shows the tasks, decisions, and information requirements associated with one of the subgoals—that of assessing patient signs and symptoms. Although the GDTA identified many information requirements for the anesthesia provider in MI management, the results of the HTA, including identification of existing information technology used in the OR, was necessary to provide information on data sources the anesthesia provider might use to address such information needs. The HTA results also complemented the GDTA findings by providing the analyst with a sense of timing of information requirements for task performance.

3.3. GOMSL Cognitive Model The HTA and GDTA descriptions of the MI treatment task were used as a basis for developing the computational GOMSL model. The outcome of this step was a high-level GOMSL model describing the MI treatment task in terms of user goals, methods, decisions, and actions. The code consisted of 13 methods for the treatment of MI, one selection rule (to route control

61



Subgoal

1.1 Assess clinical signs and symptoms

Tasks

Talk to patient if patient is under regional anesthesia or MAC

Decisions

Is patient awake ? Is patient feeling chest pain ? Where is chest pain located (does it radiate) ? What type of chest pain is patient feeling ? Is patient experiencing palpitations ? Is patient experiencing shortness of breath ?

Information Requirements

Patient’s eyes – openor closed Sedation level Ramsay score BIS monitor Patient’s response

Evaluate ECG

Assess clinical signs

Is there a ST segment depression/elevation ? Is there a flattening or inversion of T waves? Are there ventricular arrhythmias (PVCs, vtach, v-fib, etc.) ?

Are there unexpected changes in signs ? What are potential causes of changes in signs ?

ECG output Current ECG leads Baseline ECG Administered drugs that can affect heart (sux,etc.) Patient data (age,cardiac history, pulmonary problems, etc.) Inserted catheters, drips, etc. Surgical procedure (electrocautery, cardiac procedures, etc.)

Blood gases Range of normal blood gases Blood gasest rends Ventilation pressure Range of normal ventilation pressures Potassium level Range of normal potassium levels Potassium level trends

Evaluate hemodynamic status

Are there unexpected hemodynamic changes ? What are potential causes of hemodynamic changes ?

HR Range of normal HRs HR trends BP Range of normal BPs BP trends ETCO2 Range of normal ETCO2 ETCO2 trends O2% Range of normal O2% O2% trends Inspired O2 Range of normal inspired O2 Inspired O2 trends Administered drugs

FIG. 3. Goal-directed task analysis for subgoal of assessing clinical signs and symptoms (color figure available online).

to the appropriate treatment algorithm based on the current patient state), and 136 steps. Figure 4 shows an excerpt from the GOMSL code modeling anesthesia provider decision making as to whether to begin advanced cardiac life support (ACLS). For example, if systolic blood pressure falls below 40 mm Hg, mean arterial blood pressure (MAP) falls below 30 mm Hg, or severe cardiac arrhythmias are present, the anesthesia provider should carry out the ACLS algorithm. The GOMSL code simulates this thought process. The model (of the anesthesia provider) searches for variables such as systolic blood pressure in the DST interface and, based on these values, decides whether to perform ACLS. When the cognitive model decides on a certain action, it is displayed as a recommended treatment step in the DST interface. The treatment recommendation is presented along with an explanation through the use of a “Type_in” operator in the GOMSL model, which represents text entry to an interface.

Based on communication from the GOMSL model, the Java application outputs the full treatment text to the DST interface, for example, “Treat as cardiac arrest—go through ACLS— severe arrhythmias present.” As previously mentioned, limitations of GOMSL include the assumption of error-free user behaviors processed in a serial manner. Such assumptions are not always accurate for describing anesthesia provider behavior, which may be iterative in nature. For example, the anesthesia provider typically hypothesizes a reason for an observed problem, decides on a potential solution, tests it, and observes whether the appropriate result was achieved—a “trial and error” approach. This behavior was illustrated in a simulated study of anesthesiologist crisis management (Hajdukiewicz et al., 2001). Anesthesiologist actions and verbalizations were mapped onto a model of the work domain—the human body—broken down by levels of abstraction and aggregation. The anesthesiologist’s problemsolving route was observed to be cyclical, moving between


62

N. SEGALL ET AL.

FIG. 4. GOMS Language code for subgoal of preparing for advanced cardiac life support (ACLS).

higher and lower levels of abstraction and aggregation, corresponding to verification of information and monitoring the effects of interventions. As the crisis developed, the problemsolving trajectory expanded to include more levels of abstraction; as the patient condition became clear, it contracted again. In the GOMSL model, such behavior was simulated by coding exceptions into the methods, causing the human processor to periodically check patient status displays to verify the current diagnosis. A new diagnosis led to the generation of a new treatment plan in a manner similar to how an actual anesthesia provider would behave.

3.4. DST Interface Design The DST interface was prototyped in Java with four display windows (see Figure 5). The patient variables window (top left) presents physiological data relevant to diagnosing and treating MI. This part of the interface was designed according to EID principles and Hajdukiewicz et al.’s (2001) work domain model. Single variable constraints are represented by normal ranges for a healthy patient [displayed in brackets] for each variable. Multivariate constraints are illustrated by use of color. When the tool diagnoses a problem, the values of the physiological variables considered in the diagnosis are highlighted in the same color as the diagnosis text. In addition, if treatment steps are related to certain variables, the relevant treatment step and variables are highlighted in the same color. If multiple variables are relevant to a certain treatment step, lines are used to connect these variables. Means–end relationships among variables

are reflected in the grouping of data fields by traits of the heart (e.g., ECG), circulation (e.g., MAP), and respiration (end-tidal CO2 ). A diagnosis window (top right) presents the most probable diagnosis as well as how the diagnosis was derived. Red is used to indicate critical patient states, orange indicates a severe problem, and green indicates the patient state is normal. The treatment steps window (bottom right) displays recommended treatment steps for the current state of the problem as diagnosed by the tool, explanations for these recommendations, and factors for the anesthesia provider to consider when making treatment decisions (e.g., the patient’s preexisting conditions). The treatment algorithm is updated continuously based on changes in patient variables. Finally, the ABCD window (bottom left) lists airway, breathing, circulation, and drug (ABCD) treatment steps, tailored to the patient’s current condition, when they are part of the treatment algorithm. The text is color coded, similar to the diagnosis window.

3.5. Simulating the GOMSL Model The anesthesia providers’ “trial and error” approach to patient diagnosis was modeled in GOMSL using error-handling mechanisms to check the current diagnosis repeatedly and update the treatment procedure accordingly. The model begins by checking data (physiological variables) in the Java application interface. A selection rule is used to route control to an appropriate method based on a diagnosis (normal, hypotension, or MI), and the treatment algorithm is output to the DST. Every few steps in the GOMSL code, an error exception is raised and



63

FIG. 5. Decision support tool interface (color figure available online).

the diagnosis is checked again. If it has not changed, control is returned to the point in the code at which the exception was raised. If the diagnosis has changed, the model restarts. The treatment steps window is cleared and a new treatment algorithm is output to the DST. Although the majority of anesthesia provider decision making is modeled in the GOMSL code, some types of decision making cannot be handled by the model due to the current operator set defined by Kieras (2006), including complex computational operations. In these cases, control is passed to the Java application, which is capable of more complex mathematical operations. For example, some anesthesia provider decisions are made based on values of baseline variables, including heart rate and blood pressure, which are measured before surgery. When starting the DST, the user is prompted for these values. When the HTA and GDTA call for comparing current values to the baseline, these calculations are carried out in Java. Another decision managed in Java involves trend evaluation. Anesthesia providers often assess whether the value of a physiological variable has changed over a 10- to 15-s period, as when assessing MAP stability. In such an event, the Java program compares previous MAP data points to the

current point, to determine whether significant deviations have occurred. Diagnosis of patient state is also handled by both the GOMSL model and Java application. The diagnosis is updated every 5 s, following retrieval of a new set of data points (physiological variables) from the scenario file, based on the computations of the Java application and the high-level decision rules coded in the GOMSL model. 3.6. Preliminary Assessment Three anesthesiologists volunteered to complete both the applicability assessment and the heuristic evaluation of the DST. The anesthesiologists were faculty members in the Duke University Department of Anesthesiology practicing medicine at Duke University Hospital. They had an average of 13 years of clinical practice experience, and all had treated a patient for perioperative MI in the past. In addition, two usability experts— who had both attained doctorates and worked for over 10 years in the field of human factors—completed heuristic evaluation of the DST interface. Figure 6 summarizes the applicability assessment survey. Statements that pertained to a specific scenario—MI or hypotension—were rated twice, and five general statements


64

N. SEGALL ET AL.

about the DST were rated once, after anesthesiologists had viewed both scenarios. In general, anesthesiologists rated the DST favorably, giving it an average score of 6.4 on a scale from 1 (strongly disagree) to 9 (strongly agree) (∼71%). They found the tool to be useful (∼77%) and indicated that they would use the tool in the OR (∼85%). They also suggested additional features that could improve the DST’s effectiveness, including a complete differential diagnosis (a list of all possible diagnoses ranked by likelihood), with supporting and refuting evidence; links to educational materials; and a method for checking off completed treatment steps. Fixation errors often occur when anesthesia providers do not treat the critical problem at hand because of attention or actions directed at other efforts (Weinger, 1999). One anesthesiologist commented that the DST could help anesthesia providers in this respect, by allowing them to think flexibly about alternative diagnoses or treatment options. As part of the heuristic evaluation, the anesthesiologists and usability experts made a total of 22 unique remarks about the DST interface. Common usability comments pertained to the rapid refresh rate of the Treatment steps window, which made reading the text difficult. Participants also noted limited readability of the treatment steps due to small fonts and a large amount of text, requiring scrolling. Many positive comments were made, though they were not solicited, suggesting evaluator approval of the DST.

FIG. 6.

4. CONCLUSIONS The goal of this research was to develop a DST for anesthesia providers using a cognitive model as an engine and following a human factors design approach, including task analyses, cognitive modeling, and an interface design framework. The use of the cognitive model and the methodology were novel to the target domain. The approach resulted in a DST prototype with the potential to support anesthesia provider decision making in crisis management. The prototype was then assessed by both domain and usability experts. The task analyses involved formal knowledge elicitation and served as a basis for the cognitive model development. Although the analyses were labor intensive, the elements of the structured forms of the HTA and GDTA were directly translatable to the data structures of GOMSL. Each method provided complementary information for this purpose. The HTA revealed variations in anesthesia provider task requirements and operations over time. It also linked provider behaviors to events in the work environment. The GDTA identified cognitive aspects of performance, including operator goals active in working memory and critical decisions and information requirements necessary to achieve those goals (Endsley, 1993). Coupled together, the techniques allowed for determination of which pieces of information are most important to anesthesia providers at different times during a task.

Applicability assessment results. (Survey items are on the vertical axis and the rating scale is on the horizontal axis.)



Because GOMS was originally developed for the purpose of modeling human–computer interaction tasks, adaptation to more dynamic tasks, like emergent patient care, required creative use of the modeling language (as previously described). The GOMSL model was effective for representing anesthesia provider psychomotor behavior; however, the current operator set limits functionality in terms of complex data calculations. When such functionality was required of the DST, calculations were modeled in an interconnected Java application, a sequential computer programming language. In medicine, different patients may exhibit similar trends in vital signs but may experience different problems or react differently to the same treatment. Consequently, iterative problem-solving cycles, including hypothesizing the source of the problem and treating the theorized diagnosis, may be necessary to stabilize a patient. This type of stochastic, nonlinear behavior is difficult to model in GOMSL, which is better suited for structured, sequential actions (Kieras, 1999). In addition, the DST describes ideal behaviors and may not support identification of some problems that may be encountered in managing perioperative MI. Thus, there are some characteristics of the modeling approach that are supportive of representing anesthesia provider behavior in crisis management and others that may deviate from actual cognitive performance. Despite the aforementioned methodological limitations, the application of human factors methods proved effective for the development of anesthesia decision support, which provided explanations of its underlying logic. The applicability and usability analyses indicated clinician and human factors professional support for future use of the tool in ORs and use of the EID approach for presenting patient state information, diagnoses, recommended treatment steps, and explanations as part of the prototype DST interface. Several future research directions are worth pursuing with regard to the DST design methodology described here. First, developing automated tools to support the cognitive modeling process may reduce the DST development time. Creating a cognitive model for a single critical incident (i.e., MI) required substantial effort. To implement the DST in the OR would require expanding the knowledge engine to address many other critical incidents. Code generation could be streamlined by automating aspects of this process. For example, it is possible to use GOMSL in conjunction with Lisp, an expert systems language, for DST development. GOMSL models can be translated to the Adaptive Character of Thought–Rational (ACT–R) computational cognitive language (Anderson et al., 2004), implemented in Lisp, using a compiler tool, G2A, developed by St. Amant and Ritter (2004). ACT–R models can also be created using ACT–Simple, a higher level cognitive modeling language that is easier to learn but still makes use of ACT–R’s powerful features (Salvucci & Lee, 2003). After further refining the treatment algorithm and interface, a formal validation of the DST would involve comparing the tool output to actual anesthesia provider behavior in simulated or real perioperative MI cases. Many hospitals have record-keeping systems that gather information including

65

patient variables, administered drugs, lab results, anesthesia provider notes, and so on, during surgical procedures. In the present application, patient variables for a procedure in which the patient suffered MI could be fed to the DST via a scenario file, and its treatment steps could be compared to the actual anesthesia provider’s actions, as documented in OR records. If the DST recommends treatment procedures similar to those carried out by the anesthesia provider, this would constitute evidence of the accuracy of the MI treatment algorithm and underlying cognitive model, thus validating the DST development methodology. ACKNOWLEDGMENTS During the completion of this research, Noa Segall worked as a research assistant in the Edward P. Fitts Department of Industrial & Systems Engineering at North Carolina State University. This research was supported in part by an Information Technology Research grant from the National Science Foundation (NSF) (No. IIS-0426852). Melanie Wright’s participation in this project was supported in part by a grant from National Institutes of Health (NIH), Agency for Healthcare Research and Quality (K02 HS015704-01). The opinions expressed are those of the authors and do not necessarily reflect the views of the NSF or NIH. We are grateful to the physicians who volunteered to participate in the interviews and to evaluate the decision support tool. We also thank Meghan Rogers for her conscientious work in formatting the article for submission for publication. REFERENCES Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111, 1036–1060. Annett, J. (2003). Hierarchical task analysis. In E. Hollnagel (Ed.), Handbook of cognitive task design (pp. 17–35). Mahwah, NJ: Erlbaum. Becker, K., Käsmacher, H., Rau, G., Kalff, G., & Zimmermann, H.-J. (1994). A fuzzy logic approach to intelligent alarms in cardioanesthesia. Proceedings of the Third IEEE International Conference on Fuzzy Systems, 3 (pp. 2072– 2076). Piscataway, NJ: IEEE. Becker, K., Thull, B., Käsmacher-Leidinger, H., Stemmer, J., Rau, G., Kalff, G., & Zimmermann, H.-J. (1997). Design and validation of an intelligent patient monitoring and alarm system based on a fuzzy logic process model. Artificial Intelligence in Medicine, 11, 33–53. Burian, B. K. (2004). Emergency and abnormal checklist design factors influencing flight crew response: A case study, In Helen Wilson (Ed.) 2004, Toulouse, France: EURISCO International. Burns, C. M., & Hajdukiewicz, J. R. (2004). Ecological interface design. Boca Raton, FL: CRC Press. Card, S., Moran, T., & Newell, A. (1983). The psychology of human–computer interaction. Hillsdale, NJ: Erlbaum. Dix, A., Finlay, J., Abowd, G., & Beale, R. (2004). Human–computer interaction (3rd ed.). New York, NY: Prentice Hall. Endsley, M. R. (1993). A survey of situation awareness in air-to-air combat fighters. The International Journal of Aviation Psychology, 3, 157–168. Følstad, A. (2007). Work-domain experts as evaluators: Usability inspection of domain-specific work-support systems. International Journal of Human– Computer Interaction, 22, 217–245. Gaba, D. M., Fish, K. J., & Howard, S. K. (1994). Crisis Management in Anesthesiology. New York, NY: Churchill Livingstone. Hajdukiewicz, J. R., Vicente, K. J., Doyle, D. J., Milgram, P., & Burns, C. M. (2001). Modeling a medical environment: An ontology for integrated


66

N. SEGALL ET AL.

medical informatics design. International Journal of Medical Informatics, 62, 79–99. Hollnagel, E. (2003). Prolegomenon to cognitive task design. In E. Hollnagel (Ed.), Handbook of cognitive task design (pp. 3–16). Mahwah, NJ: Erlbaum. Huang, S. H., & Endsley, M. R. (1997). Providing understanding of the behavior of feedforward neural networks. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics, 27, 465–474. John, B. E., & Kieras, D. E. (1996a). The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer–Human Interaction, 3, 320–351. John, B. E., & Kieras, D. E. (1996b). Using GOMS for user interface design and evaluation: Which technique? ACM Transactions on Computer–Human Interaction, 3, 287–319. Kaber, D. B., Green, R. S., Kim, S., & Segall, N. (2011). Assessing usability of human–machine interfaces for life science automation using computational cognitive models. International Journal of Human–Computer Interaction, 27, 481–504. Karwowski, W., Kosiba, E., Benabdallah, S., & Salvendy, G. (1990). A framework for development of fuzzy GOMS model for human–computer interaction. International Journal of Human–Computer Interaction, 2, 287–305. Kieras, D. (1999). A guide to GOMS model usability evaluation using GOMSL and GLEAN3 (Tech. rep.). Ann Arbor: University of Michigan. Kieras, D. (2006). A guide to GOMS model usability evaluation using GOMSL and GLEAN4 (Tech. rep.). Ann Arbor: University of Michigan. Kieras, D. E., & Meyer, D. E. (1998). The role of cognitive task analysis in the application of predictive models of human performance (EPIC Report No. 11, TR-98/ONR-EPIC-11). Washington, DC: Office of Naval Research. Krol, M., & Reich, D. L. (1999). The algorithm for detecting critical conditions during anesthesia. Proceedings of the 12th IEEE Symposium on ComputerBased Medical Systems (pp. 208–213). Piscataway, NJ: IEEE. Lin, L., Isla, R., Doniz, K., Harkness, H., Vicente, K. J., & Doyle, D. J. (1998). Applying human factors to the design of medical equipment: Patientcontrolled analgesia. Journal of Clinical Monitoring and Computing, 14, 253–263. Liu, J., Wyatt, J. C., & Altman, D. G. (2006). Decision tools in health care: Focus on the problem, not the solution. BMC Medical Informatics and Decision Making, 6. Ludbrook, G. L., Webb, R. K., Currie, M., & Watterson, L. M. (2005). Crisis management during anaesthesia: Myocardial ischaemia and infarction. Quality and Safety in Healthcare, 14, e13. Merkelbach, E. J. H. M., & Schraagen, J. M. C. (1994). A framework for the analysis of cognitive tasks (Report No. TNO-TM 1994 B-13). Soesterberg, The Netherlands: TNO Human Factors Research Institute. Morgan, P. J., & Cleave-Hogg, D. (2005). Simulation technology in training students, residents and faculty. Current Opinion in Anaesthesiology, 18, 199–203. Mylrea, K. C., Orr, J. A., & Westenskow, D. R. (1993). Integration of monitoring for intelligent alarms in anesthesia: Neural networks—can they help? Journal of Clinical Monitoring, 9, 31–37. Nielsen, J. (1993). Usability engineering. San Diego, CA: Academic Press. Olson, J. R., & Olson, G. M. (1990). The growth of cognitive modeling in human–computer interaction since GOMS. Human–Computer Interaction, 5, 221–265. Salvucci, D. D., & Lee, F. J. (2003, April). Simple cognitive modeling in a complex cognitive architecture. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 265–272. Schecke, T., Rau, G., Klocke, H., Kaesmacher, H., Hatzky, U., Kalff, G., & Zimmermann, H.-J. (1988). Knowledge-based decision support in anesthesia: A case study. Proceedings of the 1988 IEEE International Conference on Systems, Man, and Cybernetics, 2, 962–965. Shepherd, A. (2001). Hierarchical task analysis. London, UK: Taylor & Francis. Shortliffe, E. H. (1986). Medical expert systems—Knowledge tools for physicians. Western Journal of Medicine, 145, 830–839. Soar Technology. (2005). EGLEAN science and technology report: The first six months. Ann Arbor, MI: Author.

St. Amant, R., & Ritter, F. E. (2004). Specifying ACT–R models of user interaction with a GOMS language. Cognitive Systems Research, 6, 71–88. Uckun, S. (1994). Intelligent systems in patient monitoring and therapy management: A survey of research projects. International Journal of Clinical Monitoring and Computing, 11, 241–253. Usher, J. M., & Kaber, D. B. (2000). Establishing information requirements for supervisory controllers in a flexible manufacturing system using GTA. Human Factors and Ergonomics in Manufacturing, 10, 431–452. Veterans Health Administration. (2003). Cognitive aid for anesthesiology. Ann Arbor, MI: VA National Center for Patient Safety. Vicente, K. J. (2003). Less is (sometimes) more in cognitive engineering: The role of automation technology in improving patient safety. Quality and Safety in Health Care, 12, 291–294. Vicente, K. J., & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man, and Cybernetics, 22, 589–606. Virzi, R. A. (1997). Usability inspection methods. In M. G. Helander, T. K. Landauer, & P. V. Prabhu (Eds.), Handbook of human–computer interaction (pp. 705–715). Amsterdam, the Netherlands: Elsevier. Weed, L. L. (1997). New connections between medical knowledge and patient care. British Medical Journal, 315, 231–235. Wei, J., & Salvendy, G. (2004). The cognitive task analysis methods for job and task design: Review and reappraisal. Behaviour & Information Technology, 23, 273–299. Weinger, M. B. (1999). Anesthesia equipment and human error. Journal of Clinical Monitoring and Computing, 15, 319–323. Wielinga, B. J., Schreiber, A. T., & Breuker, J. A. (1992). KADS: A modelling approach to knowledge engineering. Knowledge Acquisition, 4, 5–53. Wood, S. D. (2000). Extending GOMS to human error and applying it to errortolerant design (Unpublished doctoral dissertation). Ann Arbor: University of Michigan, Department of Electrical Engineering and Computer Science. Zhang, Y., Drews, F. A., Westenskow, D. R., Foresti, S., Agutter, J., Bermudez, J. C., . . . Loeb, R. (2002). Effects of integrated graphical displays on situation awareness in anaesthesiology. Cognition, Technology & Work, 4, 82–90.

ABOUT THE AUTHORS Noa Segall is a human factors engineer and assistant professor in the Department of Anesthesiology at the Duke University Medical Center. Her interests include human performance modeling, health information technology design and evaluation, and patient safety research. She received her PhD in industrial engineering from North Carolina State University. David B. Kaber is a professor in the Edward P. Fitts Department of Industrial & Systems Engineering and an associate faculty in the Departments of Biomedical Engineering and Psychology at North Carolina State University. His research interests include cognitive modeling of human interaction with automation in various domains. Jeffrey M. Taekman is a neuro-anesthesiologist with expertise in human simulation for healthcare education and patient safety. His research is on distributable forms of immersive learning and human performance in clinical trials. He is the Assistant Dean for Educational Technology and directs the Duke Human Simulation and Patient Safety Center. Melanie C. Wright is a human factors engineer with interests in human-centered design of intelligent and integrated health care information displays and alerting systems, assessing healthcare teamwork skills, and equipment and process design to reduce medical error. She directs Patient Safety Research at Saint Alphonsus Health System and Trinity Health.