Knowledge Elicitation for Resilience Engineering in ...

Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015

175

Knowledge Elicitation for Resilience Engineering in Health Care Sudeep Hegde1, A. Zach Hettinger2,3,4, Rollin J. Fairbanks2,3,5 , John Wreathall6, Robert L. Wears7,8, Ann M. Bisantz1 Department of Industrial and Systems Engineering, University at Buffalo-SUNY, Buffalo, NY; 2National Center for Human Factors in Healthcare, MedStar Institute for Innovation, Washington DC; 3Department of Emergency Medicine, Georgetown University School of Medicine, Washington DC; 4Department of Emergency Medicine, MedStar Union Memorial Hospital, Baltimore, MD; 5 Department of Emergency Medicine, MedStar Washington Hospital Center, Washington DC; 6John Wreathall & Co. Inc., Dublin, OH;7Department of Emergency Medicine, University of Florida, Jacksonville, FL; 8Clinical Safety Research Unit, Imperial College London, Paddington, UK

1

The Critical Incident Technique (CIT) and the Critical Decision Method (CDM) have been employed successfully to elicit information about human activities, explicate expert knowledge and model decision-making in various domains. Because of their proven efficacy in naturalistic settings, we adapted these methods to develop a script for interviews about resilience in health care. The multi-stage, semi-structured format of the CIT and CDM was adopted as a broad template for the script. However, new questions and probe-types for each stage were created which conformed to the theoretical framework of resilience engineering (RE). Hollnagel’s Resilience Analysis Grid (RAG) was used to derive a majority of the probes for each of the capabilities of a resilient organization – monitoring, anticipating, responding and learning. The interview script was iteratively revised based on responses to pilot interviews as well as feedback by human factors and RE experts. The final script was used to successfully interview physician and nurse-providers at various organizational levels and units.

Copyright 2015 Human Factors and Ergonomics Society. DOI 10.1177/1541931215591036

INTRODUCTION Resilience, the “ability to adapt or absorb disturbance, disruption, and change”, has come to the fore as a new paradigm in safety research (Patterson, Woods, Cook, & Render, 2007). Unlike the traditional approach to safety which tries to mitigate failure, resilience engineering (RE) looks instead, at what goes right as well as what goes wrong. Rather than finding root causes, RE proactively seeks to understand the forces that drive everyday performance variability and systemically support resilient behaviors that lead to success. Given the uniqueness of each patient’s case, health care can be seen as a highly complex sociotechnical system with regular variability in process and performance. With conventional methods of improving safety, e.g. root cause analysis, having only limited efficacy, RE has gained increasing importance in health care (Hettinger et al., 2013). However, while the foundational concepts of RE are well crystallized, tools, techniques and methods are only beginning to evolve. This paper describes an interview method that was developed to elicit information related to workplace resilience from health care professionals. Examples of workplace resilience include workarounds, innovations, adaptive strategies and improvisations that contribute to favorable outcomes (Debono et al., 2013; Woods, 2006). Some evidence of factors which enable resilience were found in a prior study on the efficacy of root-cause analysis (RCA) solutions, in which hospital staff were asked questions about what makes safety interventions work (Hegde et al., 2013). Although not the primary aim of that study, interviews yielded interesting information about initiatives, strategies and workarounds developed by frontline staff that play a significant role in creating safety. These narratives warranted more detailed and systematic investigation because they contained valuable information to improve safety.

There are many ways in which information about safety and resilience can be obtained, including interviews, observations, incident reports and case documentations (Benn et al., 2009; Cook & Nemeth, 2006; Jeffcott, Ibrahim, & Cameron, 2009; Pasquini, Pozzi, Save, & Sujan, 2010). Morel, Amalberti, and Chauvin (2008) conducted an interactive simulation of sea-fishing skippers to model trade-off decisions between safety and production goals. Surveys and questionnaires with ratings for safety-related themes have been used to model resilience (Gillespie et al. 2007). Praetorius and Hollnagel (2014) have used a combination of methods for studying resilience in the domain of maritime traffic management. These include study visits and observations with informal conversational interviews (Patton, 2002), focus groups and semistructured interviews. Similarly, focus groups have also been employed by Rankin, Lundberg, Woltjer, Rollenhagen, and Hollnagel (2014) to gather data related to adaptations in high-risk work. However, to our knowledge, there does not exist a prescribed method of interviewing specifically for resilience in health care. The proposed method draws on methods developed for naturalistic decision making research - the critical incident technique (CIT) and the critical decision method (CDM) (Flanagan, 1954; Klein, Calderwood, & Macgregor, 1989) – to develop an interview script geared towards eliciting information about system attributes emerging from the theoretical foundations of resilience engineering. The script was used for interviewing health care professionals, mainly frontline providers and nurses, in a study aimed at understanding resilience in health care in terms of achieving patient safety and quality of care. Basis for Choosing CIT and CDM as Primary Templates. CIT is a robust procedure for gathering important facts concerning behavior in defined situations. Past incidents or

Downloaded from pro.sagepub.com at University at Buffalo Libraries on November 13, 2015


circumstances can be analyzed in terms of the challenges faced, the steps taken to meet the challenges by individuals who confronted the situation, and reasons for success or failure of the decision. CDM is a variant of the CIT that has been successfully adapted and widely applied to understand decision-making in safety critical systems. The CDM is a structured approach for analyzing actual challenging cases that the domain practitioner has experienced. The following elements of CIT and CDM are considered particularly useful in terms of developing a similar technique to study resilience:  Knowledge elicitation for naturalistic conditions – questions relate to how people actually operate, in real scenarios. This aligns with the core philosophy of RE is to understand work as it actually gets done, as against work as imagined) (Dekker, 2014).  Focus on critical and non-routine events – a significant subset of events that represent resilience include critical or potentially critical situations and involve adaptation, maneuvering towards safety and resurrection.  CDM is designed for understanding expert cognition, but has the potential to illuminate situational details, environmental cues and other factors that influenced decision making. Such information has significant bearing on analysis for RE.  Probes to make the respondent reflect on their experience and remember details often lead them to recognize attributes of a situation that contributed to decision-making. This aspect of knowledge elicitation is highly useful in identifying properties of the system that silently contribute to its efficacy even as human players interact with these.  The semi-structured format of interviewing allows for narrative responses which provide rich contextual detail of the situation. Caveats to Consider While Adapting CIT and CDM for RE. Resilience is not only found during non-routine or critical incidents, but also, perhaps more so when incidents are not happening, i.e. during routine functioning with safe outcomes. Therefore the frameworks offered by CIT and CDM are less useful in learning about safety mechanisms and processes that form the underbelly of a successful system. Understanding system resilience goes beyond expert cognition to encompass information about systemic functions, structures and processes. Therefore questions need to extend from situational characteristics to systemic elements that contribute to resilience. In a sociotechnical system, information and knowledge exist as part of distributed cognition. Therefore, there are the features of resilience that people are conscious about, e.g. workarounds, innovations, good practices, initiatives, policies etc. But there are also other factors that people are not always aware of, e.g. enculturated behaviors, evolving processes, gradual changes, and unacknowledged contributions to safety by other staff in the organization. Such knowledge, whether it’s tacit or explicit, is distributed across the members and components of the system and not necessarily in the mental models of individuals (Hutchins,

176

2000). Therefore critical incident interviewing by itself may not lead to a capture of distributed or tacit knowledge in its entirety. METHOD Approach. Similar to CIT and CDM, this interview script is arranged in a multi-stage, semi-structured format in which the interviewee provides an account of their exemplar in narrative style while the interviewer uses probes to obtain all relevant details for later analysis. However, the actual questions and probes we developed were informed by the theoretical underpinnings of RE, as described in the next section. Pilot interviews were conducted by the first author with health care professionals, human factors experts and industrial engineering graduate students among the participants. The interview script was also reviewed by experts of human factors and resilience engineering. Feedback collected from all reviewers and pilot interviewees was used to revise or augment the questions to better meet the objectives of the interviewing technique. Additionally, once the study began, the script was continually refined as the interviews progressed. Participants: Once the script with the questions was finalized, it was used to conduct interviews of health care professionals from a large multi-specialty hospital group. 18 participants were interviewed for up to an hour each over the phone – 7 attending physicians, 4 residents, 4 nurses, 1 fellow, 1 senior clinical director and 1 senior safety administrator. The participants were associated with various units including the emergency department, internal medicine and surgery. Description of the Interview Script. The interview is broadly divided into five stages. Questions within each stage conformed to the four capabilities of a resilient system – monitoring, responding, anticipating and learning (Hollnagel, 2011b). The interview stages are described below: 1. Exemplar selection: Guiding questions were asked to help the interviewee in selecting a resilience exemplar from their own experience working in a hospital. The exemplars could be of two kinds: instance-based and process-based. The instance-based based guides pertained to three of the resilience capabilities: a) handling (responding) - “an instance in which you or your team’s handling of a challenging or complex patient situation helped ensure the safest possible outcome for the patient. b) anticipating - “an instance you were involved in, in which you anticipated an unfavorable situation and took necessary steps to avert harm to the patient.” c) noticing a problem or discrepancy (monitoring) - “an instance in which you noticed a discrepancy during a process and helped prevent an adverse event.” The process-based guide pertained to a “’good practice’ or preventive measure incorporated in routine procedure.” The interviewer would offer to elaborate each instance with examples. However, the respondents almost never required the examples and were usually immediately ready with an exemplar case.



2. Opening narrative: Following the selection of the exemplar, the respondent was asked to narrate the instance in terms of the background, sequence of events, people involved and how the situation came under control. In the case of the process-based exemplar, they were asked to explain the conditions in which they employ their ‘good practice’, how they do it and how it helps them achieve safety. They are also asked to provide any examples of specific cases or incidents, if available, where they think their measures may have contributed to safety and resilience. As in the CDM, this procedure helped accomplish two important goals: (i) it allowed the interviewer to be acquainted with the context of the case, and (ii) it helped activate the respondent’s memory of

Category Communication MONITORING Problem detection Checklist

177

the situation. The interviewer made no attempt to guide the narrative at this stage and interrupted the respondent only in case of minor clarifications. 3. Structuring the narrative and confirmation of timeline: After the exemplar was explained, the interviewer repeated back the description of the case to the respondent to achieve the following objectives: (i) establish a coherent structure for the case in terms of the various elements, sequences and processes involved, (ii) confirm the details and build a shared awareness of the case after consolidating both perspectives, i.e. the interviewee’s and the interviewer’s, and (iii) clarify inconsistencies, fill ‘gaps’ or missing facts.

Question “What kinds of communication took place between you and your colleague(s) and at what point?”

“How did you become conscious that there may be a problem?” “Was there a prepared list of symptoms to check for as part of your protocol to recognize the event/hazard or was it from experience or just something you realized in that moment/while working with the issue? Previous experience “Did any previous experience or observation contribute to your assessment of the situation and your decision?” RESPONDING Knowing what to do “What was it about the situation that let you know what to do? Was there a standard protocol specifically for such a situation?” “What made you decide on this particular procedure or preventive measure?” What Went Right “What is it that you and/or your team did right that helped bring the situation under control or prevent patient harm?” “Are there any other factors that contribute to the success of this intervention – e.g. availability of a certain technology, knowledge of co-workers, work-processes, policies, environmental factors etc.?” Resources: “What resources (e.g. time, materials, technology, people, expertise, standards/protocols) were/are necessary or helpful in this situation or at this point?” Constraints: “What were the main challenges or concerns and how were they overcome?” “Are there any challenges or concerns to making this intervention a part of your routine procedure? How do you manage those?” Adaptive Measure: “As the situation changed, how did you adapt?” “Does this practice apply to all potential situations or are there certain contingencies for which you have to adapt on a need-basis?” ANTICIPATING Expertise: “To what extent would you say that your experience/expertise helped you to anticipate the problem?” Acceptability of “The risks that you recognized – in that situation, was there some sort of a guideline available determine if Risk: those potential risks were acceptable or not – or was it purely your own judgment that made you act proactively?” LEARNING Internalizing: “How did you incorporate this action into day-to-day practice/ How do you ensure that a similar risk is avoided in the future?” Systems Support/ “Was there any systems or management-level support that you can identify that helped with this intervention Follow-up: (resources, standards, protocols, leadership, culture)?” Did the event lead to any changes would reduce the chances of the event happening again, or make responding to these events easier?” Continuity: “In general, when new staff members join this unit, are they acquainted with this issue and the method you’ve to counter it?/ Are new staff members trained to deal with such a situation?” Lesson-sharing: “Were lessons from this experience shared with others in the unit/other units?” Table 1a: Typical deepening questions and probes categorized under the four resilience capabilities and color-coded by frame type



4. Deepening questions and probes: Following the establishing of a shared awareness of the resilience exemplar, probing questions were used to address the various aspects of the case that are relevant from an RE standpoint. The questions were framed in three ways, instance-based, processbased and generic, as applicable to the related exemplar. These were listed as color-coded alternatives – red, blue and black fonts respectively, for the interviewer to conveniently choose the appropriate framing for each question. The first question was about communication events that took place during the entire episode or whenever the practice is applied, as the case may be. The discussion exclusively about communication served to add a layer of information to the narrative about interactions between people, and channels of communication and coordination. Communication is seen to play a vital role in resilience and is a feature common to all the four capabilities

Category

178

as an enabler as well as a link (Nyssen, 2011). Further questions were again broadly aligned with the four resilience capabilities. Many of the probes were derived from the list provided in the Resilience Analysis Grid (RAG) (Hollnagel, 2011a) and adapted based on relevance and importance in the health care domain. Others were formed based on definitive themes occurring in the RE literature. Table 1a lists some of the typical probing questions used for this stage of the interview. 5. Hypothetical querying and miscellaneous probes: A final set of hypothetical questions was asked to examine the generalizability of resilient actions described in the exemplar to other similar situations and opportunities to improve the specific resilience capabilities discussed. These additional probes are listed in Table 1b.

Question

Future changes

“What kind of changes would you suggest in order to ensure that future or other providers are able to successfully manage similar situations?”

Generalizability:

“Would this action have been possible at any other time and in any other unit? Or were there certain factors specific to the time, place and people involved that played a significant role in enabling success?” “Does everyone in the unit apply the same procedure as you? Do others have their own ways of dealing with the issue?” “Would it have been possible to handle the issue/situation more easily or efficiently had additional resources been available? What specific resources (time, materials, people, expertise, technology etc.) would have been useful in enhancing your response in this case?” “Would it help to have certain resources made available by the hospital to deal with the issue more effectively? What specific resources would be useful in overcoming this potential hazard?”

Responding – Resources:

Anticipating – Information:

“What type of information or system awareness would have helped you anticipate this situation better?” Table 1b: Additional questions including hypotheticals

DISCUSSION The interview method was developed with the objective of identifying opportunities to enhance resilient processes and behaviors through systems-level intervention and support. In this respect, the described protocol deviated from the cognitive probe-types used in CDM towards prompting the respondent to provide information about RE-based themes. Unlike its predecessors, this interviewing technique goes beyond the traditional case-based approach to also elicit information about successful practices or protocols incorporated as part of routine procedure. Example responses to some of the probes are presented in Table 2. A major strength of this interview script is that it affords the resilience analyst a ready frame of reference in that responses are already classified by the question category, i.e. the type of resilience capability and type of probe. However, it is not necessary that the response truly belongs to the same category as its question. Therefore caution must be maintained to not allow such a frame of reference to bias the response classification or ‘force’ it into an imprecise category. Another

advantage of this technique is that can lead to discovery of significant facts related to past experiences of the respondent. It was often witnessed during the interviews that the respondents were themselves initially not aware of what they did right in certain situations or of favorable systemic factors that played a role in success. The act of thinking through why their practices succeed makes them remember or realize facts about the scenario being related that can be of value in understanding the nature of work practice and the sources of resilience therein. Interviews were only conducted on phone by a non-medical researcher. It is therefore reasonable to presume that the interview could be more effective if supplemented by actual domain exposure such as shadowing and field observations. Future research: The interview script could be extended or generalized beyond domain-specificity and adapted for specific contexts. Within the domain itself, the script can be diversified to capture capabilities of RE at different organizational levels, e.g. individual, group and organization,



and to investigate possibilities to systematically support

179

evolutionary pathways of these capabilities across the system.

Probe Example Response Problem detection At the start of the shift – “I noticed that he wasn’t on oxygen, so I read his orders in the morning (something I do for every patient)… Then I went to the nurse (from previous shift) and asked her to tell me about the patient because that’s what we do… I noticed this patient needs to be on a 24-hr oxygen monitoring system. She said ‘oh no I didn’t do that’” What went right

“I think the timing of the ultrasound…early in the process…literally 5 minutes after I met the patient – enabled us to get data pretty quickly.”

Expertise

“I really believe that a lot of it is exposure and experience. So after you’ve experienced and seen problems, you know what to look out for.”

Lesson-sharing

“A lot of times the physicians and nurses, when they are through with a critical case, after it’s all done and said, after the patient is stabilized – we may do a group huddle; we talk about what went wrong, what went right etc.” Table 2: Example responses from the interview for typical probes

REFERENCES Benn, J., Koutantji, M., Wallace, L., Spurgeon, P., Rejman, M., Healey, A., & Vincent, C. (2009). Feedback from incident reporting: information and action to improve patient safety. Quality and Safety in Health Care, 18(1), 11-21. Cook, R., & Nemeth, C. (2006). Taking things in one’s stride: Cognitive features of two resilient performances. Resilience engineering: Concepts and precepts. Aldershot, UK: Ashgate Publishing, 205-220. Debono, D. S., Greenfield, D., Travaglia, J. F., Long, J. C., Black, D., Johnson, J., & Braithwaite, J. (2013). Nurses’ workarounds in acute healthcare settings: a scoping review. BMC health services research, 13(1), 175. Dekker, S. (2014). Resilience engineering: Chronicling the emergence of confused consensus. Flanagan, J. C. (1954). The critical incident technique. Psychological bulletin, 51(4), 327. Hegde, S., Hettinger, A. Z., Fairbanks, R. J., Wreathall, J., Lewis, V., Wears, R., & Bisantz, A. M. (2013). A Bottom-Up Approach to Understanding the Efficacy of Event-Analysis in Healthcare Paradigm Shift from Safety to Resilience Engineering. Paper presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Hettinger, A. Z., Fairbanks, R. J., Hegde, S., Rackoff, A. S., Wreathall, J., Lewis, V. L., . . . Wears, R. L. (2013). An evidence‐ based toolkit for the development of effective and sustainable root cause analysis system safety solutions. Journal of Healthcare Risk Management, 33(2), 11-20. Hollnagel, E. (2011a). Epilogue: RAG–the resilience analysis grid. Resilience engineering in practice-A guidebook, 275-296. Hollnagel, E. (2011b). Prologue: the scope of resilience engineering. Resilience engineering in practice: A guidebook, xxixxxxix.

Hutchins, E. (2000). Distributed cognition. International Encyclopedia of the Social and Behavioral Sciences. Elsevier Science. Jeffcott, S., Ibrahim, J., & Cameron, P. (2009). Resilience in healthcare and clinical handover. Quality and Safety in Health Care, 18(4), 256-260. Klein, G. A., Calderwood, R., & Macgregor, D. (1989). Critical decision method for eliciting knowledge. Systems, Man and Cybernetics, IEEE Transactions on, 19(3), 462-472. Morel, G., Amalberti, R., & Chauvin, C. (2008). Articulating the differences between safety and resilience: the decision-making process of professional sea-fishing skippers. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(1), 1-16. Nyssen, A.-S. (2011). From myopic coordination to resilience in socio-technical systems. A case study in a hospital. Resilience Engineering in Practice: A guidebook. Pasquini, A., Pozzi, S., Save, L., & Sujan, M.-A. (2010). Requisites for successful incident reporting in resilient organisations. Resilience Engineering in Practice: A Guidebook. Aldershot, UK: Ashgate. Patterson, E. S., Woods, D. D., Cook, R. I., & Render, M. L. (2007). Collaborative cross-checking to enhance resilience. Cognition, Technology & Work, 9(3), 155-162. Patton, M. Q. (2002). Qualitative Research & Evaluation Methods. Los Angeles: Sage Publications, Inc. Praetorius, G., & Hollnagel, E. (2014). Control and Resilience Within the Maritime Traffic Management Domain. Journal of Cognitive Engineering and Decision Making, 8(4), 303-317. Rankin, A., Lundberg, J., Woltjer, R., Rollenhagen, C., & Hollnagel, E. (2014). Resilience in everyday operations a framework for analyzing adaptations in high-risk work. Journal of Cognitive Engineering and Decision Making, 8(1), 78-97. Woods, D. D. (2006). Essential characteristics of resilience. Resilience engineering: Concepts and precepts, 21-34.
