Using the ShadowBox™ Method to Detect the ...

Using the ShadowBox™ Method to Detect the ‘Investigator’ and ‘Proceduralist’ Mindsets in Frontline Social Workers Emily Newsomea, Corinne Wrighta, Gary Kleina, Jan Floryb, and Amy Bakerb a ShadowBox LLC b Annie E. Casey Foundation This paper describes using a modified version of the ShadowBox training method for capturing cognitive characteristics of expertise in the Child Protective Services (CPS) field. Researchers developed realistic CPS scenarios and performed a content analysis on participants’ qualitative responses to decisions, leading to the identification of the characteristics of the Investigator and Proceduralist mindsets. We then compared Investigative preferences between groups categorized based on Department Chiefs’ perceptions of expertise to compare two models of CPS expertise. Results indicated a mismatch in the two models, potentially due to the general over-emphasis of rules and procedures in CPS organizations. Future research will use the cognitive characteristics of expertise identified in this study to develop a training program for new and novice frontline social workers. INTRODUCTION Like most operators working within complex domains, Child Protective Services (CPS) frontline personnel must make sense of ambiguous information and make decisions while faced with uncertainty. These workers and supervisors must handle situations in which information is not always complete, facts are often disputed, and there is extreme pressure to complete investigations within a given timeframe (Budd, 2005). In addition, most CPS agencies have a high turnover rate, making it necessary to get new trainees up to speed quickly. To ensure consistency in decision-making from worker to worker, many child welfare agencies have placed emphasis on rules and procedures in the form of structured decision-making systems (Schwalbe, 2004). These systems require workers to use risk and safety assessment tools when making complex decisions. These assessment tools are based on empirically evaluated risk factors; the presence or absence of these factors in an evaluation of a child’s situation determines the level of CPS agency response (Shlonsky & Wagner, 2005). Examples of risk factors include the presence of domestic violence or drug use in the home, lack of adequate housing, and prior involvement with CPS (Shlonsky & Wagner, 2005). These assessments are designed to assist workers in making tough decisions, to ensure that workers are making consistent decisions, and to target the children who are most in need of intervention (Gillingham & Humphreys, 2009). While effective in theory, these tools may have some disadvantages. Researchers have indicated that structured assessment tools are not always used as intended by their developers, and can be manipulated to back up the worker’s own perception of families under investigation (Gillingham & Humphreys, 2009). Some workers view the assessment tools as an administrative burden, which can at times interfere with their main priority—keeping children safe (Munro, 2011). The implementation and increasing level of reliance on these tools has raised important questions about the degree to which these tools improve frontline practice and how they affect the development of expertise. Making assessment tools mandatory makes an assumption that all frontline workers need a structured tool to make an informed decision,

regardless of level of experience. The Recognition Primed Decision (RPD) model (Klein, 1998) postulates that experts rely on intuition, specifically on recognizing patterns and mentally simulating situations, to make tough decisions. Novices, on the other hand, rely on procedural knowledge to guide their decisions (Klein, 1998). Despite their ability to assist in decision-making, structured assessment tools can act as a barrier for identifying experts in the CPS field. Since the tools often don’t require workers to provide their rationale behind important decisions, they don’t require critical thinking in order to develop expertise in workers. Additionally, assessment tools make it challenging for supervisors to distinguish experts, who rely on their intuition and experience to make decisions, from novices, who count on the assessment tools as a basis for making decisions. There is currently little training to facilitate the development of an expert mindset in practitioners who are used to relying exclusively on assessment tools (Munro, 1999, 2005, 2011). But before a training program can be implemented, one must identify what, precisely, differentiates expert practitioners from novices. The goal of this study was to identify patterns in the cognitive tendencies of expert and novice frontline social workers using the ShadowBox method, and to compare these patterns to existing perceptions of social work expertise. PRACTICE INNOVATION To identify cognitive tendencies of expert frontline child protective personnel, we employed a method that would allow us to examine the thought processes of CPS workers and supervisors. The ShadowBox training method was originally developed by a (now retired) Battalion Chief with the New York Fire Department, Neil Hintze (2008), in order to allow trainees to get a glimpse of the inner thought processes of expert firefighters. The ShadowBox method has since been refined into a training tool that can be used across a variety of domains to overcome a major problem encountered with traditional training—the bottleneck created by trying to get a large number of novices up to the skill level of experts (Klein, Hintze, & Saab, 2013). In this study, we use a modified

version of the ShadowBox method to capture the cognitive patterns of novice and expert frontline CPS personnel, instead of provide training. Cognitive Task Analysis (CTA) (Crandall, Klein & Hoffman, 2006) and the Critical Decision Method (CDM) (Klein, Calderwood & MacGregor, 1989) are knowledge elicitation methods that can be used to identify patterns in tacit knowledge. The ShadowBox method grows out of these methods and allows many participants’ thought processes to be directly compared to one another. ShadowBox training consists of realistic scenarios with embedded decision points. These decision points typically focus on a variety of cognitive strategies, such as: monitoring for cues, selecting best courses of action, identifying key pieces of information, or prioritizing goals (Klein, Borders, Wright & Newsome, 2015). Participants respond to decision points by ranking the options provided. The participants then view the responses from Subject Matter Experts, comparing their responses to those of the experts and generating insights about the scenarios (Klein, Hintze & Saab, 2013). In prior projects, we have provided effective ShadowBox training in domains in which expertise is characterized by a mastery of tacit knowledge, such as training military personnel to think like Good Strangers (e.g., Klein, G., Klein, H.A., Lande, Borders, & Whitacre, 2014; Klein et al., 2015) and training an Operator mindset in petrochemical operators. Like those domains, CPS is a field in which mastery of tacit knowledge is crucial, but unlike other domains, CPS workers’ decisions happen within a complex bureaucratic system. Because of the complexities that arise from the system, it is not always easy for workers to determine how their daily decisions directly affect the outcomes of families and children. Additionally, frontline workers and supervisors rarely get feedback about whether they made correct or incorrect decisions, because their cases are often closed or transferred to other departments (e.g., foster care, adoption, ongoing services) without evaluating the effectiveness of decisions made on the frontline. ShadowBox scenarios provided the means to analyze and compare patterns of cognition between workers in this complex domain. In the following sections we demonstrate how the ShadowBox method can be used to identify cognitive tendencies in frontline child protective personnel. By comparing the participants’ rationale for their decisions, we aimed to identify cognitive characteristics that distinguish the expert caseworkers and supervisors from the average ones. Procedure Researchers collected data at a large, urban Child Protective Services agency in the Midwestern United States. Department Chiefs were asked to divide participants (N = 54) into 4 groups based on position and level of expertise: Highly Proficient Supervisors (n = 9), Average Supervisors (n = 8), Highly Proficient Workers (n = 8), and Average Workers (n = 29). We provided a list of criteria to help Department Chiefs differentiate experts using two models of expertise: the RPD model (Klein, 1998) and CPS-specific characteristics. Characteristics of expertise based on the RPD model of

expertise (Klein, 1998) included being above average at the following: picking up on cues, taking the perspective of others, anticipating consequences, making sense of situations, and improvising workarounds when faced with unusual situations. We also provided characteristics of expertise based on social work practice (Munro, 2011), which were operationalized as being above average at the following: the ability to spot safety concerns, formulate plans, work with parents, gather data, manage time effectively, and read nonverbal cues. Although we provided the Department Chiefs with specific criteria for categorizing Highly Proficient workers and supervisors, we suspect that their own judgments about employee performance, such as completion of tasks within mandated timeframes, played a role in their group determinations. Data were collected in compliance with all Institutional Review Board procedures. Researchers guided participants through five ShadowBox scenarios. The scenarios were based on case files and experiences of knowledgeable practitioners in the child protective field. Each scenario presented a different child protection situation—an infant with an unexplained injury, an allegation of sex abuse, a teen mom in foster care, a mother with a serious mental illness, and a custody battle involving a teen, his mother, and grandmother. The scenarios and decision points were read aloud to participants as they also followed the passages in paper packets. After reading each passage, we allowed the participants two minutes to rank decision point options and write their rationale for their choices. The scenarios contained 3 to 8 decision points each, with a total of 25 decision points. The process took approximately 4 hours for each group. Next we report the findings of two analyses from the data: content analysis and a comparison of our model of expertise, the Investigator/Proceduralist distinction, versus the Department Chiefs’ Highly Proficient/Average worker distinctions. FINDINGS After an initial review of the 25 decision points, we selected eight that seemed to differentiate participants based on cognitive characteristics described below. These eight decision points had both a clear option that some participants prioritized and another option that other participants prioritized. In other words, these were the most polarizing decision points. We performed a content analysis of the participants’ rationale behind these eight key decision points to find cognitive characteristics that differentiated participants. We also examined the degree to which each group preferred the option that corresponded with our proposed categories to determine how well the separation of the groups a priori matched our model. Content Analysis Researchers performed a content analysis to examine the participants’ responses to the eight key decision points. By analyzing the rankings along with the rationale of the

participants, two unique patterns emerged: those who were highly inquisitive critical thinkers and those who relied on rules and procedures outlined in structured assessment tools to justify decisions. We called the first group “Investigators” and the second group “Proceduralists.” Tables 1 and 2 provide excerpts of rationale from a single decision point to illustrate examples of Investigator and Proceduralist characteristics, respectively. The tables also indicate the participants’ a priori rank of either Average or Highly Proficient, as established by their respective Department Chiefs. The examined decision point asks participants to prioritize the most critical issue in an investigation of potential abuse of a teenager, Jason, by his maternal grandmother, Mary, who is a former corrections officer and entangled in a custody battle over Jason with his mother, Susan. The participants could choose to: gather more information about how the child is disciplined by his grandmother; meet with the child to learn about his view of the situation; gather information from the child’s school about his attendance and behavior; investigate the possibility of a firearm in the home; or investigate the custody battle between the mother and grandmother. Proceduralist rationale, presented in Table 1, did not prioritize talking to Jason to get his point of view about the situation (see all participants in Table 1). The allegation of a gun in the home prompted Proceduralists to prioritize making sure the rules and procedures pertaining to keeping a firearm in the home were followed (e.g. that the gun was kept unloaded, under lock and key out of the reach of children) (see Participant 19 and 10). Proceduralists did not indicate that the custody struggle between the mother and the grandmother had any positive or negative bearing on which option they prioritized (see all participants in Table 1).

whether the procedures regarding firearms in the home were followed. In addition, Investigators were not concerned with the custody struggle, as it did not impact the child’s safety (see Participant 18 and 29). Table 2 The Investigator characteristics of Supervisors and Workers Supervisors Participant 3 (highly proficient) “Since I was unable to interview Jason the day before, interview Jason to get his view on living with his grandmother. While at the school discuss any concerns the school has. Ask MGM [maternal grandmother] how she disciplines Jason and has she ever threatened him with a gun." Participant 18 (average) “Safety is the initial concern. The presence of a gun was only speculated, so the allegation of inappropriate discipline is primary. The custody dispute does not address safety. Talking with Jason will give insight into the situation.” Workers Participant 14 (highly proficient) “Since you are in Mary's home, you should get all information possible. Getting Jason's input is important and he is old enough to speak for himself. Mary is a former corrections officer. Having a gun in the home would not be a surprise or necessarily a problem." Participant 29 (average) “I would interview Jason about the current allegations and determine if there is a safety concern. While at school I would gather info from school about his behavior/attendance. I would schedule to meet with Mary again to discuss what is going on between herself and her daughter. The possibility of a weapon in the home is probably true since she was a corrections officer. See about permit.”

Investigative Preference analysis

Table 1 The Proceduralist characteristics of Supervisors and Workers Supervisors Participant 8 (highly proficient) “Identifying the likelihood of threats is critical when they are identified and present.” Participant 19 (average) “Need to make sure if there is a weapon in the home that it is safely locked and stored. After that discussing if/how discipline is used in the home.” Workers Participant 10 (highly proficient) “Possibility of gun in home. If it is, ensure that the gun is placed in lock box, out of reach of child. Discuss discipline technique, discuss his needs and services if needed. Have MGM sign release and give consent for CW to interview child in school.” Participant 54 (average) “Assess in home safety/risk while at the home. Then interview Jason-assess safety/risk and get collateral info from school.”

In sharp contrast to Proceduralists, Investigators were more concerned with how Jason felt about the situation (see Participant 3, 14, and 29 in Table 2). They determined that it might not be unusual for a retired corrections officer to have a gun (see Participant 18, 14 and 29). They did not focus on

We compared our data split using Investigators and Proceduralists to the Average and Highly Proficient categories grouped by the Department Chiefs. Table 3 shows the average percentage of the eight decision points on which participants in each group chose meaningful options, either Investigator or Proceduralist, as their top option. Few participants chose other options that were seen as irrelevant to the Investigator and Proceduralist mindsets, such as pursuing an alternative theory or gathering information from a collateral source. These options are not included in Table 3. Table 3 The average investigative preferences of Workers and Supervisors Investigator Proceduralist Number of Group match match participants percentage percentage Highly Proficient Supervisors

9

50%

46%

Average Supervisors

8

55%

44%

Highly Proficient Workers

8

33%

66%

Average Workers

29

48%

50%

We wanted to examine whether the Department Chiefs’ criteria for Highly Efficient/Average supervisors and caseworkers matched our Investigator/Proceduralist findings. We questioned if most of the Highly Proficient participants would also be Investigators and if most Average participants would be Proceduralists. Three of the four groups (Highly Proficient Supervisors, Average Supervisors, and Average Workers) showed equal preference for both the Investigator or Proceduralist, indicated by the similar frequencies with which participants in each group chose the Investigator and Proceduralist options. We then used independent samples Ttests to compare the Investigative and Proceduralist preferences of the participants in the Highly Proficient Workers group to the preferences of the other groups. The group identified as Highly Proficient Workers by Department Chiefs showed a significantly higher level of preference for the Proceduralist options when compared to Average Workers, t(35) = 2.64, p = .01 and when compared with Highly Skilled Supervisors, t(15) = 2.59, p = .02. Conversely, they showed a significantly lower preference for the Investigator options, both when compared to Average Workers, t(35) = 2.72, p = .01, and when compared to Highly Skilled Supervisors, t(15) = 3.43, p < .01. DISCUSSION The goal of this study was to employ the ShadowBox method to capture cognitive tendencies of frontline Child Protective workers and supervisors. We captured the characteristics of two distinct mindsets: the Investigator mindset and the Proceduralist mindset. These mindsets are compilations of characteristics that led us to develop speculative trends about the nature of expertise in Child Protective Services. In the following paragraphs we will first present characteristics and habits of each mindset. We will then compare the Investigator/Proceduralist distinction to existing literature about expertise in CPS, and compare and contrast the Investigator mindset against the criteria that was used to separate participants into levels of expertise before the study. We also will discuss methodological and theoretical implications, limitations, and directions for future research. In general, we found that Proceduralists did not have a desire to probe ambiguous facts, instead taking them at face value. They wanted to make sure children were safe, but didn’t show that they were curious about what happened inside the home after caseworkers left. They were likely to see ambiguous pieces of information as supporting their preconceived opinions. They tended to focus on procedural knowledge and often could not conceive of breaking rules under any circumstances. In contrast to the Proceduralist mindset, we found that Investigators probed ambiguous facts to make sure they understood the broader context of each family involved in an investigation. They were curious about what life was like for a child after the caseworker left the home and searched for all facts, not only the ones that supported their position. They had a strong sense of professional identity, ensuring that their standard of work didn’t degrade under pressure and that they fostered good rapport with families.

Next, we look to the literature for support for the Investigator/Proceduralist dimension. The characteristics of Investigators as experts in the Child Protective field are in line with descriptions of expertise in other research. Fook, Ryan and Hawkins (1996) found that novices relied more on a rigid theoretical framework when making decisions. Additionally, Gillingham (2011) found that newer workers expressed a dependence on assessment tools to make decisions. This is in line with our findings in the present study; supervisors and workers who displayed characteristics of the Proceduralist mindset frequently relied on rules and procedures as their rationale behind complex decisions. In contrast, expert social workers were better able to assess complex factors within a situation, and used their intuition, based upon years of experience, instead of rules and procedures, to help make tough decisions (Fook et al., 1996). Gillingham (2011) found that more experienced workers expressed minimal reliance on assessment tools when making decisions. This corresponds to the characteristics of the Investigator mindset present in our findings; Investigators did not rely on rules and procedures as a basis for making decisions. The distinction between Investigators and Proceduralists did not completely match the definition of expertise from Department Chiefs within this particular CPS agency. The groups categorized as “Highly Proficient” did not prefer the Investigator options to the Proceduralist options, as we would expect if the two models of expertise were similar. The finding that three of the four groups did not show a preference for either the Investigator or Proceduralist mindset suggests that Investigators and Proceduralists are evenly distributed throughout the three groups. The significantly lower proportion of Investigative responses, and higher proportion of Proceduralist responses, in the “Highly Proficient Worker” category indicates that there were vastly different definitions of expertise between Department Chiefs and the Investigator/Proceduralist distinction. Specifically, the workers who the Department Chiefs identified as being Highly Proficient showed that they were more procedurally minded than their average worker counterparts, as well as their supervisor colleagues who had also been identified as “Highly Proficient.” In her review of the Child Protective system in the United Kingdom, Munro (2011) provides a possible explanation for this: Child Protective systems have become too reliant on using rules and procedures to ensure quality of practice. While adherence to rules and procedures, on which the Chiefs may have based their perception of expertise, is important for all caseworkers and supervisors, our model of expertise describes expert caseworkers and supervisors as being curious, asking more questions than their colleagues, and being more thorough in their investigations. The mismatch in defining expertise illustrates a need for separating cognitive characteristics of expertise from specific rules and procedures used by Department Chiefs to define expertise. The mismatch in definitions of expertise also represents an opportunity for organizational growth at the administrative level—valuing and evaluating investigative characteristics instead of procedural knowledge can change the culture of compliance in this particular CPS organization and many others across the nation.

Implications The Shadowbox training method is an innovative way of capturing cognitive characteristics of expertise that can be applied to various domains, but is especially effective in fields in which cognition is not necessarily linked to measurable behaviors. The ShadowBox method allows researchers and practitioners the opportunity to get inside the heads of system operators, identifying the differences of the mindsets of experts and novices. The mindsets captured in the current study converge and expand upon existing literature, which provides support for the external validity of the ShadowBox method. Accurately capturing the mindsets of expert and novice practitioners is the first and most crucial step in identifying the goals of cognitive training. By examining the differences between how experts think about situations and how novices think about the same situations, we can develop learning objectives for a cognitive training program that teaches novices how to monitor the same cues as experts and prioritize goals and courses of action the way experts would. Limitations We encountered some limitations of our approach, particularly in regard to the necessity of hypothetical reasoning and the reliability of the Department Chiefs’ selection criteria. We asked participants to put themselves into hypothetical situations and to approximate their courses of action as if they themselves were in the situation. Some participants were more able to imagine themselves in these situations than others, indicating that the fidelity between what participants said they would do and what they actually would do could reflect individual differences. Additionally, the scenarios were based on real cases, so a participant might have experienced a similar case, which could have affected their performance. However, we believe this is unlikely due to the purposefully heightened degree of complexity of the scenarios. Secondly, a variation of selection criteria emerged during a debriefing session with Department Chiefs. Some of the Chiefs indicated that they largely selected people to be in the Highly Proficient groups based on completing paperwork on time, needing little assistance from Chiefs, or generally if the employee made their supervisor’s job less difficult. This alternate heuristic could explain and possibly confound the mismatch between the Chiefs’ groupings and the Investigator and Proceduralist mindsets. This finding also highlights the necessity to use methods such as ShadowBox to establish a reliable distinction between expert and novice mindsets. Future Research We suspect that the Investigator mindset represents a set of universal characteristics for expertise in CPS. Additional data collection from other CPS agencies could support this claim. We intend to explore the use of the ShadowBox method as a means to facilitate the development of a program for

training the cognitive aspects of expertise in CPS. We will use the characteristics of expertise identified in this study as learning objectives for this training. Additionally, we intend to apply the ShadowBox method of capturing expert and novice mindsets to other complex domains in which expertise entails the development of tacit knowledge. ACKNOWLEDGEMENTS This research was funded by the Annie E. Casey Foundation. We thank them for their support but acknowledge that the findings and conclusions presented in this report are those of the authors alone, and do not necessarily reflect the opinions of the Foundation. Additionally, we appreciate the assistance and feedback we received from Patricia Rideout, John Mattingly, and Jack Whitacre. REFERENCES Budd, K. S. (2005). Assessing parenting capacity in a child welfare context. Children and Youth Services Review, 27(4), 429-444. doi: 10.1016/j.childyouth.2004.11.008 Crandall, B., Klein, G. A., & Hoffman, R. R. (2006). Working minds: A practitioner's guide to cognitive task analysis. MIT Press. Fook, J., Ryan, M., & Hawkins, L. (1996). Expertise in social work practice: an exploratory study. Canadian Social Work Review, 13(1), 7-22. Gillingham, P. (2011). Decision-making tools and the development of expertise in child protection practitioners: are we ‘just breeding workers who are good at ticking boxes’? Child & Family Social Work, 16(4), 412-421. doi: 10.1111/j.1365-2206.2011.00756.x Gillingham, P., & Humphreys, C. (2009). Child Protection Practitioners and Decision-Making Tools: Observations and Reflections from the Front Line. British Journal of Social Work, 40(8), 2598-2616. doi: 10.1093/bjsw/bcp155 Hintze, N. R. (2008). First responder problem solving and decision making in today’s asymmetrical environment. Unpublished Master’s thesis, Naval Postgraduate School, Monterey, CA. Klein, G. A. (1998). Sources of power: how people make decisions. Cambridge, Mass.: MIT Press. Klein, G., Borders, J., Wright, C., & Newsome, E. (2015). An Empirical Evaluation of the ShadowBox Training Method. Manuscript submitted for publication. Klein, G. A., Calderwood, R., & Macgregor, D. (1989). Critical decision method for eliciting knowledge. Systems, Man and Cybernetics, IEEE Transactions on, 19(3), 462-472. Klein, G., Hintze, N. & Saab, D. (2013). Thinking inside the box: The ShadowBox method for cognitive skill development. In H. Chaudet, L., Pellegrin & N. Bonnardel (Eds.) Proceedings of the 11th International Conference on Naturalistic Decision Making, Marseille, France, 21-24 May 2013. Paris, France: Arpege Science Publishing. Klein, G., Klein, H. A., Lande, B., Borders, J., & Whitacre, J. C. (2014, September). The Good Stranger Frame for Police and Military Activities. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 58, No. 1, pp. 275-279). SAGE Publications. Munro, E. (1999). Common Errors of Reasoning in Child Protection Work. Child Abuse & Neglect, 23(8), 745-758. Munro, E. (2005). Improving practice: Child protection as a systems problem. Children and Youth Services Review, 27(4), 375-391. doi: 10.1016/j.childyouth.2004.11.006 Munro, E. (2011). The Munro review of child protection: final report, a childcentered system (Vol. 8062). London, UK: The Stationery Office. Schwalbe, C. (2004). Re-visioning risk assessment for human service decision making. Children and Youth Services Review, 26(6), 561-576. doi: 10.1016/j.childyouth.2004.02.011 Shlonsky, A., & Wagner, D. (2005). The next step: Integrating actuarial risk assessment and clinical judgment into an evidence-based practice framework in CPS case management. Children and Youth Services Review, 27(4), 409-427. doi: 10.1016/j.childyouth.2004.11.007