Playback Technique using a Temporally Sequenced Cognitive Artifact

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 52nd ANNUAL MEETING—2008

523

Playback Technique using a Temporally Sequenced Cognitive Artifact for Knowledge Elicitation Janet E. Miller Air Force Research Laboratory Human Effectiveness Directorate Dayton, Ohio Emily S. Patterson VA Getting at Patient Safety (GAPS) Center, Cincinnati VAMC Institute for Ergonomics, The Ohio State University Columbus, Ohio Developing effective computer support for work done in complex socio-technical environments requires understanding the necessary expertise for the work. Preserving and examining cognitive artifacts, such as notes, made by the practitioner while working is valuable in ensuring an accurate understanding of the required expertise. However, how the notes are analyzed for discovery needs to include the fact that a note artifact has a temporal dimension. Exploiting the temporal dimension of note-making reveals the sequence of the practitioner’s thought process and relays what was considered important at points in time. This paper focuses on the development of a methodology to unfold a novice practitioner’s task performance in order to reveal the novice’s process to an expert so that future computer system requirements can be identified. An important key to the unfolding was the notes pages. Several methods of using the note sheet artifacts were explored and the resulting methodology is described.

10.1518/107118108X349025

INTRODUCTION Developing effective support for work done in complex socio-technical systems requires understanding the nature of expertise, much of which is difficult for experts to reliably articulate in response to a direct question. In addition, the nature of the environment needs to be understood, usually with a multitude of challenging attributes to consider such as uncertainty, coupling of subsystems, and being distributed (Vicente, 1999). These systems generally involve preserving human life, directly or indirectly, so developing competent, efficient support for performing tasks in these fields is critical. Examples of research done in these domains to develop support are public safety dispatchers (Hunt & Joslyn, 2000), forensic scientists (Schraagen & Leijenhorst, 2001), and patient safety (Weinger & Slagle, 2002) to name but a few. Truly understanding this type of work is difficult because much work is done ‘in the head.’ It is non-controversial during knowledge elicitation in these and other domains that participants can frequently intentionally or unintentionally provide data with extremely poor reliability, validity, and generalizability regarding cognitive processing strategies. In order to increase the reliability and warrant of collected data during knowledge elicitation interviews, many researchers employ techniques that ground elicitation by focusing on specific cases. By doing this, the interviewer can reduce “textbook” answers that describe idealized views of how work is conducted as well as generalizations that are not well-supported by personal experience or expert knowledge.

This work is not subject to U.S. copyright restrictions.

One particular knowledge elicitation technique that grounds interviews around specific cases is the so-called “Critiquing Method” (Miller et al., 2006). Critiquing takes advantage of the ability to critically assess another’s task performance, an ability which is often conducted by more skilled practitioners to train less skilled practitioners. With this approach, a practitioner’s cognitive process is sequentially revealed with “pauses” at strategic areas to elicit critiques. Similar approaches have been employed in Critical Decision Method (CDM) interviews that use timelines with critical decisions to structure elicitations, as well as experimental and field studies where practitioners provide “cued retrospective” accounts of their own cognitive processes, often with the help of video data to remind them of the details of the action. A significant challenge is how to “playback” processes compactly in order to support these techniques. One item that may be used for playback is the cognitive artifact. Norman (1991) defines a cognitive artifact as an “artificial device designed to maintain, display, or operate upon information in order to serve a representational function.” Cognitive artifacts may be formal, structured items such as a schedule at a healthcare facility (Nemeth, et al., 2006). However, the artifact could also be informal. Notes made by a practitioner performing a task are an example of an informal cognitive artifact that would help reveal cognition during a process. In this paper, we describe the results of using variations on playback techniques during piloting for a study, and in particular how the use of temporally sequenced “snapshots” of handwritten notes, as the relevant cognitive artifact, provided advantages in comparison with alternatives. We discuss how

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 52nd ANNUAL MEETING—2008

this playback technique might generalize to other domains where cognitive artifacts serve as repositories for naturally generated notations. METHOD Advantages and disadvantages were identified during the exploration of alternative approaches to “playback” of the process a novice intelligence analyst used for a critiquing study described in detail in Miller et al. (2006). In this study, the process that a novice intelligence analyst employed while answering a “Quick Reaction Task” (QRT) was provided to expert intelligence analysts to critique as a form of knowledge elicitation. The specific task was: “In 1996, the European Space Agency lost a satellite during the first qualification launch of a new rocket design. Give a short briefing about the basic facts of the incident: when it was, why it occurred, and what the immediate impacts were.” The electronic data set had 2000+ documents from open source literature, such as Aviation Week articles. The provided data set contained enough information to afford a detailed answer to when the accident occurred, why the incident occurred and what the short- and long-term impacts might be. The majority (~ 60%) of the documents contained information that input to answer the task questions. Some of the documents (~ 35%) contained information that helped to provide context, such as information about other rocket launch failures, but were not directly relevant to the specific question. Only a small portion contained completely irrelevant information (~ 5%). Technique #1: Direct observation of process. The first technique attempted was direct performance observation by experts of the novice doing the task using a think-aloud protocol. The novice had a Bachelor’s of Science in physics and had recently been assigned to his intelligence analysis role. While he had done research in college, he had not performed actual intelligence analysis. The novice was instructed to state whatever he was looking at or thinking as he completed the task. Two expert analysts watched while the novice performed his analysis. The experts made no comments to the novice while he performed his analysis so that his process was uninterrupted. At the end of the session, the two experts critiqued the novice simultaneously in a faceto-face session. While the novice appreciated the immediate feedback, this method was judged insufficient to meet the goals of the study given the disadvantages of: • The experts having to sit through even the uneventful portions of the novice’s performance such as reviewing the list of returned documents or waiting while a document opened, • The experts not being able to closely view what the novice was viewing to see what might have been missed, • No allowance for probing as that would affect the novice’s performance, • Interdependencies in the feedback from two experts since it was conducted simultaneously, making it difficult to separate the data into two critiques, This work is not subject to U.S. copyright restrictions.

524

• Impact of social “risk to reputation” considerations on the type and quality of critiquing data, particularly given that the participant was a junior member of the same organization, and • Not being able to repeat the observation for other experts to critique. Technique #2: Review video-tape of session. During the task, the novice participant was video-taped and audio-taped. The investigators considered and rejected piloting a session using video and audio data because: • The session lasted approximately 70 minutes, during which much time was spent quietly scrolling through electronic information displayed on the screen • It is frustrating to see portions of documents being scrolled through on the screen without being able to see the entire document personally in order to better understand the context • The video had the participant’s back cover some of the computer screen as well as the notepad on which notes were handwritten • The video of the computer screen flickered frequently since the screen’s refresh rate was not calibrated to the video camera to eliminate it Technique #3: Review think-aloud protocol transcript annotated with reviewed documents and supporting materials The “think aloud” verbal protocols were transcribed and used as the centerpiece for unfolding the process to an expert during a two-hour interview, organized partly by annotations about which documents were reviewed in relation to the verbal statements. Supporting materials were also provided as a hard copy package, including: the question, title lists, documents reviewed, and a copy of the actual two handwritten note sheets (Figures 1 and 2). This approach was promising, with the following advantages: • The expert did not have to be idle at any point during the critiquing session • The expert could focus on the relevant portions of the novice’s task, • The process was repeatable in that multiple critiques could be obtained in the same fashion, • The expert could closely review the documents, title list and note sheets, and • The expert could work at his leisure, including receiving the materials before the knowledge elicitation session if desired. The important disadvantage of this approach was that the expert expressed difficulty in understanding details about the process. The note sheets held the information that the novice identified as important at various stages of his analysis and, therefore, the sheets seemed to hold a wealth of information on the novice’s process. On the other hand, several questions were asked about the temporal nature of how the notes were generated, such as what information was written when, in response to what documents, and in what order. For example, the note sheet showed that the novice had found and written down the date of ‘5 June 1996.’ However, most documents in the data set that the novice opened and

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 52nd ANNUAL MEETING—2008

used listed the date as ‘4 June 1996’which was the correct date. In fact, in the transcript, the novice read aloud the correct date several times but did not make a correction to his note sheet. It was, therefore, unclear whether the circle around the (erroneous) date of “5 Jun 1996” was made immediately after writing the information, after the information had been verified by two sources, when reviewing the question tasking in the middle of the process, or at the end of the process when the analyst was determining what would be included in the briefing. In including the temporal dimensions of the note sheets, the fact was shown that the novice came upon the ‘5 June 1996’date before he found the ‘4 June 1996’date. He wrote the erroneous date down upon first encounter with the date and did not further corroborate information for that part of the task illustrating a poor analysis procedure. Technique #4: Review “chunked” think-aloud protocol transcript annotated with headers, reviewed documents, and supporting materials. A minor variation of this playback technique was successfully employed in five 90-minute critiquing sessions with novice and senior intelligence analysts where the focus was on understanding how the domain challenges and expert strategies differed across types of analytic tasks and across agencies (Grossman et al., 2007). The primary difference was that information was “chunked” and headers were provided (e.g., “Queries”, “Browsing titles by document number”, “”Opens document”) in bold to facilitate scanning the transcript.

525

notes sheets were in the package but no indication of the sequence of annotations was included. This method overcame the important disadvantage of being unable to faithfully follow the novice’s process with regard to the documents and novice but there was no explicit temporal correlation to the notes sheets. Without the temporal dimension, the expert had to make assumptions about the novice’s process. For example, the annotation was made on the note sheet “More thorough testing.” The expert had to make assumptions about what the notation meant and its value to the novice in his thinking if it could not be readily associated with presented data. Another disadvantage was that the verbal walkthrough increased the risk of the investigator influencing the critiquing content.

Figure 2. Novice Note Sheet 2

.

Figure 1. Novice Note Sheet 1

Technique #5: Verbal walkthrough of novice process, correlating novice comments and reviewed documents and title lists generated by search queries. The fourth attempt involved the same expert as with the third technique, but instead the expert was verbally walked through the process by the investigator to allow a directed correlation between novice comments and the documents and title lists being reviewed. The note sheets were included, but without the temporal dimension of development. That is, a copy of the completed This work is not subject to U.S. copyright restrictions.

Technique #6: Temporally sequenced cognitive artifact unfolding in relation to supporting documents. The final attempt, which was used in the subsequent study for all participants as it was judged superior to the alternatives, provided the expert the previous materials of the complete hard copy package of question, title lists, documents reviewed, and transcript. Additionally in this package, the note sheet was provided as 19 sequential sheets showing the various stages of the note-making process with annotations in the script so that the expert could understand which notes applied to what was going on in the rest of the novice’s analysis.

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 52nd ANNUAL MEETING—2008

526

observing experts did not have direct access to the documents or note sheet during the observations and so were not able to fully understand the process, specifically on what the novice considered important enough to write down at what time in his analysis. These results suggest that how the notes are analyzed for discovery of cognition is important and needs to take into account the reality that a note artifact is actually developed over time. DISCUSSION

Figure 3. Evolution of Novice's Note Sheet 2 In order to generate these “snapshots,” the video-tape and transcript of the novice’s process were reviewed in detail so that when the novice made which annotations were definitively captured. The result was a re-creation of the note sheets in Powerpoint• to allow an unfolding of the novice’s process over time. Figure 3 illustrates how novice’s note sheet 2 evolved over time. It was readily apparent to the investigators and the expert that there was more information about the cognitive processing strategies employed in seeing the development of the note sheet starting with 1 and then going to number 5 compared to the compiled note sheets alone (Figures 1 and 2). The different stages of the note sheets’ development were coordinated with the comments the novice made at that point of time and with the documents that he was viewing on screen. RESULTS Technique # Focused Time Easy to follow cognitive process Easy to follow briefing rationale Time for reflecting

1 N N

2 N N

3 Y N

4 Y N

5 Y Y

6 Y Y

Y

Y

N

N

N

Y

N

Y

Y

Y

Y

Y

Separation of Reviewers Spare Reputation

N

Y

Y

Y

Y

Y

N

N

Y

Y

Y

Y

Repeatable

N

Y

Y

Y

Y

Y

Table 1. Summary of Technique Comparison The different techniques can be compared by a variety of attributes a shown in Table 1. By these dimensions, Technique 6 was superior. All of the techniques employed yielded comments concerning the novice’s cognitive process. These were grouped into the categories of information selection, corroboration/resolution, and story construction (cf. Patterson, 1999; Elm et al., 2005). While all comments were valuable, more comments were generated with Technique 6. Although this might be due to individual differences, it is our belief that more comments were yielded at least in part because the novice’s process itself was better understood with the temporally sequenced artifact. An interesting point is that the sequenced method provided more comments compared to the direct observation approach which is often considered the ‘gold standard’ for understanding work. However, the This work is not subject to U.S. copyright restrictions.

Developing effective systems to support tasks done in complex socio-technical system environments requires understanding the expertise needed to perform the tasks. While the computer systems community currently uses a variety of requirements analysis methods, including use cases and prototypes, recently, the community is incorporating more of the methods developed by other communities such as the psychological and ethnographic communities. This has required paying attention to more details within a user’s environment than a user might mention in a stakeholder interview or the like. Cognitive artifacts hold some of these important details about the tasks to be addressed by computer systems. As stated above, formal cognitive artifacts are representations of what matters in a work domain. These items are vetted through the applicable community of practitioners for agreement. Informal cognitive artifacts, such as notes made during a task, are also indicative of what is of importance. However, as the notes have no obvious structure and have no predetermined fields or other requirements, they must be properly interpreted to be of value. Using the temporal dimension of the evolution of the notes was shown to be useful to support the expert in understanding the novice’s process in two ways based on the experts’ comments about the study after their session. First, the unfolding of the notes page allowed the expert to be grounded in the context of the novice’s process. The experts made comments such as “I would have pulled them up” when reviewing what documents the novice overlooked. Grounding in context is important during a cognitive task analysis to encourage comments that are true and interesting for the domain being investigated. Being bound in a realistic context allows for natural triggering of domain knowledge and promoting accurate verbalizations. Notes, ‘yellow stickies,’ and the like are commonly copied or taken with permission from a work site as artifacts of the practitioner’s process. While these are useful, without understanding of the process used to derive the notes, the value is lessened. Including the temporal dimension grounds the reviewer of the note artifact in the context of its development and allows a better understanding of the requirements for computer system development to support the domain. Second, revealing the unfolding note sheets with the search results and the documents used, along with the script, supported the experienced analysts’ in focusing attention on what document or information the novice considered important at various points in time during the novice’s

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 52nd ANNUAL MEETING—2008

analysis task. The experts did not have to make assumptions about the novice’s thought process or the source of the annotations during the critiquing process. Rather, the comments and documents were specifically related to annotations on the cognitive artifacts. An example is when the script of the novice’s comments stated, “Now I’m confused. Sounds like Ariane 502 ran into a problem too, but it’s 1997. I think I’m looking for 501.” The expert was able to focus on the documents the novice had pulled up to see why the novice was confused and stated “He’s correct. This one talks about the 2nd.” One early belief for this study was that by presenting the unfolding, packaged representation of the novice's process, the experts would use less time to comment than would have been used by having them directly observe the novice. Surprisingly, the novice took about 1 hour and 20 minutes to actually perform his task and so the two experts who directly observed used the same amount of time. The experienced analysts participating in the critiquing exploration who used the final methodology took from 1 hour and 10 minutes to 1 hour and 21 minutes. The times, then, for the two experts who directly observed performance and the experts who used the final developed methodologies were about the same. The experts who used the final methodology did not have to sit through the less informative times, such as waiting for query results, so the time they spent in their sessions was generally all productive time focused on the critiquing task. Many knowledge tasks, from program management to research, generate informal cognitive artifacts. This playback technique might generalize to other situations where “private” notes are written to support cognitive work, such as handwritten notes for the director of an Emergency Operations Center on a notepad. It might also be interesting to consider how the approach taken here might generalize to the use of a shared cognitive artifact, such as the use of whiteboards by teams in Emergency Departments in hospitals. The whole breadth of intelligence analysis work includes analyzing various types of media, such as physical artifacts and photography as well as text, and reporting in diverse ways from short briefings to long reports. The domain of intelligence analysis can be considered knowledge management work. As with other knowledge management professions, the majority of the work involves visually accessing information from a computer screen, mentally processing the information, and outputting the resulting analysis in an electronic format. While there are some verbalized discussions with colleagues, most of the collaboration and information sharing is done in an electronic environment, such as by email or through document sharing. Therefore, alternatives to naturally occurring verbal behavior in teams need to be developed to provide structure to depicting the flow of a process over time. To understand the process that an intelligence analyst or other knowledge worker uses, documents and metadata about the documents (such as title and author) which are used during analysis can be captured electronically and examined. This work is not subject to U.S. copyright restrictions.

527

However, this examination would not necessarily tell the whole story of the practitioner’s process which is required to develop useful computer support. Think-aloud protocol can also be used, captured and transcribed for richer understanding but there might still be gaps between what the practitioner says during his process and what information the practitioner ultimately uses to make a decision or other cognitive act. Note artifacts, with an explicit temporal dimension, are an important component in revealing the practitioner’s cognitive process as fully as possible. Using the note artifacts in this way can help in understanding the requirements for developing effective computer support for tasks done in complex socio-technical and other system environments. DISCLAIMER Partial funding was provided by the Department of Defense (BAA-001-04). The views expressed in this paper are those of the authors and do not reflect the official policy or position of the Department of Defense, the United States Air Force, the Department of Veteran’s Affairs, or the U. S. Government. REFERENCES Elm, W., Potter, S., Tittle, J., Woods, D.D., Grossman, J., & Patterson, E.S. (2005). Finding decision support requirements for effective intelligence analysis tools. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. [CD-ROM] Grossman, J., Woods, D.D., & Patterson E.S. (2007). Supporting the cognitive work of information analysis and synthesis: A study of the military intelligence domain. Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting. Baltimore, MD: Human Factors and Ergonomics Society, 348-352. [CD-ROM] Hunt, E. & Joslyn, S. (2000). A functional task analysis of time-pressured decision making. In Schraagen, J.M.C., Chipman, S.F., & Shalin, V.L. (Eds.), Cognitive task analysis (119-132). Mahwah, NJ: Lawrence Erlbaum Associates. Miller, J. E., Patterson, E. S., & Woods, D. D. (2006). Elicitation by critiquing as a cognitive task analysis methodology. Cognition, Technology and Work, 8(2), 90-102 Nemeth, C., O’Connor, M., Klock, P. A., & Cook, R. (2006). Discovering healthcare cognition: The use of cognitive artifacts to reveal cognitive work. Organization Studies, 2 (7), 1011-1035 Norman, Donald A. (1991). Cognitive artifacts. In John M. Carroll, (Ed.), Designing interaction: Psychology at the human-computer interface (17-38). Cambridge, England: Cambridge University Press. Patterson, E.S. (1999). Aiding the intelligence analyst in situations of data overload: A simulation study of computer-supported inferential analysis under data overload. Columbus, OH: The Ohio State University. Schraagen, J. M. & Leijenhorst, H. (2001). Searching for evidence: knowledge and search strategies used by forensic scientists. In E Salas & G Klein (Eds) Linking expertise and naturalistic decision making (263274). Mahwah, NJ: Lawrence Erlbaum Associates. Vicente, K. J. (1999). Cognitive work analysis: Towards safe, productive, and healthy computer-based work. Mahwah, NJ: Lawrence Erlbaum Associates. Weinger, M. B., & Jason Slagle. (2002). Human factors research in anesthesia and patient safety: Techniques to elucidate factors affecting clinical task performance and decision making. Journal of the American Medical Informatics Association, 9(6, Suppl), s58-s63.