Training for Collaborative Problem Solving: Improving ...

Training for Collaborative Problem Solving: Improving Team Process and Performance through Metacognitive Prompting Travis J. Wiltshire1, Kelly Rosch1, Logan Fiorella2 & Stephen M. Fiore1 1 University of Central Florida, Orlando, FL, 2University of California Santa Barbara, Santa Barbara, CA Increasing complexity in socio-technological systems of domains such as aviation, aerospace, and the military gives rise to equally complex problems. Solving these complex problems requires the collaborative efforts of teams who are able to not just integrate their collective knowledge, but also to monitor and regulate their collective problem solving performance. Unfortunately, current training practices have not yet been developed to promote the metacognitive processes necessary for teams to successfully solve problems in these complex domains. In this paper, we outline a theoretical framework based on the systematic use of metacognitive prompting to improve collaborative problem solving. Our goal is to explicate a theoretically and empirically grounded instructional strategy with testable propositions in support of the development and empirical evaluation of training for complex problem solving.

INTRODUCTION Increasing complexity in socio-technological systems of domains such as aviation, aerospace, industrial process control, and the military gives rise to equally complex problems (Letsky, Warner, Fiore, & Smith, 2008). We consider complex problems to be those where the solution requires the integration of knowledge across a large number of interconnected factors distributed across sociotechnological systems (e.g., Fischer, Greiff, & Funke, 2012; Quesada, Kintsch, & Gomez, 2005). But, more than just knowledge is required. Solving these complex problems requires the collaborative efforts of teams who are able monitor and regulate their collective problem solving performance as they work to integrate complementary perspectives. However, current training practices have not yet been developed to enhance these kinds of collaborative processes. At most, existing theory for training complex problem solving has focused on comprehension processes (Rosen, Fiore, McDaniel, & Salas, 2010). Without effective training to help teams monitor their collaborative processes, they are likely to fail when complex problems arise. In this paper, we outline a theoretical framework based on the systematic use of metacognitive prompting to improve team process and performance. In doing so, we include theoretically and empirically grounded instructional strategies with testable propositions to support the development and evaluation of training to guide future research for collaborative problem solving in complex domains.

COLLABORATIVE PROBLEM SOLVING Collaborative problem solving (CPS) involves the coordination of actions among individuals as they adapt their existing knowledge or generate new knowledge to solve novel and complex problems (Fiore, Smith-Jentsch, Salas, Warner & Letsky, 2010b). Collaborative problem solving can be described by two parallel and interdependent processes: knowledge building and knowledge transformation. Knowledge building involves individuals processing raw data into knowledge by viewing it in the context of the problem and integrating it with prior knowledge. Knowledge transformation is an emergent property that involves the dissemination and externalization of internalized knowledge by teammates, making the knowledge actionable through team interaction (Fiore et al., 2010b). With regard to collaborative problem solving processes, Fiore and colleagues provided a theoretical foundation for understanding the particular problem solving processes (Fiore et al., 2010a) and the ways they may be observed and measured (Fiore et al., 2010b). These are outlined in the Macrocognition in Teams model. The Macrocognition in Teams model (Fiore et al., 2010a; 2010b) consists of five major components that characterize the collaborative problem solving process: individual and team knowledge building, internalized and externalized knowledge, and team problem solving outcomes. Individual knowledge building occurs when an individual processes data and incorporates it into his or her knowledge base. This process may involve reading task-relevant information or interacting with task-relevant technology. Team knowledge building involves the transformation and dissemination of

individual knowledge into actionable team knowledge. Internalized team knowledge describes the knowledge each member holds individually, while externalized team knowledge describes relationships constructed from knowledge and the task-relevant concepts the team has established (or not challenged). Team problem solving outcomes are influenced by interactions among team members and whether these interactions contribute to fulfillment of critical task requirements (Fiore et al., 2010a). Teams with effective collaborative problem solving strategies engage in parallel and iterative processes in service of constructing knowledge, understanding the problem, and evaluating possible solutions. Empirical research utilizing the Macrocognition in Teams model has begun to test components of this model. For example, Rosen (2010) used team communications data during a simulated collaborative problem-solving task to identify team knowledge building processes. Generally, the results showed that processes associated with team knowledge building were related to team problem solving outcomes. Further support was found for the utility of the Macrocognition in Teams model from the communication logs of experienced teams performing tasks in domains such as North American Aerospace Defense command, Air Operations Center, and unmanned aerial vehicle planning (Hutchins & Kendall, 2010). Additionally, recent research found support for team knowledge building processes during a collaborative problem solving activity where teams were required to analyze and write a report regarding a fictitious information systems company (Seeber, Maier, & Weber, 2013). In particular, Seeber et al. (2013) developed the Collaborative Process Analysis (CoPrA) tool to capture fine-grained temporal aspects of CPS processes derived from the Macrocognition in Teams model. Last, evidence was found for many of the CPS processes predicted by the Macrocognition in Teams model when examining retrospective accounts of a complex problem faced by diverse experts in NASA’s Mission Control Center (Fiore, Wiltshire, Oglesby, O’Keefe, & Salas, 2014). In short, while there is growing evidence for CPS processes predicted by Fiore and colleagues (2010a; 2010b), there has been little work describing how to train for collaborative problem solving. Therefore, we outline a training approach in the next section.

METACOGNITIVE PROMPTING AND THE TRAINING CYCLE Metacognition is the awareness of one’s own cognitive processes and the ability to consciously monitor and control these processes (Fiore & Vogel-Walcutt, 2010; see Schraw, 1998). Metacognitive ability has been shown to be beneficial in communication (see Flavell, 1979), complex problem solving, and the development of expertise during task training (Fiore, Cuevas, Scielzo & Salas, 2002). More specifically, in computer-based training environments, metacognitive prompting (such as generic question stems or diagrams with contentfree prompts) has been shown to facilitate learning and memory during complex tasks (Cuevas & Fiore, 2014; Cuevas, Fiore & Oser, 2002). Further, evidence of metacognitive prompting, as an instruc-

tional strategy, was found to be used by expert air traffic control instructors (Wiltshire, Neville, Lauth, Rinkinen, & Ramirez, 2014). Indeed, recent meta-analyses have shown that metacognitive processes contribute to better learning outcomes in both classroom and workplace environments (Dignath & Bütner, 2008; Sitzmann & Ely, 2011). Importantly, metacognitive processes play a role in different phases of learning and performance. Fiore and Vogel-Walcutt (2010) provided a framework for differentiating metacognition across what was referred to as the “training cycle” (which includes preparation for, execution of, and reflection upon, learning). Specifically, this involves activities before (pre-process), during (in-process), and after (post-process), a learning episode. Studies have recently begun to validate the differential impact of these kinds of metacognitive manipulations. For example, in-process metacognitive prompting (manipulations designed to affect the interaction with the training system during training, as opposed to before training or after training) has been shown to increase participants’ ability to apply new knowledge to solve novel problems while decreasing their cognitive load compared to groups that did not receive metacognitive prompting (Fiorella & Vogel-Walcutt, 2011; Vogel-Walcutt, Fiore, Bowers & Nicholson, 2009; Vogel-Walcutt, Fiorella, & Malone, 2013). As noted, metacognitive prompting has also been suggested as beneficial throughout the entire training cycle; that is, preparation, execution, and reflection (see Fiore et al., 2003; Fiore, Johnston, & Van Duyne, 2004; Smith-Jentsch, Zeisig, Acton, & McPherson, 1998). Preparation involves pre-task behaviors, such as planning, which create initial expectations; execution involves activities that improve performance during the execution of the task; and reflection involves feedback and post-task analysis of performance, which can aid in recognizing errors. By conceptualizing this three-phase cycle, training designers are better able to devise instructional strategies appropriate at different phases; that is, given before the task, during the task, and after the completion of the task. Note also that this expands the traditional brief-debrief model, where instruction is only given pre- and post-task (see Fiore, Hoffman, & Salas, 2008). This brief overview of metacognitive prompting was provided to ground our adaptation of this approach for complex problem solving. While metacognitive prompting has been studied in depth in the context of knowledge acquisition (e.g., Fiorella & Mayer, 2012; Fiorella, Vogel-Walcutt, & Fiore, 2012), it has not explicitly been proposed as a strategy to improve team processes. We propose that when systematically incorporated into the three phases of the training cycle, metacognitive prompting will improve team performance when facing complex problems. •

Proposition I. Metacognitive prompting that leverages team planning, knowledge building, and reflexivity training techniques, when integrated systematically across the three phases of the training cycle, will improve the collaborative problem solving processes of teams. With the above as an overarching proposition, the following subsections aim to outline a plan for training CPS using metacognitive prompting based on the integration of a number of pre-existing and empirically studied training interventions.

Preparation Preparation phase activities are pre-task behaviors, such as planning, where initial expectations and contexts are created. Metacognition during this phase involves planning and training activities during which an individual or a team prepares for a learning episode. Here, metacognitive prompting takes the form of preparatory questions that require team members to self-assess and envision key elements of the upcoming learning session such that anticipation of the upcoming operational context is elicited (e.g., Klein, 2008). It is towards this

end that we draw from research on team planning to specify preparation phase metacognitive prompts. Team-planning based prompts. Team planning is a team cognitive process (Cooke, Gorman, Myers, & Duran, 2013) that can improve team performance when conducted before or during a given mission. Team planning typically includes setting goals, clarification of team member roles, prioritization of tasks, assessment of what types of information all team members require (as well as what types of information specific members require), and also, how team members intend to back each other up in the event of errors (Hackman, Brousseau, & Weiss, 1976; Stout, Cannon-Bowers, Salas, & Milanovich, 1999). Theorizing in this area has also differentiated team planning into three sub-dimensions (Marks, Mathieu, & Zacarro, 2001). Deliberate planning can be defined as development and transmission of the primary course of action for the team prior to engaging in the task. Contingency planning, also occurring prior to engaging in a task, can be defined as the development and transmission of back-up or alternative plans in the event that anticipated issues arise that detract from the primary plan. Lastly, reactive strategy adjustment is the modification of the team’s current plan as a function of unanticipated occurrences during the actual performance (Marks et al., 2001). Recently, DeChurch and Haas (2008) sought to investigate the relationship between these three types of team planning processes and team performance in the context of a team scavenger hunt task. Results showed that reactive strategy adjustment planning had the strongest relationship with team performance and that individuals required to plan (in general) prior to performance also had better coordination and task outcomes. Contingency planning also had a strong effect on performance albeit less so than reactive strategy adjustment. Overall, the three types of planning were all related to improved team coordination, but contingency and reactive strategy planning enabled teams to better adapt to change. Therefore we assert the following: •

Proposition II. Metacognitive prompting that leverages team planning behaviors during the preparation phase of the training cycle will increase planning-related collaborative problem solving processes of teams, and in turn, improve team performance during the execution phase. Prompts that are based on team planning interventions could take the form of written prompts given to team members prior to their task performance episode. These prompts would target the specific components of team planning (e.g., setting goals, role clarification, etc.) and sub-dimensions (e.g., contingency planning), as discussed above. A time limit should be pre-specified, but this may vary as a function of the complexity of the task. A set of notional metacognitive prompts based on team planning is shown in Table 1.

Execution Execution phase activities encompass strategies used to bring attention to monitoring both learning and performance during task execution. Metacognition during this phase involves monitoring and regulating cognitive processing as it occurs during task performance. Here, metacognitive prompting takes the form of questions that engage the team in self-assessments that help the team determine if they clearly understand their approach to a given task, the extent to which they are meeting the goals they set during preparation activities, identifying any needs to modify their strategies (cf. Cuevas & Fiore, 2014; Fiorella et al., 2012), and engaging in team knowledge building and sharing behaviors. Team knowledge building prompts. Problem solving teams are often composed of individuals with distinct sets of knowledge and expertise, which require integration in order to effectively perform (Fiore, 2008). This can be problematic as a number of barriers such as differential mental models of the task and a tendency for team

members to discuss commonly held information, as opposed to unique information, are pervasive. Renstch, Delise, Salas, and Letsky (2010) conducted one of the only studies to explicitly focus on team training for knowledge building as a means to mitigate this problem. Knowledge building training consists primarily of a schema-enriched communication (SEC) component. For SEC, team members were trained to engage in communicative processes that elicit the structure and organization of their knowledge, as well as the assumptions, meaning, rationale, and interpretations associated with each member’s knowledge. Using a sample of undergraduate students, they tested the effectiveness of the knowledge building training on a 3-person team task designed by Navy SEALs. The results showed that the knowledge building training led to improved knowledge transfer (i.e., the exchange of knowledge from one team member to another), knowledge interoperability (i.e., knowledge that multiple team members are able to recall and use), cognitive congruence (i.e., an alignment or matching of team member cognitions), and higher overall team performance on the task (Rentsch et al., 2010). In light of this research, we assert the following: •

Proposition III. Metacognitive prompting that leverages team knowledge building interventions during the execution phase of the training cycle, will increase the team knowledge building and externalized team knowledge collaborative problem solving processes and in turn, improve team problem solving outcomes.

Prompts that are based on team knowledge building training could take the form of prompts that that elicit telling and asking behaviors between teammates that facilitate the exchange and comprehension of task-relevant knowledge. Because these prompts would be delivered during task execution, they should be as unobtrusive as possible. It is likely they could be delivered in written or auditory form through whatever interface team members need for their task. A notional set of metacognitive prompts based on team knowledge building training is shown in Table 1.

Reflection Reflection phase activities involve a post-task review of performance (e.g., debriefing) where some form of feedback is utilized for an after-action review. Metacognition during this phase involves contemplation and evaluation of the prior task and learning episode. Here, metacognitive prompting could take the form of questions that engage the team in reflection of errors that occurred during the learning or performance session (Cuevas et al., 2004; Hoffman & Spatariu, 2008). More specifically, trainees could be prompted to reflect on what they did wrong, the reasons why they chose those actions, and how the performance can be improved (cf. Smith-Jentsch, Cannon-Bowers, Tannenbaum, & Salas, 2008); strategies which are akin to team reflexivity and knowledge sharing interventions (see Gabelica et al. 2012; Gabelica, Van Den Bossche, Segers, & Gijselaers, 2014). Team reflexivity and knowledge-sharing based prompts. Team reflexivity training is an intervention that guides teams in reflecting upon their objectives, strategies, and processes and in doing so, encourages them to adapt to current and potential changes in the environment and ultimately ensures similarity in team-member task representations (Gurtner, Tschan, Semmer, & Nägele, 2007). Typically, reflexivity interventions include the following after a team performance episode: 1) reviewing the task performance of the team, 2) elaborating on potential improvements in the processes and methods used to complete the task, and 3) creating suggestions for future work such that the next time the task is done the team’s processes and outcomes are improved. Results of an experiment conducted by van Ginkel, Tindale, and van Knippenberg (2009) showed that team reflexivity training increases information elaboration and performance on a collaborative decision making task.

Similarly, Sikorski, Johnson, and Ruscher (2012) conducted an experiment to investigate the effects of a team knowledge sharing (TKS) training intervention, an intervention based partially on team reflexivity training, in order to improve team collaboration in a science classroom setting. This intervention is designed to improve the sharing of both team- and task-related knowledge by prompting teams to engage in reflective processes that emphasize certain types of communication and planning to improve future performance. This training was predicted to improve the teams’ shared mental models (SMM) and lead to better performance outcomes. The TKS intervention specifically takes the form of having team members complete a post-task questionnaire centered on the five SMM components identified by Johnson et al. (2007): general task and team knowledge, communication skills, attitude toward teammates and task, team dynamics and interactions, and team resources and working environment. Upon completion of the questionnaire, team members were prompted to collectively discuss areas of similarity and discrepancy in their responses as well as to discuss how to improve these discrepancies and poorly rated dimensions in future performance episodes (Sikorski et al., 2012). Overall, the results showed that the TKS training intervention was able to improve similarity in teams’ SMMs as well as lead to improvements in performance when compared to a control condition. Therefore, we assert the following: •

Proposition IV. Metacognitive prompting that leverages team reflexivity and team knowledge sharing interventions during the reflection phase of the training cycle will increase the collaborative problem solving processes of teams and in turn, improve team process in subsequent performance episodes and team problem solving outcomes. Given the above, prompts that are based on team reflexivity and knowledge sharing interventions could take the form of a questionnaire, spoken questions, and facilitated discussion between team members post task-performance. Typically, these reflection phase activities would occur immediately after performance. Further, if the training facilitator requires time to assess the task performance, he or she has several minutes to do so while team members complete their questionnaires. A notional set of metacognition inducing strategies based on team reflexivity and knowledge sharing interventions are shown in Table 1.

DISCUSSION Complex problems are inherently challenging in that solving them requires not just the identification and integration of relevant knowledge, but also the collaborative ability of teams to plan, monitor, and reflect on their performance as it evolves. Current training methods have not yet been developed for promoting the metacognitive processes necessary for CPS. Towards this end, we have presented a theoretical framework, based on empirical research in a variety of domains, detailing how to incorporate metacognitive prompting across the three phases of the training cycle. We proposed that promoting team planning behaviors during the preparation phase of the training cycle will improve team performance during the execution phase; that team knowledge building interventions during the execution phase will improve team problem solving outcomes; and that metacognitive prompting that promotes team reflexivity and team knowledge sharing interventions during the reflection phase will improve team process in subsequent performance episodes and team problem solving outcomes. Our goal was to provide a set of training propositions detailed enough for researchers to empirically test in complex problem solving environments. Indeed, our propositions are readily reframed as experimental hypotheses suitable for testing in a variety of contexts. Furthermore, while we have proposed a systematic implementation across the three phases of the training cycle,

Table 1. Notional Metacognitive Prompts or Metacognition Inducing Strategies for Improving CPS Processes across the Training Cycle Phases. Notional Metacognitive Prompts or Metacognition Inducing Strategies

References

Preparation

Team planning-based prompting • What are your goals as a team for the upcoming performance episode? • What is your plan for accomplishing these goals? • What roles will each of you play? • What types of tasks are of priority and which ones are of less importance to your goals? • During the mission, what types of information do all team members require access to and which types are only required by specific team members? • What are the likely issues that can arise with the current plan? • How do you intend to back each other up in the event of errors or issues with the plan?

DeChurch & Haas (2008) Hackman et al. (1976) Marks et al. (2001) Stout et al. (1999)

Execution

Team knowledge building-based prompting • What do you know about the task that your teammates must know? Tell them what. • What might your teammate know that you need to know? Ask them what. • Why is certain knowledge important for your teammate to know? Tell them why. • Why is certain knowledge important for your teammate to tell you? Tell them why. • How is your knowledge connected to prior knowledge you have shared and towards some aspect of your teammate’s task? Tell them connections. • How is knowledge your team member just shared connected to prior shared knowledge and/or some aspect of your task? Ask them connections.

Rentsch et al. (2010)

Reflection

Team knowledge sharing-based prompting • Deliver Johnson et al.’s (2007) SMM questionnaire to each team member. This will prompt each team member to individually reflect on: General task and team knowledge, communication skills, attitude towards teammates and task, team dynamics and interactions, and team resources and working environment • Have team members compare their responses and ask them the following questions: o What discrepancies are there in your ratings and why do you think these exist? o How can you improve upon discrepantly rated areas? o How can poorly rated areas be improved?

Sikorski et al. (2012)

Team reflexivity-based prompting • What type of errors were made during your previous performance and why? • How did you ask for information as a team and could this be improved upon? • How did you pass information as a team and could this be improved upon? • How was the team organized compared to the way you had planned? • Are there alternatives to your chosen planned and executed behaviors, and if so, what are they? • Are there any other improvements you can you make to improve your performance?

researchers could also examine the relative effects of prompting at each phase with regard to team problem solving outcomes. Such research, for example, could answer the question of whether metacognitive prompting at just a certain phase is more beneficial than another type and further, how this compares to the use of all three types. An additional challenge for studying problem solving in this context is the complexity of the test-bed or synthetic task environment to be developed. Thus far only a limited number of lab studies have produced this kind of testing environment (Cooke & Shope, 2005; Duran, Goolsbee, Cooke, & Gorman, 2009; Fischer, McDonnell, & Orasanu, 2007; McComb, Kennedy, Perryman, Warner, & Letsky, 2010; Rosen, 2010; Salas, Elliott, Schflett, & Coovert, 2004). This remains a significant hurdle in the progress of understanding and training complex and collaborative problem solving. Following this line of thinking, one area of study could be the development of a testbed flexible enough for utilization in a variety of problem solving contexts, but still modifiable to suit the study of different collaborative problem solving processes (cf. Letsky et al., 2008). Further, additional CPS methods and appropriate CPS measurement techniques are needed. The body of work cited here examining CPS has primarily relied on the laborious process of using human raters to identify CPS processes in transcriptions of team communications (Fiore et al., 2014; Hutchins & Kendall, 2010; Rosen, 2010). While the CoPrA tool, developed by Seeber et al. (2013), does simplify this process, and create useful visualizations, efforts are needed to develop measures that can assess aspects of CPS in real time (cf. Gorman, Hessler, Amazeen, Cooke, & Shope, 2012). Further, the types of analyses for CPS communications can be improved by drawing from related work that quantifying the non-linear dynamics (e.g., Russell, Funke, Knott, & Strang, 2012; Louwerse, Dale, Bard, & Jeuniaux, 2012; Strang et al., 2012). Such analyses can provide an

Gurtner et al. (2007)

index of the coupling, complexity, and temporal regularity of communication structures and help to parse those CPS communication patterns that will lead to effective problem solving. With regard to metacognitive prompting, more generally, additional research questions can be pursued. For example, could the technique be used too much, in which there are no beneficial effects? Perhaps the quantity of prompts proposed here is too large, such that their use would greatly increase cognitive load (cf. Cuevas & Fiore, 2014). Also, should the prompts be used as only a training tool, or, if they can be designed to be minimally disruptive and effective, can they be used during real-world tasks? These questions illustrate the richness of this area of research and to help guide the training community in further exploration of this important area of inquiry. In conclusion, the study of complex collaborative problem solving is still nascent. While the social and cognitive sciences has investigated group problem solving in a variety of more simple settings (e.g., Deek, & McHugh, 2003; Laughlin, 2011; Theiner, Allen, & Goldstone, 2010), exploration of complex collaborative problem solving (Letsky et al., 2008), is still evolving. This represents a critical gap because the need for teams to collaboratively solve complex problems is going to become increasingly prevalent as the sociotechnical systems that comprise modern work advance. Therefore, we have provided an outline for next steps in furthering this line of inquiry. It is our hope that the strategies proposed herein will be empirically tested and support the idea that metacognitive prompting, when integrated systematically across the phases of the training cycle, will improve collaborative problem solving process and performance.

ACKNOWLEDGMENTS This work was partially supported by a HFES Training Technical Group student grant awarded to Travis J. Wiltshire and by NASA/JSC contract NNJ13HE70P "Critical Team Cognitive Processes for Long-Duration Exploration Missions," awarded to Stephen M. Fiore.

REFERENCES Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J. L. (2013). Interactive team cognition. Cognitive Science, 37(2), 255-285. Cooke N. J. & Shope, S. M. (2005). Synthetic Task Environments for Teams: CERTT's UAV-STE. In Stanton N., Hedge A., Brookhuis K., Salas E., Hendrick H., Handbook of Human Factors and Ergonomics Methods (pp. 450-457). CRC Press. Cuevas, H. M. & Fiore, S. M. (2014). Enhancing learning outcomes in computer-based training via self-generated elaboration. Instructional Science. 10.1007/s11251-0149315-8 Cuevas, H. M., Fiore, S. M., Bowers, C. A., & Salas, E. (2004). Fostering constructive cognitive and metacognitive activity in computer-based complex task training environments. Computers in Human Behavior, 20(2), 225-241. Cuevas, H. M., Fiore, S. M., & Oser, R. L. (2002). Scaffolding cognitive and metacognitive processes in low verbal ability learners: Use of diagrams in computer-based training environments. Instructional Science, 30(6), 433-464. DeChurch, L. A., & Haas, C. D. (2008). Examining Team Planning Through an Episodic Lens Effects of Deliberate, Contingency, and Reactive Planning on Team Effectiveness. Small Group Research, 39(5), 542-568. Deek, F. P., & McHugh, J. A. (2003). Group Problem-Solving: Tasks, Productivity, Early Experiments. In Computer-Supported Collaboration (pp. 55-66). Springer US. Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231-264. Duran, J. L., Goolsbee, Z., Cooke, N.J., & Gorman, J.C. (2009). Embedded Collaboration Metrics in the MacroCog Synthetic Task Environment. Proceedings of the 53rd Annual Meeting of the Human Factors and Ergonomics Society, (pp. 1388-1392). Fiore, S. M. (2008). Interdisciplinarity as teamwork how the science of teams can inform team science. Small Group Research, 39(3), 251-277. Fiore, S. M., Cuevas, H. M., Scielzo, S., & Salas, E. (2002). Training individuals for distributed teams: Problem solving assessment for distributed mission research. Computers in Human Behavior, 18(6), 729-744. Fiore, S. M., Hoffman, R. R., & Salas, E. (2008). Learning and performance across disciplines: An epilogue for moving multidisciplinary research toward an interdisciplinary science of expertise. Military Psychology, 20, S155-S170. Fiore, S. M., Johnston, J., & Van Duyne, L. (2004). Conceptualizing the Training Space: Constructing hierarchies to integrate time and space for debriefing distributed simulation-based exercises. In Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society (pp. 2562-2566). Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M. & Warner, N. (2010a). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52, 2, 203-224. Fiore, S. M., Smith-Jentsch, K. A., Salas, E., Warner, N., & Letsky, M. (2010b). Toward an understanding of macrocognition in teams: Developing and defining complex collaborative processes and products. Theoretical Issues in Ergonomic Science, 11, 4, 250-271 Fiore, S., Vogel-Walcutt, J. J. (2010). Making metacognition explicit: Developing a theoretical foundation for metacognitive prompting during scenario-based training. Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting, 2233-2237. Fiore, S. M., Wiltshire, T. J., Oglesby, J., O’Keefe, W., & Salas, E. (2014). Complex collaborative problem solving processes in Mission Control. Aviation, Space, and Environmental Medicine, 85(4), 456-461. Fiorella, L., & Mayer, R. E. (2012). Paper-based aids for learning with a computer-based game. Journal of Educational Psychology, 104(4), 1074-1082. Fiorella, L., & Vogel-Walcutt, J. J. (2011). Metacognitive prompting as a generalizable instructional tool in simulation-based training. In Proceedings of the 55th Human Factors and Ergonomics Society Annual Meeting, 55(1), 565-569. Fiorella, L., Vogel-Walcutt, J. J., & Fiore, S. (2012). Differential impact of two types of metacognitive prompting provided during simulation-based training. Computers in Human Behavior, 28(2), 696-702. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive– developmental inquiry. American Psychologist, 34(10), 906. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4(1), 19-42. Fischer, U., McDonnell, L., & Orasanu, J. (2007). Linguistic correlates of team performance: Toward a tool for monitoring team functioning during space missions. Aviation, Space, and Environmental Medicine, 78(Supplement 1), B86-B95. Gabelica, C., Van den Bossche, P., Segers, M., & Gijselaers, W. (2012). Feedback, a powerful lever in teams: A review. Educational Research Review, 7(2), 123-144. Gabelica, C., Van den Bossche, P., Segers, M., & Gijselaers, W. (2014). Dynamics of team reflexivity after feedback. Frontline Learning Research, 2(2), 64-91. Gorman, J. C., Hessler, E. E., Amazeen, P. G., Cooke, N. J., & Shope, S. M. (2012). Dynamical analysis in real time: detecting perturbations to team communication. Ergonomics, 55(8), 825-839. Gurtner, A., Tschan, F., Semmer, N. K., & Nägele, C. (2007). Getting groups to develop good strategies: Effects of reflexivity interventions on team process, team performance, and shared mental models. Organizational Behavior and Human Decision Processes, 102(2), 127-142. Hackman, J. R., Brousseau, K. R., & Weiss, J. A. (1976). The interaction of task design and group performance strategies in determining group effectiveness. Organizational Behavior and Human Performance, 16, 350–365. Hoffman, B., & Spatariu, A. (2008). The influence of self-efficacy and metacognitive prompting on math problem-solving efficiency. Contemporary Educational Psychology, 33(4), 875-893.

Hutchins, S. G., & Kendall, T. (2010). Analysis of team communications to understand cognitive processes used during team collaboration. In Proceedings of the 15th International Command and Control Research and Technology Symposium (ICCRTS ’10), Santa Monica, CA, June 22-24. Johnson, T. E., Lee, Y., Lee, M., O'Connor, D. L., Khalil, M. K., & Huang, X. (2007). Measuring sharedness of team-related knowledge: Design and validation of a shared mental model instrument. Human Resource Development International, 10(4), 437-454. Klein, G. (2008). Performing a project premortem. Engineering Management Review, IEEE, 36(2), 103-104. Laughlin, P. R. (2011). Group Problem Solving. Princeton, NJ: Princeton University Press. Letsky, M. Warner, N., Fiore, S.M., & Smith, C. (Eds.). (2008). Macrocognition in Teams: Theories and Methodologies. London: Ashgate Publishers. Louwerse, M. M., Dale, R., Bard, E. G., & Jeuniaux, P. (2012). Behavior matching in multimodal communication is synchronized. Cognitive Science, 36(8), 1404-1426. Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review,26(3), 356-376. McComb, S., Kennedy, D., Perryman, R., Warner, N., & Letsky, M. (2010). Temporal patterns of mental model convergence: Implications for distributed teams interacting in electronic collaboration spaces. Human Factors, 52(2), 264-281. Quesada, J., Kintsch, W., & Gomez, E. (2005). Complex problem-solving: A field in search of a definition?. Theoretical Issues in Ergonomics Science, 6(1), 5-33. Rentsch, J. R., Delise, L. A., Salas, E., & Letsky, M. P. (2010). Facilitating Knowledge Building in Teams: Can a New Team Training Strategy Help?. Small Group Research, 41(5), 505-523. Rosen, M. A. (2010). Collaborative problem solving: The role of team knowledge building processes and external representations (Doctoral dissertation). University of Central Florida, Orlando, FL. Rosen, M. A., Fiore, S. M., McDaniel, R., Salas, E. (2010). Adapting narrative theory to improve the implementation of story in problem-based learning. In: Andrews DH, Hull TD, DeMeester K. (eds.) Storytelling as an Instructional Method: Research Perspectives. Rotterdam: Sense Publishers(pp. 87-101). Russell, S. M., Funke, G. J., Knott, B. A., & Strang, A. J. (2012). Recurrence quantification analysis used to assess team communication in simulated air battle management. In Proceedings of the 56th Human Factors and Ergonomics Society Annual Meeting (pp. 468-472). Salas, E., Elliott, L. R., Schflett, S. G., & Coovert, M.D. (Eds.) (2004). Scaled Worlds: Development, validation, and applications. Burlington, VT: Ashgate Publishing. Seeber, I., Maier, R., & Weber, B. (2013). Macrocognition in collaboration: Analyzing processes of team knowledge building with CoPrA. Group Decision and Negotiation, 1-28. Sikorski, E. G., Johnson, T. E., & Ruscher, P. H. (2012). Team Knowledge Sharing Intervention Effects on Team Shared Mental Models and Student Performance in an Undergraduate Science Course. Journal of Science Education and Technology, 21(6), 641-651. Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421-442. Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26(1-2), 113-125. Smith-Jentsch, K. A., Cannon-Bowers, J. A., Tannenbaum, S. I., & Salas, E. (2008). Guided team self-correction impacts on team mental models, processes, and effectiveness. Small Group Research, 39(3), 303-327. Smith-Jentsch, K. A., Zeisig, R. L., Acton, B., & McPherson, J. A. (1998). Team dimensional training: A strategy for guided team self-correction. In J. A. Cannon Bowers & E. Salas (Eds), Making decisions under stress: Implications for individual and team training, (pp. 271-297). Washington, DC: American Psychological Association. Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors: The Journal of the Human Factors and Ergonomics Society, 41(1), 61-71. Strang, A. J., Horwood, S., Best, C., Funke, G. J., Knott, B. A., & Russell, S. M. (2012, September). Examining temporal regularity in categorical team communication using sample entropy. In Proceedings of the 56th Annual Meeting of the Human Factors and Ergonomics Society Annual Meeting (pp. 473-477). Theiner, G., Allen, C., & Goldstone, R. L. (2010). Recognizing group cognition. Cognitive Systems Research, 11(4), 378-395. van Ginkel, W., Tindale, R. S., & van Knippenberg, D. (2009). Team reflexivity, development of shared task representations, and the use of distributed information in group decision making. Group Dynamics: Theory, Research, and Practice, 13(4), 265280. Vogel-Walcutt, J. J., Fiore, S., Bowers, C., & Nicholson, D. (2009). Embedding metacognitive prompts during SBT to improve knowledge acquisition. In Proceedings of the 53rd Annual Meeting of the Human Factors and Ergonomics Society, 19391943. Vogel-Walcutt, J. J., Fiorella, L., & Malone, N. (2013). Instructional strategies framework for training systems. Computers in Human Behavior. 29(4), 1490-1498. Wiltshire, T. J., Neville, K., Lauth, M., Rinkinen, C., & Ramirez, L. (2014). Applications of cognitive transformation theory: Examining the role of sensemaking in the instruction of air traffic control students. Journal of Cognitive Engineering and Decision Making. DOI: 10.1177/1555343414532470