Utilizing Telemedicine in the Trauma Intensive Care Unit: Does It ...

Utilizing Telemedicine in the Trauma Intensive Care Unit: Does It Impact Teamwork?

Elizabeth H. Lazzara, PhD,1 Lauren E. Benishek, PhD,2,3 Brady Patzer, BS,4 Megan E. Gregory, MS,3 Ashley M. Hughes, MS,3 Kyle Heyne, MS,3 Eduardo Salas, PhD,5 Fernanda Kuchkarian, MPH,6 Antonio Marttos, MD,6 and Carl Schulman, MD, MSPH, PhD6 1

Department of Human Factors, Embry Riddle Aeronautical University, Datona Beach, Florida. 2 Department of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Armstrong Institute for Patient Safety and Quality, Baltimore, Maryland. 3 Department of Psychology and Institute for Simulation and Training, University of Central Florida, Orlando, Florida. 4 Department of Psychology, Wichita State University, Wichita, Kansas. 5 Department of Psychology, Rice University, Houston, Texas. 6 William Lehman Injury Research Center, DeWitt Daughtry Family Department of Surgery, Division of Trauma and Surgical Care, University of Miami Miller School of Medicine, Miami, Florida.

ysis of variance (ANOVA) to compare trust at Times 1, 2, and 3, there was no significant effect on trust: F(2, 14) = 1.20, p = 0.16. When a one-tailed, one-way repeated-measures ANOVA to compare transactive memory systems (TMS) at Times 1, 2, and 3 was conducted, there was no significant effect on TMS: F(2, 15) = 1.33, p = 0.15. We conducted a onetailed, one-way repeated-measures ANOVA to compare team psychological safety at Times 1, 2, and 3, and there was no significant effect on team psychological safety: F(2,15) = 1.53, p = 0.12. There was a significant difference in communication between rounds with and without telemedicine [t(25) = - 1.76, p < 0.05], such that there was more taskbased communication during telerounds. Telemedicine increased task-based communication and did not negatively impact team trust, psychological safety, or TMS during rounds. Conclusions: Telemedicine may offer advantages for some teamwork competencies without sacrificing the efficacy of others and may be adopted by intact rounding teams without hindering teamwork.

All opinions expressed in this presentation are those of the authors and do not necessarily reflect the official opinion or position of the University of Central Florida, the University of Miami, Ryder Trauma Center, or the Department of Defense.

Key words: emergency medicine/teletrauma, telemedicine, telehealth, telecommunications

Abstract

Introduction

Background: The aim of this study was to examine the impact of a telemedical robot on trauma intensive care unit (TICU) clinician teamwork (i.e., team attitudes, behaviors, and cognitions) during patient rounds. Materials and Methods: Thirty-two healthcare providers who conduct rounds volunteered to take surveys assessing teamwork attitudes and cognitions at three time periods: (1) the onset of the study, (2) the end of the 30-day control period, and (3) the end of the 30-day experimental period, which immediately followed the control period. Rounds were recorded throughout the 30-day control period and 30-day experimental period to observe provider behaviors. For the initial 30 days, there was no access to telemedicine. For the final 30 days, the rounding healthcare providers had access to the RP-7 robot (Intouch Health Inc., Santa Barbara, CA), a telemedical tool that can facilitate patient rounds conducted away from bedside. Results: Using a one-tailed, one-way repeated-measures anal-

A

670 TELEMEDICINE and e-HEALTH A U G U S T 2 0 1 5

s technology and clinical knowledge expand, care processes also evolve. For example, healthcare providers have begun to conduct rounds (i.e., a formal, daily meeting comprising healthcare providers who discuss the status of the patient and evaluate the treatment plan) remotely (i.e., not in immediate proximity of the patient). Remote rounds are often an appealing option when considering that bedside rounds can interfere with visitor time and potentially increase infection, and during rounds it can be difficult to hear discussions and see patients when the size of the rounding team becomes so large. Although remote rounds have advantages when compared with bedside rounds (e.g., X-rays can be projected on large monitors for many to see), they have the disadvantage of not affording auditory or visual access to the patient or current vitals. Telemedicine may be one mechanism that can assist providers by enabling access to additional information (e.g.,

DOI: 10.1089/tmj.2014.0074

IMPACT OF TELEMEDICINE ON TEAMWORK

visual and audio feed of patients, bedside nursing, and vital signs) during remote rounds. Research regarding telemedicine within the intensive care unit has demonstrated a reduced length of stay and preventable complications as well as increased adherence to clinical best practices.1 Additionally, meta-analytic evidence suggests that telemedicine is associated with reduced mortality among critically ill patients.2 Although such research indicates the positive association between telemedicine and clinical care, few studies have investigated the impact of telemedicine on teamwork and team climate despite the fact that quality clinical care is traditionally practiced in teams, especially in high-stakes environments. In fact, some even suggest that ‘‘teamwork is the ultimate prerequisite for successful treatment in a high-stakes medical environment.’’3,p.196 Given the importance of teams within the rounding process as well as the proliferation of telemedicine, it is fundamental to understand the impact of such technology on teamwork. Specifically, research is needed to gain a better understanding of the effect of telemedicine on team attitudes (i.e., feelings or beliefs), behaviors (i.e., actions), and cognitions (i.e., knowledge) as well as team climate (i.e., collective norms and values) while conducting rounds. Therefore, the aim of this study was to examine the impact of a telemedical robot on trauma intensive care unit (TICU) team trust (attitude), communication (behavior), transactive memory systems (TMS) (cognition), and psychological safety (climate) during rounds. Each of these constructs (i.e., trust, communication, TMS, and psychological safety) will be discussed later.

Materials and Methods

Fig. 1. Image of the remote rounding room.

video feed of the patient and up-to-date vital signs). Subsequently, the experimental period (i.e., 30 days immediately following the control period) allowed participants to access telemedicine and telemedical information during remote patient rounds. All of the participants provided consent. Participants also completed online surveys at the onset of the study (i.e., Time 1), at the end of the control 30-day period (i.e., Time 2), and at the end of the experimental 30-day period, which immediately followed the control period (i.e., Time 3). Figure 2 gives a visual of the timeline of the surveys. Rounds were recorded via video cameras throughout the 30-day control period and the 30-day experimental period to observe clinician behaviors. Due to technical issues and human error, in total, 27 recorded rounds were used for data analysis. Rounds where healthcare providers did not use the telerounding robot (control group; n = 17 rounds) were compared with rounds that used the telerounding robot (experimental group; n = 10 rounds).

PROCEDURE This study examined a treatment versus control betweengroups factor (i.e., availability and use of telemedicine during remote rounds versus no telemedicine during remote rounds) and a within-groups factor (i.e., time). Both groups ROUNDING TECHNOLOGY (i.e., control and experimental groups) provided patient care In both conditions, patient rounds were conducted in a within the same unit (i.e., TICU) and conducted rounds in a room separate from the patients’ bedsides, and providers had remote room. The remote room is adjacent to the patient access to paper-based patient files, as well as patient X-rays floor and contains a table, desks, a computer with three and laboratory results. Furthermore, the experimental group large monitors on the wall, speakers, a telephone, and video was able to access supplementary information provided by the cameras. The monitors enable everyone in the room to see the same projected information (e.g., patient X-rays, labs, or video feed of the patient floor). Figure 1 gives a pictorial representation of the remote room. During the control period (i.e., 30 days prior to introduction of the robot), providers conducted remote patient rounds without access to telemedicine and telemedical information (i.e., a live audio/ Fig. 2. Study timeline.

ª M A R Y A N N L I E B E R T , I N C . VOL. 21

NO. 8 AUGUST 2015

TELEMEDICINE and e-HEALTH 671

LAZZARA ET AL.

robot’s screen. Meanwhile, the robot projects a live visual feed directly from the unit to a large monitor in the remote room. Moreover, a phone on the robot facilitates real-time, verbal, and confidential communication between individuals on the unit and those in the remote room. The RP-7 robot is equipped with speakers so that individuals in the remote room can verbally communicate with staff and/or patients in real time. The remote room is outfitted with speakers, so all personnel within the room are capable of hearing real-time, auditory information from the unit when desired.

Fig. 3. Picture of the RP-7 robot from Intouch Health Inc.

telemedical rounding robot (the RP-7 robot by Intouch Health Inc., Santa Barbara, CA) (Fig. 3). The RP-7 robot is a wireless, mobile device approximately 5½ feet tall with a flat panel screen mounted at the top operated remotely with speeds up to 5 mph by the rounding healthcare providers with a proprietary joystick and maneuvered throughout the unit as needed via high-speed Internet connection. The robot’s base rolls on three balls, and infrared sensors at the robot’s midpoint detect obstacles, so the operator may adjust accordingly, enabling easy and safe maneuvering in any direction. It has a battery life of approximately 2 h before it must be plugged into a wall outlet to recharge. Individuals in the unit (e.g., bedside nursing) are able to see the healthcare provider’s (i.e., operator) face on the

METRICS To understand the impact of using telemedicine on teambased attitudes, behaviors, and cognitions while conducting rounds within the TICU, several metrics were collected throughout the study period. All self-report measures were administered and completed on iPad (Apple, Cupertino, CA) devices using Qualtrics, a survey software tool (Qualtrics, Provo, UT) that enables online data collection. The surveys examined participant’s trust (i.e., willingness to be vulnerable based on the positive expectations of others’ intentions and behaviors4), transactive memory (i.e., shared understanding about who knows what information5), and psychological safety (i.e., a shared sense it is acceptable to take interpersonal risks6). Each of these metrics will be discussed in greater detail within their respective sections below. Furthermore, team rounding behaviors (i.e., communication—the information verbally exchanged between healthcare providers) were examined by observer ratings of performance on recorded video of rounds. Raters were trained and took a systematic approach to measuring participation and specific communication patterns

Table 1. Summary of Study Variables CONSTRUCT

DEFINITION

METRIC TYPE (SOURCE)

RESPONSE OPTIONS

Trust

A psychological state comprising the intention to accept vulnerability based on positive expectations of the intentions or behavior of another

Survey (Wildman et al.7)

1 (not at all) to 7 (very much so)

Communication

The amount of information exchanged between a sender and a receiver

Video recordings

Total number of different meaningful task-based utterances

Transactive Memory Systems

Shared knowledge about who knows what

Survey (Lewis8)

1 (strongly disagree) to 7 (strongly agree)

Team Psychological Safety

A shared sense among the team that it is safe to take interpersonal risks

Survey (Edmonson6)

1 (never) to 7 (always)

ITEMS 8

NA

15

7

SAMPLE ITEMS ‘‘To what extent do you feel assured that the other department will make intelligent decisions?’’ NA

‘‘Different team members are responsible for expertise in different areas.’’ ‘‘Working with members of this team, my unique skills and talents are valued and utilized.’’

NA, not applicable.


ª MARY ANN LIEBERT, INC.


Table 2. Trust Items

Table 3. Transactive Memory Systems Items

1. Assured that the other healthcare providers will make intelligent decisions?

Specialization

2. Confident that the other healthcare providers will try to do things that benefit the team?

1. Each healthcare provider has specialized knowledge of some aspect of our project.

3. Faith that the other healthcare providers can do the task at hand?

2. I have knowledge about an aspect of the project that no other healthcare provider has.

4. Convinced that you can rely on the other healthcare providers to try their hardest? 5. Confident in the other healthcare providers’ ability to complete a task?

3. Different healthcare providers are responsible for expertise in different areas.

6. Confident that the other healthcare providers will do as they say?

4. The specialized knowledge of several different healthcare providers was needed to complete the project deliverables.

7. Positive that the other healthcare providers will try and do what is best for the team?

5. I know which healthcare providers have expertise in specific areas.

8. Certain that the other healthcare providers will perform well?

Credibility 1. I was comfortable accepting procedural suggestions from other healthcare providers.

during rounds. That is, all raters coded communication in the same manner to ensure standardization. Table 1 gives a summary of each of the metrics. TRUST Trust was evaluated using a survey developed by Wildman et al.7 containing eight items. Responses for this scale ranged from 1 (‘‘not at all’’) to 7 (‘‘very much so’’). Items from this measure include statements such as ‘‘[I am] assured the other healthcare providers will make intelligent decisions.’’ Table 2 gives a complete list of all of the items.

2. I trusted that other healthcare providers’ knowledge about the project was credible. 3. I was confident relying on the information that other healthcare providers brought to the discussion. 4. When other healthcare providers gave information, I wanted to double-check it for myself. (R) 5. I did not have much faith in other healthcare providers’ ‘‘expertise.’’ (R) Coordination 1. Our team worked together in a well-coordinated fashion. 2. Our team had very few misunderstandings about what to do. 3. Our team needed to backtrack and start over a lot. (R)

COMMUNICATION Communication was measured via video observations and a communication coding scheme. Communication was unitized (i.e., broken down) based on the smallest meaningful statement, and all statements were classified as either task-based (i.e., communication that pertained to the task at hand) or non–task-based (i.e., communication not related to the task). An example of task-based communication was ‘‘He’s tachycardic,’’ and an example of non–task-based communication was ‘‘I’ve heard of iHeartRadio.’’ To avoid artificially inflating the frequency, we included the total amount of novel taskbased statements in analyses. TMS TMS were measured using a questionnaire developed by Lewis.8 Transactive memory measures the participants’ selfreport ratings for their knowledge of other healthcare providers’ specialization (degree of expertise), credibility (reliability of others’ knowledge), and perceptions of team coordination (orchestrated knowledge processing). This survey contained 15 questions that included items such as ‘‘I know which healthcare providers have expertise in specific areas’’ and ‘‘I have knowl-

4. We accomplished the task smoothly and efficiently. 5. There was much confusion about how we would accomplish the task. (R) R indicates the item was reverse-scored.

edge about an aspect of the project that no other healthcare provider has.’’ The response options for each item ranged from 1 (‘‘strongly disagree’’) to 7 (‘‘strongly agree’’). Table 3 gives a complete list of all of the items. Table 4. Team Psychological Safety 1. If you make a mistake on this team, it is often held against you. (R) 2. Members of this team are able to bring up problems and tough issues. 3. People on this team sometimes reject others for being different. (R) 4. It is safe to take a risk on this team. 5. It is difficult to ask other members of this team for help. (R) 6. Working with members of this team, my unique skills and talents are valued and utilized. R indicates the item was reverse-scored.


NO. 8 AUGUST 2015


LAZZARA ET AL.

PSYCHOLOGICAL SAFETY Team psychological safety was assessed using a scale adapted from Edmondson.6 Responses for these scales were Likert-type and ranged from 1 (‘‘never’’) to 7 (‘‘always’’). Sample items include statements such as ‘‘If you make a mistake on this team, it is often held against you’’ and ‘‘It is safe to take a risk on this team.’’ Table 4 gives a complete list of the items used for analyses.

Results PARTICIPANTS Thirty-two healthcare providers (16 male, 16 female; age range, 24–56 years; mean age, 37.87 years) who interact with patients while working in the TICU at a large urban Level 1 trauma center in a teaching hospital volunteered to participate in this study. Due to natural attrition, 16 healthcare providers completed surveys across three time periods, with 32 completing the initial survey and 26 completing the second survey. Participants consisted of nurses (n = 18), attending physicians (n = 5), residents (n = 4), fellows (n = 4), and one nurse practitioner (n = 1).

used. In addition, there were significant differences in the frequency of explicit mentions of not having sufficient time to discuss patients (e.g., ‘‘We don’t have time to discuss patients anymore’’) between rounds where telemedicine was neither available nor used (mean = 0.24, SD = 0.44) and rounds where telemedicine was available and used (mean = 0.00, SD = 0.00) [t(25) = 2.22, p < 0.05] (mean difference = 0.24, 95%), such that groups mentioned that there was insufficient time less often when telemedicine was available and used. It should be noted that the mean and SD in the latter group is 0 because there were no mentions of running out of time during rounds using telemedicine. TEAMWORK COGNITIONS—TMS Cronbach’s alpha of the TMS scale was 0.87. We conducted a one-tailed, one-way repeated-measures ANOVA to compare TMS at Time 1 (mean = 5.36, SD = 0.91), Time 2 (mean = 5.36, SD = 0.70), and Time 3 (mean = 5.51, SD = 0.78). There was no significant effect on overall TMS over time: F(2, 15) = 1.33, p = 0.15. In other words, the use of telemedicine did not change TMS.

TEAMWORK ATTITUDES—TRUST Cronbach’s alpha (i.e., reliability as captured by the internal consistency) of the trust scale was 0.96, which is above the accepted level of 0.70.9 Using a one-tailed, one-way repeatedmeasures analysis of variance (ANOVA) to compare trust at Time 1 (mean = 5.70, standard deviation [SD] = 1.07), Time 2 (mean = 5.55, SD = 1.51), and Time 3 (mean = 6.01, SD = 0.65), there was no significant effect on trust: F(2, 14) = 1.20, p = 0.16. This means there was no degradation of trust when using the robotic telemedicine technology.

TEAM CLIMATE—TEAM PSYCHOLOGICAL SAFETY After one item that was underperforming was removed, Cronbach’s alpha of the six-item psychological safety measure was 0.86. We conducted a one-tailed, one-way repeatedmeasures ANOVA to compare team psychological safety at Time 1 (mean = 5.14, SD = 1.01), Time 2 (mean = 5.33, SD = 0.97), and Time 3 (mean = 5.63, SD = 0.80). With the six-item scale, there was no significant effect of team psychological safety over time: F(2,15) = 1.53, p = 0.12. Simply stated, psychological safety remained unchanged by the use of telemedicine.

TEAMWORK BEHAVIORS—COMMUNICATION The Pearson correlation between two coders for one of the videos was 100% (correlation = 1.00), which is above the acceptable level.10 To elaborate, two individuals highly agreed in their observations and coding of communication on one of the videos. To test the effect of telemedicine on teamwork behaviors, we conducted a series of independent-samples t tests to compare rounds without telemedicine available and rounds with telemedicine available and used. There was a significant difference in communication between rounds with and without telemedicine [t(25) = - 1.76, p < 0.05], such that there was more task-based communication during rounds when telemedicine was available and used. Moreover, there was a significant difference in how long patients were discussed (in minutes) [t(25) = 5.12, p < 0.01], such that patients were discussed for less time when telemedicine was available and

Discussion


This study investigated the impact of the use of telemedicine on teamwork competencies during remote patient rounds. Because investigating all team competencies is beyond the scope of any one study, this effort focused on a subset of generalizable (i.e., transportable) teamwork competencies. That is, competencies that are requisite for any given team performing any task in any context.11 Specifically, this research examined trust, communication, TMS, and psychological safety because it could be argued that these competencies are fundamental within any team context. Considering telemedicine is being used in a variety of contexts (e.g., radiology, psychology, trauma management, and surgery) including rounding, these targeted generalizable team competencies were a suitable fit. It was expected that the features and capabilities of the telemedical robot would facilitate team trust, communication,



psychological safety, and TMS. Specifically, the real-time visual information provided by the robot may direct healthcare providers’ attention to the status of the patient and serve to prompt task-based discussion. Furthermore, the presence of the robot on the TICU floor signals to other providers when particular patients are being discussed during rounds. The awareness afforded by the robot’s presence should enable nurses to better contribute to the rounding dialogue, either from the bedside via the robot communication capabilities or by physically attending rounds when they know their patient(s) are being discussed. Similarly, the robot’s video and audio technologies may also be used to bring healthcare providers closer such that healthcare providers (i.e., nurses) who are typically unavailable to attend remote rounds may become more active participants in the rounding process. Such active involvement may facilitate trust and psychological safety among all healthcare providers. Finally, access to additional patient information beyond the information available during rounds without telemedicine should increase team TMS by providing the rounding healthcare providers with a heightened understanding of what is occurring real time on the TICU floor. Despite the expected effects, results indicate that the use of a telemedical robot has mixed effects on teamwork. On one hand, healthcare providers were not any more likely to trust one another when the telemedical robot was available and used. Similarly, healthcare providers did not experience increased psychological safety, nor were they more cognizant of where information was located (i.e., transactive memory). On the other hand, the amount of healthcare provider task-based communication during TICU rounds increased with the use of telemedicine. Moreover, the length of time healthcare providers spent talking about patients decreased. This pattern of findings indicates that healthcare providers’ communication becomes more efficient when the RP-7 robot is used during remote rounds. Increased communication efficiency has important implications for patient care. For instance, orders may be placed more quickly when rounds are efficient, delivering critical patient care sooner. In units such as the TICU where patient status can deteriorate rapidly, prompt administration of treatment can be critical to patient outcomes. Consequently, for those programs considering moving patient rounds from hallway/bedside to remote rooms, there should be consideration of including technology that brings the nursing and bedside clinical information to the remote rounding site without interrupting typical bedside care. LIMITATIONS This study has several noteworthy limitations. First, this study was conducted using a small sample size, which reduces

the statistical power of the analyses and limits our ability to detect an effect of telemedicine on our outcomes of interest. Furthermore, the small sample size restricts the generalizability of our findings, which are at risk of being unduly influenced by individual differences among study participants. Second, several analyses were conducted using self-report data. Data collected through self-report is susceptible to several responder biases (e.g., leniency of ratings) that can inflate ratings.12 Meaning, respondents may artificially rate themselves higher or lower on certain dimensions. In this study, these biases may have contributed to the relatively high baselines observed for trust, psychological safety, and TMS. Because higher baselines restrict potential improvement, this study may have failed to detect an effect of telemedicine on team trust, psychological safety, and transactive memory because baseline ratings were inflated by responder bias. Indeed, self-ratings tend to be more lenient than peer or supervisor ratings.13–15 Third, it is possible that the intact nature of the TICU rounding team explains the null findings for the effect of telemedicine on trust, TMS, or psychological safety. To elaborate, the majority of healthcare providers had a prior history of working together, and they were already familiar with one another at the start of the study. It is possible that the addition of the robot to the rounding process did not affect the trust and psychological safety experienced by the team or the team’s TMS. However, it is notable that telemedicine did not hinder any of these competencies, indicating that the addition of telemedicine does not harm teamwork. Thus, telemedicine can be adopted by intact teams without sacrificing their teamwork. Finally, telerounds were considered to be standard practice for the TICU before implementation of the study intervention. As such, participants’ familiarity with telemedicine could have minimized its impact on teamwork during the time in which the study was conducted. FUTURE RESEARCH The nascent nature of empirical evidence regarding the effect of telemedicine on teamwork points to several directions for future research. Of primary interest is the generalizability of our findings to other samples (e.g., larger teams unfamiliar with telemedicine) and to other types of telemedicine. It is likely that teams new to telerounds may experience greater changes in teamwork than were observed in this study. Additionally, the mechanisms through which communication is improved by telemedicine are a ripe avenue for exploration. For example, does task-based communication increase because the rounding team has access to more information that requires additional discussion, or perhaps does communication increase because


NO. 8 AUGUST 2015


LAZZARA ET AL.

the rounding team can visually observe the patient in real time, making salient to them the fact that their decisions impact real, living patients? There are many teamwork dimensions known to influence team performance that were not investigated in this study (e.g., collective efficacy, shared mental models, backup behavior, and leadership). It is necessary for future research to also explore the effect of telemedicine on additional teamwork dimensions. Finally and potentially most importantly, it is pertinent to determine the relationships among telemedicine, teamwork processes (e.g., communication), teamwork outcomes (e.g., productivity and efficiency), and patient outcomes (e.g., hospital-acquired infections). For instance, can the use of telemedicine enhance attendance during rounds, thus eliminating the need to have a large group of healthcare providers rounding at the bedside exposing patients to potentially harmful pathogens affecting the rate of hospital-acquired infections?

Conclusions This study provides initial support for the efficacy of using telemedicine to facilitate teamwork. The adoption of telemedicine by TICU healthcare providers improved the efficiency with which patients were discussed during remote rounds. Increased task-based communication efficiency can potentially improve the quality and promptness of patient care and subsequent patient health outcomes. Although this study did not support the expected positive effects of telemedicine on trust, psychological safety, or team TMS, telemedicine does not appear to harm these teamwork competencies. However, investigation of the relationship between telemedicine and teamwork is still in its infancy, and there is much opportunity for additional research in this area. Future investigations are needed to fully understand the relationship between telemedicine and teamwork.

Acknowledgments This work was supported by funding from the Department of Defense (award number M162298).

Disclosure Statement

3. St. Pierre M, Hofinger G, Simon R, Buerschaper C. The key to success: Teamwork. In: Crisis management in acute care settings. Berlin: Springer, 2011:195–220. 4. Rousseau DM, Sitkin SB, Burt RS, Camerer C. Not so different after all: A crossdiscipline view of trust. Acad Manag Rev 1998;23:393–404. 5. Ren Y, Argote L. Transactive memory systems 1985–2010: An integrative framework of key dimensions, antecedents, and consequences. Acad Manag J 2011;5:189–229. 6. Edmondson A. Psychological safety and learning behavior in work teams. Adm Sci Q 1999;44:350–383. 7. Wildman JL, Lazzara EH, Shuffler ML, et al. The development and validation of interpersonal trust and distrust scales. Orlando, FL: University of Central Florida, 2009. 8. Lewis K. Measuring transactive memory systems in the field: Scale development and validation. J Appl Psychol 2003;88:587–604. 9. Nunnally JC, Bernstein IH. Psychometric theory, 3rd ed. New York: McGraw-Hill, 1994. 10. Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among multiple raters. Gaithersburg, MD: Advanced Analytics, LLC, 2012. 11. Cannon-Bowers JA, Tannenbaum SI, Salas E, Volpe CE. Defining competencies and establishing team training requirements. In: Guzzo R, Salas E, eds. Team effectiveness and decision making in organizations. San Francisco: Jossey-Bass, 1995:333–380. 12. Borman WC. Job behavior, performance, and effectiveness. In: Dunnette MD, Hough LM, eds. Handbook of industrial and organizational psychology, Vol. 2, 2nd ed. Palo Alto, CA: Consulting Psychologists Press, 1991: 271–326. 13. Kirchner WK. Relationships between supervisory and subordinate ratings for technical personnel. J Industr Psychol 1965;3:57–60. 14. Liaw SY, Scherpbier A, Rethans JJ, Klainin-Yobas P. Assessment for simulation learning outcomes: A comparison of knowledge and self-reported confidence with observed clinical performance. Nurse Educ Today 2012;32: e35–e39. 15. Parker JW, Taylor EK, Barrett KS, Martens L. Rating scale content: Relationship between supervisory and self-ratings. Pers Psychol 1959;12:49–63.

Address correspondence to: Elizabeth H. Lazzara, PhD Department of Human Factors Embry Riddle Aeronautical University 600 S. Clyde Morris Blvd. Daytona Beach, FL 32114

No competing financial interests exist. E-mail: [email protected] REFERENCES 1. Lilly CM, Cody S, Zhao H, et al. Reengineering of critical care processes. JAMA 2011;305:2175–2183. 2. Wilcox ME, Adhikari NKJ. The effect of telemedicine in critically ill patients: Systematic review and meta-analysis. Crit Care 2012;16:R127.


Received: April 16, 2014 Revised: October 1, 2014 Accepted: October 2, 2014
