ARTICLE IN PRESS Journal of Surgical Research -, 1–6 (2010) doi:10.1016/j.jss.2010.04.052
Description of Web-Enhanced Virtual Character Simulation System to Standardize Patient Hand-Offs Lori Filichia, M.D.,1 Shivashankar Halan, B.Tech., Ethan Blackwelder, M.S., Brent Rossen, M.S., Benjamin Lok, Ph.D., James Korndorffer, M.D., and Juan Cendan, M.D. Department of Surgery, Health Science Center, University of Florida, Gainesville, Florida Submitted for publication January 13, 2010
A catalog of hand-off modules could be easily developed. Wide-scale web-based deployment was uncomplicated. Identification of the critical findings appropriately translated to user concern for the patient though our series was too small to reach significance. Performance metrics based on the identification of critical discoveries could be used to assess readiness of the user to carry off a successful hand-off. Ó 2010 Elsevier Inc. All rights reserved. Key Words: resident hand-off; resident sign-out; simulation; resident education.
Introduction. The 80-h work week has increased discontinuity of patient care resulting in reports of increased medication errors and preventable adverse events. Graduate medical programs are addressing these shortcomings in a number of ways. Methods. We have developed a computer simulation platform called the Virtual People Factory (VPF), which allows us to capture and simulate the dialogue between a real user and a virtual character. We have converted the system to reflect a physician in the process of ‘‘checking-out’’ a patient to a covering physician. The responses are tracked and matched to educator-defined information termed ‘‘discoveries.’’ Our proof of concept represented a typical postoperative patient with tachycardia. The system is web enabled. Results. So far, 26 resident users at two institutions have completed the module. The critical discovery of tachycardia was identified by 62% of users. Residents spend 85% of the time asking intraoperative, postoperative, and past medical history questions. The system improves over time such that there is a near-doubling of questions that yield appropriate answers between users 13 and 22. Users who identified the virtual patient’s underlying tachycardia expressed more concern and were more likely to order further testing for the patient in a post-module questionnaire (P [ 0.13 and 0.08, respectively, NS). Conclusions. The VPF system can capture unique details about the hand-off interchange. The system improves with sequential users such that better matching of questions and answers occurs within the initial 25 users allowing rapid development of new modules.
INTRODUCTION
Since the ACGME introduced the 80-h work week in July 2003 residency programs rely on a model where multiple teams of residents are responsible for a given patient’s care. ‘‘Night-float’’ residents or teams take care of patients overnight and on weekends, increasing the need for patient ‘‘hand-offs’’ to transfer the care of patients from one resident team to another [1]. Individual residents and groups of residents often transition patient care between one another using a process commonly referred to as a ‘‘sign-out’’ or a patient ‘‘handoff’’ [2]. A hand-off is the transfer of information and professional responsibility from one provider to another while patient is in hospital. This type of transfer of patient care information has been most commonly performed and studied in the nursing profession, however, more recently, attention is being shifted toward hospitalist and resident physician hand-offs [2–4]. With the shift away from the traditional surgical residency structure of in-house call every second to every third night, which provided excellent continuity of care, to the paradigm of multiple teams of residents taking care of a single
1 To whom correspondence and reprint requests should be addressed at Department of Surgery, Health Science Center, University of Florida, P.O. Box 100-286, Gainesville, FL 32610-0826. E-mail:
[email protected].
1
0022-4804/$36.00 Ó 2010 Elsevier Inc. All rights reserved.
ARTICLE IN PRESS 2
JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 2010
patient, the discontinuity of patient care is greater than ever before. To this end the Joint Commission has developed expectations that hospital systems would ‘‘implement standardized approaches to ‘hand-off’ communications’’ in their National Patient Safety Goals, 2009 [5]. Suboptimal communication between physicians during sign-out has the potential to increase the incidence of preventable adverse events [6]. A case-control study performed at Brigham and Women’s Hospital found that preventable adverse events were strongly associated with coverage by a physician from a team other than the patient’s primary team, which nearly doubled when the cross-covering physician was an intern [7]. A literature review on the subject of physician patient handoffs reveals that most reports were performed at single institutions and there were few controlled interventions. These papers most often utilized subjective data such as observations and/or participant surveys [2–4]. At the conclusion of these reports, there are recommendations for improvements in the process of patient hand-off; however, little has been done to systematize an educational curriculum or series of metrics in order to prepare residents for this responsibility [3, 5]. Many programs are currently addressing the issue of patient hand-offs, and this was one of the primary focus points at the 2009 AAMC Annual Congress. METHODS Our team has developed a web-based computer application called the Virtual People Factory (VPF) for creating modeled conversations between real and virtual humans. This system allows us to capture a dialogue between a real user and a virtual character; the system then learns from each user interaction thereby improving the virtual character’s ‘‘intelligence.’’ We have developed a hand-off case scenario between two resident physicians in the process of ‘‘checking-out’’ a patient. For proof of concept, we developed a scenario where a patient underwent a laparoscopic adjustable gastric banding earlier in the day and is having a postoperative course remarkable only for tachycardia. The user interacts with the system in their own language and without any prompts by typing in a box, similar to instant messaging, the system responds by matching stimuli provided by the user to stimuli from a previous user, and a response entered by the script author. Statistical analysis of the likelihood that participants were concerned or ordered further testing for the patient was conducted using a goodness of fit c2 evaluation using the expectation that all users that identified tachycardia would be concerned and that further testing would be warranted for the patient. The statistical package http:// www.people.ku.edu/wpreacher/chisq/chisq.htm was used for the calculation; P value was set at 0.05.
Scenario Development The educator begins to create a script by choosing broad based topics and ‘‘discoveries,’’ which are important concepts that should be gleaned from the conversation. Ideally, the discoveries would be evidence-based in nature. Next, the scenario is created by adding several human conversational stimuli, ‘‘triggers,’’ and a number of virtual human responses, ‘‘speeches.’’ These are the educator’s anticipation as
to the most frequently asked questions and statements that real users would provide in an actual conversation [8]. Next, the educator has a number of users test the scenario, who interact with the system and provide feedback, suggestions, and mark any incorrect responses by the virtual character. The system also records which stimuli failed to match any responses. Generally, it can be noted that as the number of interactions progresses the number of unanswered questions decline for the average user. Accordingly, the number of suggestions that a user leaves also correlates with interaction duration and questioning style of the user. As the system and other users provide feedback, the author of the module reviews and validates the suggested stimuli and edits the responses appropriately. This helps expand the scenario and adds a wider variety of speeches and triggers that the average user would provide. The educator can spend as much or as little time as needed to edit the script. For example, some users may leave 10 suggestions, others may leave as little as one or none. For the script described in this paper, the educator typically logged on to edit the script once every several days and would spend between 5 min to half an hour at a time editing and improving the script. The author then continues to test the scenario with other users until the level of error in the virtual human’s responses is low; practically, this is when most posed questions from the user are met with a valid answer from the virtual physician. The end result is a virtual human that appears to converse about a given subject in a natural way.
Specific Module The module user first encounters the following description: ‘‘There has just been a shift change and you are an incoming physician. Dr. Johnson is turning his patient over to your care. You have 3 min to complete the process. Simply type in the box what you would normally ask.’’ The following prompt then appears: ‘‘The patient is Mr. Harris, a 32 y old with a body mass index of 42 who underwent laparoscopic place of a ‘‘Realize’’ adjustable gastric band today. He was the first case of the day.’’ After that, the user engages the system with the goal of eliciting the necessary information to take care of the patient overnight (Fig. 1). The system tracks the information exchange and an educator defines critical information that must be gleaned; these are termed ‘‘discoveries.’’ The resident receives feedback via a bar that is located in the top right hand corner of the screen, which reveals the number of discoveries as the user obtains them. When all of the discoveries are made, for example, the bar is completely colored in and 6/6 is shown. This information can then be used as both a feedback and educational tool for the participating resident. Individuals can interact with the system multiple times and their individual performance can be tracked in order to assess for improvements over time. At the end of the encounter, several follow-up questions are provided to assess the user’s understanding of the encounter and include questions assessing the user’s level of concern for the patient and further actions to be taken.
RESULTS
Phase 1 included the development of a scenario of a resident checking out a post-operative patient who had a laparoscopic gastric band placed the same morning. Tachycardia was chosen as a frequent but concerning presentation. We began the initial script development and testing using surgical residents at the University of Florida. When the script was adequate, it was distributed to a group of surgical residents at Tulane University for feedback. We then analyzed the transcripts and tabulated the time users spent on
ARTICLE IN PRESS FILICHIA ET AL.: WEB-ENHANCED VIRTUAL CHARACTER SIMULATION SYSTEM
3
FIG. 1. Hand off screen shot. Screen shot of the hand-off interaction between the VPF system and the user.
the task, as well as the number of discoveries each user made. There was a post-interaction questionnaire regarding the level of concern for the patient and any further interventions planned. So far, 26 residents have logged into the system (Fig. 2) with 18 residents interacting for more than 2 min on the exchange. The median time spent on the system for users spending more than 2 min was 6:06 (range 2:21–19:24). Residents spend 85% of the time asking intraoperative, postoperative, and past medical history questions, with minimal time dedicated to other critical questions (Fig. 3). The amount of time spent interacting with the system correlated linearly with the number of discoveries (Fig. 4). Accordingly, it can be seen that some users attain a higher number of discoveries in a much shorter time frame than others. The critical discovery of tachycardia was identified by 62% of users (Table 1). All users who identified tachycardia expressed more concern for the patient compared with 25% of those that did not identify tachycardia (P ¼ 0.13). Sixty percent of users that identified tachycardia ordered either testing or a transfer to higher care status compared with 25% of those that did not identify the problem (P ¼ 0.08) (Table 2). The script was tested with different groups of users and then improved upon by updating new speeches and trigger questions, as well as incorporating user suggestions and identifying missed or incorrect responses recognized by previous users. We have found
that it takes about 20 users to reach a plateau where the level of error in the virtual human’s responses are considered low enough. Practically, this is when almost all of the questions from the user are met with a valid response from the virtual physician. The system improves over time such that there is a near-doubling of effective questions between users 13 and 22. After three rounds of user testing, there was an increase in ‘‘character intelligence’’ as the number of triggers increased by 120 (449 to 569) and the number of speeches increased by 29. The number of discoveries increased as the system learned from prior users: The median number of discoveries identified increased from 4 (range 2–11) in group 1 to 7.5 (range 4–11) in group 2. The system offers the ability to track a number of issues of potential educational research interest. In particular, the system allows the educator to link the question asked with a particular sub grouping (e.g., questions about prior surgeries are marked under a heading of past medical history) allowing for the graphical analysis of the topic flow during a discussion (see dot-graph, need to remove names). Alternatively, the rate of critical discoveries can be used in a manner that informs the educational curriculum. The graphic (bar graph of discoveries) reveals that although most users asked about tachycardia; however, few asked about the patient’s use of insulin—this could be seen as a systematic concern and addressed appropriately.
ARTICLE IN PRESS 4
JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 2010
FIG. 2. Topic flow diagram. Diagram shows each of the 26 users who engaged with the system, the time each user spent, and the number and category of topics covered in the user’s interaction.
FIG. 3. Time spent on each topic. Pie chart shows each of the topics and the percentage of time users spent asking questions falling into each category. Residents spent the most time asking questions about the postoperative course and the least time on preoperative issues.
ARTICLE IN PRESS FILICHIA ET AL.: WEB-ENHANCED VIRTUAL CHARACTER SIMULATION SYSTEM
5
FIG. 4. Discoveries versus time graph. Graph shows the number of discoveries versus time. Users who spent more time interacting with the system achieved a higher number of critical discoveries.
TABLE 1 Frequency of Discoveries. Table Lists the Critical Discoveries and Shows the Number of Times Each Discovery Was Made. The Critical Discovery Was Made by 16 Users (62%)
ARTICLE IN PRESS 6
JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 2010
TABLE 2 The Table Represents the Likelihood of Pursuing Additional Testing Following Interaction with the Virtual Patient Hand-Off System. System Users Who Identified that the Patient Was Tachycardic Ordered Testing More Frequently Though Not Statistically So (P [ 0.08) Which of the following actions would you recommend? (Please check all that apply) Answer options None X-rays Lab work Transfer to higher care unit Other (please specify) Answered question Skipped question
Response percent 34.8% 26.1% 26.1% 4.3%
Response count 8 6 6 1 2 23 0
DISCUSSION
As many residency programs have to switch to a ‘‘crosscover’’ system of patient coverage in order to comply with resident work hour restrictions, there has been an increasing focus on improving patient hand-offs between residents. Common conclusions that are drawn throughout the literature are that in addition to providing written or computerized sign-out information on the patients, verbal communication between practitioners, preferably a face-to-face, interactive hand-off process is essential [3, 9, 10]. Ideally, residents will engage in an active dialogue regarding essential patient care activities, and includes the opportunity to ask and respond to questions [3, 10]. Most current systems of hand-offs focus on the physician who is handing off the patient, rather than the physician who will be taking over the patient’s care [2]. Recognition that this should be an interactive process between both physicians and that the receiving physician has a responsibility to extract critical information is essential. This is a proof of concept study developed to assess the feasibility of developing a scenario, testing it, and then distributing it to a number of users. With the VPF, we can capture unique details about the handoff interchange. The system can learn and approach a peak within 25 users, allowing rapid wide-scale web-based deployment. Web-based deployment was used without difficulty in the trial. The development phase of the system is near completion and efforts are now being turned to developing a catalogue of handoffs scenarios. These can be unique to each specialty rotation, and a catalog of the most common postoperative situations for a given specialty could be easily developed. The system allows the educator to develop these scenarios without the need for a computer scientist and has the potential to be deployed to multiple
institutions via the internet. There is no need for additional interactive tools or interfaces beyond the typing and point-click technologies. Residents who identified tachycardia were more apt to develop concern for the patient and intervene. The study group was small and the statistical analysis of these subgroups was, therefore, insufficient. As we proceed with this research we will investigate the ability to reach statistical conclusions about patient care concerns using this educational methodology. The program could be used as a training and evaluation tool for all levels of residents. It could be used to help beginning residents develop the ability to be able to elicit the appropriate information from the primary physician about various patient situations when assuming the care of a patient in a cross-covering role. Performance metrics based on the identification of critical discoveries could be used to assess readiness of the user to carry off a successful hand-off. The ultimate goal is to provide a reproducible method that can track performance metrics and can be used to help standardize a mechanism for patient hand-offs between residents, to highlight specific educator-driven teaching points and to develop educational level-specific training materials that can be distributed widely over the web without difficulty. SUPPLEMENTARY DATA
Supplementary data associated with the article can be found in the online version, at doi:10.1016/j.jss.2010.04.052. REFERENCES 1. Okie S. An elusive balance: Residents’work hours and the continuity of care. N EnglJ Med 2007;356:2665. 2. Solet DJ, Norvell JM, Rutan GH, et al. Lost in translation: Challenges and opportunities in physician to physician communication during patient handoffs. Acad Med 2005;80:1094. 3. Vidyarthi AR, Arora V, Schnipper JL, et al. Managing discontinuity in academic medical centers: Strategies for a safe and effective resident sign-out. J Hosp Med 2006;1:257. 4. Sharit J, McCane L, Thevenin D, et al. Examining links between sign-out reporting during shift changeovers and patient management risks. Risk Anal 2008;28:969. 5. National Patient Safety Goals. Available at: www.JCAHO.com. Accessed December 2009. 6. Arora V, Johnson J, Lovinger D, et al. Communication failures in patient sign-out and suggestions for improvement: a critical analysis. Qual Saf Health Care 2005;14:401. 7. Petersen LA, Brennan TA, O’Neil AC, et al. Does house staff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med 1994;121:866. 8. Rossen B, Lind DS, Lok B. Human-centered distributed conversational modeling: Efficient modeling of robust virtual human conversations. Proceedings of the 9th International Conference on Intelligent Virtual Agents–2009. Amsterdam, The Netherlands, Sept. 14–16, 2009. 9. Kemp CD, Bath JM, Berger J, et al. The top 10 list for a safe and effective sign-out. Arch Surg 2008;143:1008. 10. Van Eaton EG, Horvath KD, Lober WB, et al. Organizing the transfer of patient care information: The development of a computerized resident sign out system. Surgery 2004;136:5.
