Improving Critical Thinking Using a Web-Based ... - IngentaConnect

RESEARCH ARTICLE

Improving Critical Thinking Using a Web-Based Tutorial Environment Stephen M. Wiesner, PhD, MT(ASCP) J.D. Walker, PhD Craig R. Creeger, BS the intent of enhancing clinical reasoning and critical thinking among undergraduate students in the MLS program that will help them integrate their new knowledge. The immune system is a critical organ system often overlooked in traditional undergraduate anatomy and physiology courses. Yet immunological theory and techniques are ubiquitous in both medical and research laboratories. Thus, the inclusion of comprehensive immunology instruction in MLS programs provides a foundation for understanding many of today’s laboratory tests and techniques, as well as its role in health and disease. The complexity and breadth of the immune system can be overwhelming for undergraduate students as they try to learn the details of immunology, comprehend the intricacies of the system, and understand its application in a clinical setting.1–3 Intelligent, game-informed tutoring systems (ITS) have been developed to aid learning in complex systems.4–6 However, most of these systems are constrained to specific content areas. The fidelity of some of these systems is high, incorporating complex graphic interfaces. The expertise required to achieve this level of fidelity exceeds that of most instructors, making them impractical for broad implementation. Furthermore, high-fidelity systems may tax the cognitive load, especially in the early stages of learning.7 Interest in such systems has increased due to their positive effect on learning at all levels of education from primary school through professional graduate schools, and they may approach the effectiveness of human tutoring in some cases.8 ITS have been evaluated extensively and their effectiveness recently summarized by Steenbergen-Hu and Cooper.9 SaBLE differs from some of these systems in that it is intentionally low fidelity, enhancing accessibility and usability. SaBLE was developed to be domain-independent and as a development engine that enables instructors from all disciplines to utilize an intelligent tutoring system as an instructional resource.

With a broad range of subject matter, students often struggle recognizing relationships between content in different subject areas. A scenario-based learning environment (SaBLE) has been developed to enhancing clinical reasoning and critical thinking among undergraduate students in a medical laboratory science program and help them integrate their new knowledge. SaBLE incorporates aspects of both cognitive theory and instructional design, including reduction of extraneous cognitive load, goal-based learning, feedback timing, and game theory. SaBLE is a website application that runs in most browsers and devices, and is used to develop randomly selected scenarios that challenge user thinking in almost any scenario-based instruction. User progress is recorded to allow comprehensive data analysis of changes in user performance. Participation is incentivized using a point system and digital badges or awards. SaBLE was deployed in one course with a total exposure for the treatment group of approximately 9 weeks. When assessing performance of SaBLE participants, and controlling for grade point average as a possible confounding variable, there was a statistically significant correlation between the number of SaBLE levels completed and performance on selected criticalthinking exam questions addressing unrelated content. J Allied Health 2017; 46(2):111–116.

OUR MEDICAL LABORATORY Science (MLS) program has undergone a prolonged transition to digital course delivery, a dramatic shift from the way the curriculum has been taught in the past. The program covers the major and minor subdisciplines within the MLS profession, including transfusion medicine, hematology, microbiology, and chemistry. Students often struggle to forge relationships between different subject domains. To address this issue, a Scenario-Based Learning Environment (SaBLE, www.sable.umn.edu) was developed with Dr. Wiesner is at the Center for Allied Health Programs, Program in Medical Laboratory Science, Masonic Cancer Center, University of Minnesota, Minneapolis; Dr. Walker is at the Center for Educational Innovation, University of Minnesota, Minneapolis; and Mr. Creeger is at Expert Interactive, Inc., Apple Valley, MN. The authors report no funding or conflicts of interest related to this study.

SaBLE Description RA1662—Received Nov 30, 2015; accepted Mar 21, 2016.

Theory Construct

Address correspondence to: Dr. Stephen M. Wiesner, Center for Allied Health Programs, University of Minnesota, MMC 711, 420 Delaware St. SE, Minneapolis, MN 55455, USA. Tel 612-625-6465, fax 612-625-5901. [email protected].

SaBLE is designed to enhance clinical thinking and integration of subject matter for students with a high level of prior knowledge.10,11 SaBLE seeks to incorporate aspects

© 2017 Association of Schools of Allied Health Professions, Wash., DC.

111

of both cognitive theory and instructional design, including reduction of extraneous cognitive load, goal-based learning, feedback timing, and game theory. SaBLE employs unique elements when compared to traditional e-learning technology. SaBLE is adaptive to specific combinations of user responses. In most e-learning systems, feedback is dependent upon the choice the user makes and evaluation of the response is absolute. Often in medical science, challenges do not have absolute solutions. Rather, there are optimal and suboptimal responses or combinations of responses, and choices that are omitted are as important as those included.12,13 In SaBLE, content is compiled considering both selections and omissions within a combination of responses and is therefore adapted to demonstrated reasoning, without extraneous information, reducing excessive cognitive load.12–17 SaBLE has been designed to accommodate all of these possibilities with a structure that delivers feedback to the user when the user is most likely to recognize and understand the consequences of the choices made.18–21 In addition, the system incorporates pathway randomization that can be dependent upon user choices, so that, even with an optimal response, each user may experience a different outcome. Randomization enhances user engagement and allows for enough varied interaction so that the user can achieve success through multiple attempts without excessive redundancy. System Framework SaBLE is entirely web-based. The system is standardscompliant HTML with a database on a server and is accessible from most mobile devices and through traditional web browsers. User interactions and progress are recorded to allow comprehensive data analysis of changes in user performance from the beginning to the end of instruction. Feedback to the user can be immediate or delayed. Interaction is incentivized using points and digital badges or awards. Through a WYSIWYG interface, four basic pages can be constructed. Media on any page can include static images, local audio and video, and URLs. All images are embedded according to responsive design standards to accommodate both full-sized monitors and small tablets. Simple text/graphics pages with basic text formatting options can be created, though with this and any other pages more complex HTML tags, cascading style sheet (CSS), or JavaScript can be entered. Text/graphic pages serve a dual purpose of presenting feedback that may be delayed after a challenge. These checkpoint pages (CHK) compile stored feedback from previous pages and display the feedback in paragraph form. In fill-in-the-blank (FiB) interactions answers are coded in standard regular expression (RegEx) format for powerful pattern matching. FiB interactions may be encoded to be evaluated or utilized to promote reflec112

tive analysis of a user’s response to encourage deeper learning without evaluation.12,22 These pages can be adapted to present other kinds of challenges such as matching or multiple true/false questions. In multiple-choice and multiple-select interactions, an infinite number of potential distractors can be included. In multiple-choice interactions, a single answer is correct or optimal, and all other distractors are suboptimal or incorrect. In multiple-select interactions, a combination of responses may be correct or optimal. Inclusion of a single suboptimal distractor invalidates the response. Omission of an optimal distractor also invalidates the response. Unique Features Feedback

Although this system has foundations in problembased learning, the system is intentionally constructed to provide guidance to the user through a variety of feedback mechanisms to avoid a detriment to novice learners, even though users of the system should have sufficiently high prior knowledge.10,23 Feedback can be delivered in contexts defined during page construction. Immediate feedback is delivered on the same page immediately upon response to a challenge. Delayed feedback can be delayed until the next CHK page. When a user reaches a CHK page, all feedback stored in the system as delayed will be compiled and delivered in paragraph format. This includes feedback for both optimal and suboptimal selections in prior interactions. Alternatively, feedback can be delayed until the end of the scenario. The final page in a scenario will compile and display any remaining delayed feedback as well as any feedback defined as end of level feedback. This configuration allows the author to determine the complexity and timing of the feedback provided. Rather than direct feedback, feedback may be presented as a secondary challenge intended to expose the reasoning associated with a response. Thus, indirect feedback can be provided to guide the user in developing an alternative rationale based upon the implied reasoning behind specific distractor selections. In the context of SaBLE, feedback may or may not conform to other definitions or criteria found in the literature for delayed feedback.20,24–26 The intent of the system is to provide feedback within a temporally short time frame. With the exception of feedback delivered immediately after a response, the timing of feedback in this system conforms most closely to definitions of delayed feedback used by others.21,24–27 Further, the system is intended to deliver elaborate feedback, rather than simple knowledge of the correct response.25,28,29 Randomization and Next Page Pool

This framework provides for randomized pages. From any page, the next page to be displayed can be ranWIESNER ET AL., Improving Clinical Thinking

TABLE 1. Summary of SaBLE Content Game Level I II III IV

Scenario Title

Summary of Content

Anatomy of the Immune System Bringing Up Baby The Right B Cell for the Job Crossing your T’s

Anatomic location, organization and structure of organs and tissues of the immune system Developmental program of and somatic recombination in B- and T-cells Selection and maturation of tolerant or self-reactive B-cells Selection and maturation of tolerant or self-reactive T-cells

domly chosen from a pool of next pages. This provides a mechanism for randomly branching scenarios. Further, a different pool of next pages can be associated with each distractor on any interaction page. For example, on a single page selection of one distractor will result in the display of a page randomly selected from a pool associated with that distractor, while selection of a second distractor will result in display of a page randomly selected from a pool associated with the selection of that response. This Next Page Pool is a key differentiator between this system and standard e-learning frameworks. Most e-learning courses have a linear navigation. The Next Page Pool, in conjunction with randomization, introduces adaptive and variable design that enables more complex and nuanced experiences for the user. The system can also be configured so that previously viewed pages within the Next Page Pool are removed, which effectively eliminates duplicate views of a page. This prevents excessive redundancy within a particular scenario, even when the user is forced into repetition by suboptimal performance. For multiple-select interactions, the combination of next pages can vary depending on the distractors selected by the user. For each distractor selected, its associated next page is pushed onto the Next Page queue. When the user advances from the interaction, Next Page logic will first look in the queue for any pending pages, and if there are some, it will remove the first one in the queue and display that page. When it is time to determine the next page (from a Next Page Pool on the currently displayed page), that page will be added to the end of the queue if there is at least one page in the queue. This process continues until the queue is empty. If the queue is empty, then the next page displayed is determined from the currently displayed page’s pool. This feature ensures that each component or missing component within a combination of responses is addressed. Thus, feedback or secondary challenges specific to each component of the combination is presented, rather than elaborate feedback that may include information that does not reflect the reasoning demonstrated in the selected combination.12–17 The concept of a Next Page Queue is a novel feature that other systems do not have. Points, Scoring, and Data Reporting

As a game-informed system, SaBLE incorporates the accumulation of points during participation. Points Journal of Allied Health, Summer 2017, Vol 46, No 2

serve multiple purposes. Awards or badges are achieved by users based upon the points accumulated within a scenario. In addition, progression to other scenarios can be controlled by the accumulation of points. SaBLE tracks and records all user interactions. The system assigns a unique path number to each attempt for a scenario. Data include the user name, path identification number, pages viewed by the user during that attempt, points earned, and the time spent viewing each page. The system also maintains a record of the specific choices made by the user, including text input from FiB interactions. While SaBLE seeks to implement components from multiple theories of learning into a single instructional design, the intent of the system is not improvement in learning outcomes with respect to content knowledge. Rather, by incorporating these design components, improvement in the users’ ability to utilize prior knowledge and synthesize optimal outcomes is the goal using metacognitive feedback derived from specific responses throughout each scenario.30,31 In the current study, we wished to investigate whether repetitive exposure to challenging scenarios would force the user to improve integration of prior knowledge while detecting errors in critical thinking followed by effective corrective action given appropriate feedback.30,32,33

Methods In accordance with IRB regulations, students gave their consent to participate in this study. Ninety-seven students were enrolled in an immunobiology course, 46 of whom self-identified as non-native speakers of English; 81% of students consented, for a total of 76 participants. Participants were randomly assigned to either the SaBLE (treatment) or control group. Thirty-eight students who participated self-reported as non-native speakers of English. They were evenly divided between groups. Students in the SaBLE group were provided encrypted key codes to access each scenario (Table 1) and were awarded partial course credit for each scenario successfully completed to incentivize participation. Total exposure to SaBLE for the treatment group was approximately 9 weeks. Students in the control group were provided traditional study questions that reflected content presented in each scenario (Table 1) of SaBLE. Students were awarded partial course credit for each study question 113

TABLE 2. Examples of Study and Exam Questions Example Study Questions • Compare and contrast signals required for antigen-specific stimulation of CD8 T-cells. • List and define the mechanisms by which diversity is added to lymphocyte receptors during somatic recombination. • Describe the developmental program of a B-cell, including clonal deletion and changes in phenotypic markers. Example Exam Questions • Neisseria meningitidis and Haemophilus influenzae are encapsulated organisms. Explain why these particular organisms are resistant to complement lysis and why deficiency in C4 leads to susceptibility to these particular organisms. • You need to use flow cytometry to identify a population of otherwise normal, mature B-cells in peripheral blood that do not express CD19. Many cells in the peripheral blood lack CD19 expression. Your flow cytometer can only detect phycoerythrin (PE) and fluorescein (FITC). You can use as many antibodies as are needed. Which fluorochromes would you conjugate to what antibodies? Identify only those antibodies specific for B-cells.

that was completed to incentivize participation. Eleven study questions were provided throughout the semester. The primary outcome measure in this study was student performance on preselected, graded and ungraded exam questions requiring critical thinking and higher level processing. Students were evaluated on preselected exam questions with content unrelated to that presented in SaBLE (Table 1) or study questions (Table 2). Student foundational content knowledge supporting evaluated exam questions was also assessed within each exam. Total points earned in the class were examined as a dependent variable. The total number of points earned in the class was also used as an outcome variable.

Results The group number was relatively low (38 participants/ group). To ensure external validity, appropriate tests were conducted to determine whether the students who consented to participate were similar to students who did not consent. The two groups did differ significantly in terms of their academic level: 80% of the consenting group were seniors, while only 29.4% of the non-consenting group were seniors (p