synthesize diverse historical information in a well-‐argued, logical and coherent
... Progress is assessed by comparing the critical thinking and writing skills of.
Improving Critical Thinking and Writing Skills through Weekly Discussions and Journals Kimberly Breuer and Staci Swiney
That students can learn and integrate new and complex course content through writing is well-‐ established,1 the question remains: which types of weekly writing assignments develop the critical thinking skills2 necessary for a deeper understanding of the detailed and interconnected materials typically found in freshman level college U.S. history survey courses? Two possible types, journals and online discussions, are frequently used. Journaling allows students to reflect on their course materials in new ways. When coupled with specific content guidelines (such as use of evidence, a requirement to use primary documents in entries, etc.), journals seem a likely vehicle to develop analytical thinking in an historical context. Discussions place emphasis on collaborative learning and are a useful tool to hone critical thinking skills. While online discussions rely upon clarity of thought and expression in writing, the primary benefit is one of student engagement. Exposure to the ideas of others helps students to synthesize complex information and understand different points of view. This study explores the efficacy of using discussions and journals to develop critical thinking skills in average to below average students3 across modes of instruction (face to face, 8 week online, and 15 week online). The best evidence that students successfully mastered these skills is their ability to synthesize diverse historical information in a well-‐argued, logical and coherent comprehensive essay at the end of the course. Progress is assessed by comparing the critical thinking and writing skills of individual students in the first writing assignment to the level of mastery they achieve in writing assignments at the end of the course. PEDAGOGICAL ISSUES AND COURSE DESIGN The fundamental goal of the freshman level US History survey is to teach students the basic facts of the political, social, economic, and cultural history of the United States and to introduce the concept of thinking historically.4 In the state of Texas, the American history survey is a required part of the core 1
See Janet Emig, “Writing as a Mode of Learning,” College Composition and Communication 28, no. 2 (May, 1977): 122-‐128; Judith Langer and Arthur Applebee, How Writing Shapes Thinking: A Study of Teaching and Learning [Urbana, Ill.: National Council of Teachers of English, 1987]; and Gamze Çavdar and Sue Doe, “Learning Through Writing: Teaching Critical Thinking Skills in Writing Assignments,” PS: Political Science & Politics 45, no. 2 (April 2012): 298-‐306, doi:10.1017/S1049096511002137 [accessed July 27, 2013]. 2 For a detailed guide to elements of critical thinking see Richard Paul and Linda Elder, The Miniature Guide to Critical Thinking: Concepts & Tools [Dillon Beach, CA: Foundation for Critical Thinking, 2009]. 3 “Average” is defined as meeting the expectations of freshman level written communication in a history course: logical and well-‐organized expository writing which is grammatically correct and historically relevant. 4 Historical thinking is defined as the set of reasoning skills necessary to ask questions of the past, to analyze and interpret evidence found in the historical record (primary source materials created at the time under study which can be either written or material in nature) in order to create an historical narrative based upon effective and persuasive arguments. See below for a description of the specific historical thinking skills assessed in this study. For a more general discussion of the historical craft see Mark T. Gilderhus, History and Historians: A th Historiographical Introduction, 7 ed. [Upper Saddle River, NJ: Prentice Hall, 2010] and John H. Arnold, History: A Very Short Introduction [New York: Oxford University Press, 2000].
curriculum for all degree programs and one of its key core objectives is to inculcate students with the critical thinking skills that are the hallmark of a college education. While the course designation is lower division, students taking the survey vary in their classification and educational backgrounds. They range from “just out of high school” underclassmen, to juniors or seniors who have completed college composition and rhetoric courses, to older than average adults returning to earn a second degree. Some students come into the class with well-‐developed critical thinking and writing skills while others have difficulty with the basics of essay writing. Few come into the course with an understanding of what it means to think historically. The “gold standard” assessment utilized to gauge historical and critical thinking skills is the basic analytical essay. When I first started teaching the survey, I assessed student learning based upon the results of three or four high stakes exams which included an essay. Unfortunately, students who lacked basic critical thinking and writing skills did poorly on the essay portion of the first test and often found themselves in a “hole” with little chance of making a good semester grade. This, in turn, led to retention issues. Hoping to improve outcomes, I allowed students to write their essay as a take-‐home exercise, but I saw little improvement in learning outcomes for below average students. At this point, I began developing an online version of the American history survey. I retained the testing/essay format but added the use of asynchronous discussion boards as a means to encourage the student engagement and collaborative learning found in the face to face classroom.5 These graded discussion boards were forums where students were free to comment upon the course content and bring up points that interested them. The first run of the online course revealed that face-‐to-‐face (F2F) students had an advantage after the first exam/essay because of an in-‐class review and one-‐on-‐one office time, where I could go over common mistakes, suggest ways to improve, and provide tutorials on writing. In response, I tweaked the online class by changing the discussion prompts/guidelines (to be less freeform and more like an essay prompt) to teach/assess critical thinking and writing skills. Anecdotal evidence suggested that this change in discussion format improved essay submissions. However, there was a detrimental impact to the discussions of the below average and late posting students, many of whom simply parroted the comments of the above average students (who tended to post earlier).6 There was also a decline in the quality of the required reply posts and the conversational element of the discussion was lost. Despite these issues, I wanted to mirror the success of the online asynchronous discussion boards in improving essay writing skills in my F2F class. Unfortunately, this type of assignment does not translate easily to the F2F format. I eventually settled on adding ten small, team-‐based learning sessions, which 5
See Caroline Hodges Persell, "Using Focused Web-‐Based Discussions to Enhance Student Engagement and Deep Understanding." Teaching Sociology 32, no.1 (January 2004): 61-‐78. 6 See Jim Waters, “Thought-‐Leaders in Asynchronous Online Learning Environments,” Journal of Asynchronous Learning Networks 16, no. 1 (January 2012), http://sloanconsortium.org/jaln/v16n1/thought-‐leaders-‐ asynchronous-‐online-‐learning-‐environments [accessed July 27, 2012] for a discussion of peer influence in discussion boards.
resulted in improved abilities to think historically, but did not improve critical thinking/writing in the essay.7 Ultimately, I decided to move away from high stakes testing in my F2F course. I replaced the tests with several short, low-‐stakes quizzes that covered the basic facts and weekly one-‐paragraph journals8 in response to a prompt (similar to an essay prompt). The first two journals focused on teaching students how they could demonstrate their basic critical thinking skills (writing a strong thesis statement, using specific and detailed pieces of evidence in support of the thesis, having good organization and making conclusions) in response to a fairly straightforward question. The instructions and prompts of the next few journals taught students to think historically. As the class progressed, the journal prompts required more in-‐depth and analytically complex responses. The revised F2F course culminated in a comprehensive analytical essay in which the students showcased their critical thinking and writing skills and their abilities to think historically.9 Anecdotal evidence suggested that journaling led to better outcomes for the essay and overall outcomes for all students. The first semester I moved to this format I had a more than a 50% improvement in retention (calculated as the number of drops, Ds and Fs in the course) without changing my grading standards.10 Over the next year, I modified the F2F and online formats of the course to bring them more into alignment. Both courses now utilized low stakes quizzes, exercises in document analysis, and exercises in using secondary readings (the latter assignments were designed to improve skills in critical thinking and thinking historically). The F2F and online sections used the same books, covered the same materials, and shared lectures.11 They each continued with weekly writing assignments in either journals or asynchronous discussions and each class culminated in the comprehensive essay. While student learning outcomes improved in both formats, the face to face class exhibited better analytical skills on the comprehensive essay. The final stage in the evolution of the online course came with the decision to use both weekly journals and discussions as a way of duplicating the success of the F2F format. Discussions returned to their 7
Team-‐based learning exercises included discussions with hot-‐seat reporting, role-‐playing, and team writing. For more information on team-‐based learning see Michael Sweet and Larry K. Michaelsen, eds., Team-‐Based Learning in the Social Sciences and Humanities: Group Work That Works to Generate Critical Thinking and Engagement, [Sterling, Va: Stylus, 2012]. 8 See Susan Leighow Meo, ""In Their Own Eyes": Using Journals with Primary Sources with College Students," The History Teacher 33:3 (May, 2000): 335-‐341 for more information about the use of primary sources in journaling exercises. 9 The current course design allows students to develop mastery by first acquiring skills, then integrating these skills through practice, and finally applying the skills appropriately on their own. See Susan Ambrose et al., How Learning Works: 7 Research-‐Based Principles for Smart Teaching [San Francisco: Josey Bass, 2010], especially Chapter 4, “How Do Students Develop Mastery?”, for an informative investigation into how students achieve mastery of a skill. 10 Moving away from the high stakes exam format allowed students to do poorly on the first quiz or two without a disastrous impact on their final grade. In addition, my teaching assistant and I reached out to poorly performing students and offered individualized help in ways to improve their study habits to improve content mastery. 11 In the F2F course I utilize PowerPoints as the background for my lectures. In the online courses I utilize voice over PowerPoint lectures using the same slides. Any media shown in my F2F course is also linked to in my online course.
original role of engaging students with the materials and fostering collaborative learning. Discussion prompts now encourage students to lead discussions in directions they see fit, based upon the course materials and a general discussion prompt. Journal prompts are meant to elicit critical thinking and students are expected to show evidence of their understanding of the materials by synthesizing diverse information in well-‐substantiated and logical arguments. Since the journal entries are private, students cannot simply parrot the work of others. My teaching assistants and I agreed that this new format improved critical thinking and writing skills in the final comprehensive essay. All evidence that journaling and discussion lead to improvement in critical thinking/writing skills and the ability to think historically remained anecdotal. The purpose of this study is to demonstrate statistically whether or not my anecdotal impression of improvements in critical thinking and writing skills is correct and to establish comparability of outcomes across modes of instruction. OVERVIEW OF STUDY This project compares seven sections of the US history survey. The survey is divided into two separate courses, USI (before 1865) and USII (after 1865). Students typically take USI before USII, but USI is not a pre-‐requisite for USII. The USI and USII courses are designed as companion courses. They utilize common curriculum and materials and students complete the same types of assignments in both courses utilizing the same mode of instruction. The F2F and the online versions share many of the same assignments. The lecture videos in the online courses are identical to lectures given in the F2F sections. Where elements differ between F2F and online, great care has been given to provide comparable assignments/practices reflecting the unique nature of the learning environment utilized. Students in all sections submit their work through Blackboard. The online courses are offered in two formats: an 8 week Academic Partnerships (AP) version12 and a 15 week traditional online, open to the general student body.13 The chart below identifies the sections used in this study.14 Figure 1. Descriptions of the Course Sections COURSE DESCRIPTION LENGTH 12
ASSESSMENTS
COMMENTS
Most of the students in the 8 week course are in the online RN to BSN and BSN degree programs. There are also a handful of dual credit students in each section and the occasional student from another Academic Partnerships program. 13 The 15 week version can be further modified to run in the 11 week summer session. 14 I do not have access to data from earlier versions of the course. The first offerings of the online course were in WebCT and access to these modules ended after our migration to Blackboard. Students in my F2F classes did not submit exam essays through an LMS in the original version of the course. Submission of all assignments into Blackboard in my F2F courses became standard after my institution of weekly journaling.
1
USI F2F
15 WEEKS
2
15 WEEKS
4
USII TRADITIONAL ONLINE USII TRADITIONAL ONLINE USI AP ONLINE
5
USI AP ONLINE
8 WEEKS
6
USII AP ONLINE
8 WEEKS
7
USII AP ONLINE
8 WEEKS
3
15 WEEKS 8 WEEKS
Weekly journal and 10 in-‐ class small group discussions Weekly discussions Weekly discussions and journals Two weekly discussions Weekly discussions and journals Two weekly discussions Weekly discussions and journals
Mostly freshmen with some upperclassmen Mostly upperclassmen also taking F2F classes Mostly upperclassmen also taking F2F classes One of the first classes for AP Nursing students One of the first classes for AP Nursing students Many students have already taken USI with us Many students took USI in the discussion only format
METHODOLOGY Initially, I intended to select 20 students from each course divided into four different cohorts based upon classification (upper-‐ or lowerclassmen) and the analytical and mechanical skills demonstrated on the first writing assignment (average or below average). After analyzing the courses, I determined that separation into cohorts along the lines of classification was not viable for all sections. As students graduate or leave the university, they are automatically removed from the Blackboard grade book. In some sections this limited the number of students available for inclusion in this study. Since there were many low stakes assessments, not all students completed all assignments. I eliminated all students who completed less than 90% of the journals/discussions from consideration since the purpose of this study is to assess the efficacy of these assignments as a learning tool. I also eliminated all students who did not submit the final comprehensive essay.15 I then perused the entries in the first two journals/discussions and sorted the remaining students into two groups –below average and average entries. I then randomly selected 10 students from these two groups for inclusion in the study.16 ASSESSMENT RUBRIC 15
Because of the availability of extra credit and the fact that the final essay was worth less than a full letter grade, some students chose not to submit the essay because they were satisfied with the grade they had already earned for the course. 16 Some sections have less than 20 students selected for analysis. If I did not have 10 students who met all the criteria for selection in one of the initial groupings, I did not make up the difference from the other set of students. This is an issue in the smaller traditional online classes which had more upperclassmen whose work could no longer be accessed through the Blackboard grade book after graduation.
My research assistant and I assessed three discussions and/or three journals and the comprehensive essay for each student in four key categories: mechanics-‐focused writing, basic knowledge of facts, thinking historically, and critical thinking. Each of these categories had 2-‐7 components. We graded the students on a 0-‐3 scale (whole numbers only) based upon their performance in the category. See the chart below for the grading criteria. We then averaged the components of each category to produce the overall category score. We selected an entry (journal and/or discussion) from early in the course as a baseline, the second came from midway in the course, and the third from the end of the course. The goal was to see if there was improvement over time. Figure 2. Assessment Scale Used to Evaluate Student Skill Levels 0 1 2 3
ASSESSMENT SCALE Below Average/Remediation Needed (D to F) Below Average Freshman Level (D+ to C) Average Freshman Level (C+ to B-‐) Above Average Freshman Level (B to A)
In the first category, mechanics/focused writing, we looked at three components: the use of a thesis statement, the overall organization of the assignment, and the grammar. Since the use of a thesis is the key to producing a well-‐written and well-‐argued essay, we weighted this component double in producing the average category scale. The next three categories were based upon Bloom’s levels.17 The second category, basic knowledge of facts, examined the evidence of content mastery (represented by the first two Bloom’s levels of remembering and understanding). We scored the assignment based upon the accuracy of the information presented and the student’s level of understanding of the information, as exhibited by the appropriate use of historical evidence. Figure 3. The Bloom’s Levels18
17
David R. Krathwohl, “A Revision of Bloom's Taxonomy: An Overview,” Theory into Practice, 41, no. 4 [Autumn, 2002]: 212-‐218. 18 Triangle based upon Richard C. Overbaugh and Lynn Schultz, Bloom’s Taxonomy – Old Dominion University, http://ww2.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm [accessed on July 27, 2013]
The third category, thinking historically, is a new skill for many students in the freshmen survey. It is not uncommon to hear students complain about the requirement that they take the U.S. History survey as a degree requirement. They argue that history is merely a dry list of dates, events and important dead people which they learned about in high school. A primary goal of the survey course is to get students to think about history in new ways, beyond mere rote memorization, and to increase student engagement with the subject. To think historically, students need to consider various and complex factors and evidence to answer a deceptively simple looking question: “Why?” We ask students to consider multiple points of view, to analyze primary documents, to identify the most salient arguments, and to consider context and change over time. Ultimately, we want them to apply what they have learned to answer the “why” question in a logical and well-‐reasoned written response. To introduce students to the concept of thinking historically, I assign the short piece “What Does It Mean to Think Historically?” by Thomas Andrews and Flannery Burke.19 The authors provide a concise description of the core elements needed to think historically: understanding change over time and the context of the events, recognizing causality, complexity and contingency, and having the ability to draw meaningful conclusions (in toto these elements comprise the “analyzing” and “applying” Bloom’s levels). We used these same elements, as defined by Andrews and Burke, in our rubric. Our final assessment category appraises critical thinking skills (“evaluating” and “creating” in Bloom’s levels). These represent the highest order of learning. Do students gather and assess relevant information and evidence in their analysis? Do students offer well-‐reasoned conclusions tested against relevant criteria? Are students able to recognize and assess implications and consequences? Do they display abstract thinking in their interpretations? Can they effectively communicate their ideas and is their argument logical and powerful? Mastery of these skills should develop as the students learn to think historically and practice making arguments in their discussions and journals. 19
Thomas Andrews and Flannery Burke, “What Does It Mean to Think Historically?,” AHA Perspectives 45, no. 1 (July 2007): 32-‐35. http://www.historians.org/perspectives/issues/2007/0701/0701tea2.cfm [accessed July 27, 2013].
Figure 4. The Assessment Rubric
ANALYSIS METHODOLOGY Once we had gathered the raw data we calculated category averages for all assignments. We used these averages to track improvement from the beginning to end of the course. Closer analysis of the baseline average scores revealed that we actually had four distinct categories of students, instead of the initial two: remedial (score of less than 1); below average (score of 1-‐1.8); average (score of 1.8-‐2.5); and above average (score of 2.5 and above). We compared this baseline with the final journal/discussion score and calculated an aggregate numeric and percentage change. See Tables 1 and 2 in Appendix A. We repeated this process using the first discussion/journal score as a baseline compared to the final comprehensive essay score in the same category. See Table 3 in Appendix A. In the sections with both journals and discussions, we used the journal score as the baseline since this assignment utilized a prompt similar to the comprehensive essay. In the discussion only sections, the prompts were more similar to the essay and journal prompts than they were to the discussion prompts in the sections utilizing both assignments. Therefore Table 3 reflects similar assignments.
Finally, to analyze the comparability of the sections across all modes of instruction, we calculated the average score for each of the four categories of analysis in addition to an overall average for the comprehensive essay. These are graphically represented in histograms in Appendix B. RESULTS – IMPROVEMENT OVER BASELINE Analysis of the improvement in discussions over time (See Table 1 in Appendix A) reveals that students who fell into the remedial and below average groupings at the beginning of the course improved whether they were assigned discussions alone or discussions and journals.20 Comparing the class average initial score to the final discussion score reveals that all course sections improved from first to last, indicating that both methods worked in teaching underperforming students critical thinking and writing skills and how to think historically. After averaging results based upon assignment types, the data show that, as expected from the anecdotal evidence, course sections using both discussions and journals improved more than those using only discussions across all categories. See Figure 5 below. The least change came in mechanics, which is unsurprising. Our goal was to teach the skill of writing an analytical essay for a history class and we focused on thesis and organization; we did not focus on teaching grammar skills. There was a 27% increase in critical thinking skills in the courses utilizing both journals and discussions. This suggests that the more opportunities students have to organize their thoughts and put forth a logical argument, the better abled they become in critical thinking. Students in the sections using both journals and discussions had twice the practice in learning how to make strong and logical arguments – and twice the grading feedback in ways to improve. The most dramatic improvement occurred in the basic knowledge of the materials (46%) and thinking historically (53%) categories. This cannot be explained by simple repetition. I would argue that being asked to think about course materials in two different ways leads to better learning outcomes. In the journals, students are asked to explain how the pieces all fit together according to their personal understanding of the materials. In the discussions, students not only see others’ viewpoints and arguments, but they also learn collaboratively, perhaps picking up on content they missed in their own studies of the materials. Figure 5. Comparisons of Discussions COMPARISONS OF IMPROVEMENT IN SECTIONS WITH 20
Interestingly, the data also supports the anecdotal evidence that students who do well on the first discussion tend to “slack off” at the end of the course. This corresponds to the general wisdom that students who have already “made their grade” do just enough to maintain that grade in the small stakes assignments at the end of the course.
JOURNALS OVER SECTIONS WITH DISCUSSIONS ALONE Category Percentage Improvement Mechanics 13 Basic Knowledge 46 Thinking Historically 53 Critical Thinking 27 Composite 27 Analysis of the improvement in journals over time (see Table 2 in Appendix A) demonstrates that students who fell into the remedial and below average groupings at the beginning of the course improved with practice as they moved through the course.21 Comparing the F2F class average with the average of the three online sections yielded some interesting results, with the online courses seemingly showing a 70-‐100% improvement over the online course. See Figure 6 below. These results, however, are skewed. The F2F course had the advantage of an in-‐class workshop on how to write a journal entry before the first journal assignment. They worked together in teams to write a practice entry and then the class came back together to create the “A” entry, on the classroom whiteboard, with the help of the instructor. Therefore, the baselines are more representative of students who did need more remediation (as opposed to merely needing to learn how to use existing skills in a new way) than the online class without the workshop. The F2F class was also a predominantly freshmen class of traditional 18-‐19 year old students, many of whom were taking a composition or rhetoric class concurrently. The online cohorts were primarily upperclassmen and older than average students. Figure 6. Comparisons of Journals COMPARISONS OF IMPROVEMENT IN ONLINE SECTIONS OVER THE FACE TO FACE SECTION Category Percentage Improvement Mechanics 100 Basic Knowledge 87 Thinking Historically 70 Critical Thinking 85 Composite 85 Comparisons of the comprehensive essay scores versus the initial journal or discussion scores (see Table 3 in Appendix A) show that underperforming students in the remedial and below average groups improved across all categories. Comparing the F2F written journal only, online discussion only, and online discussion and journal sections, the average improvement from the initial assignment to the final comprehensive essay was dramatically improved in the sections using both discussions and journals 21
As with the discussions, average to above average students declined in the end journals, for likely the same reason as mentioned above.
across all categories. See Figure 7 below. There was a 47% improvement in mechanics for students who utilized discussions and journals over the other sections. This improvement is likely due to fact that they had more practice writing and more grading feedback on how to improve their essay writing skills. Comparing the online sections, those assigned both journals and discussions showed the most improvement in thinking historically, supporting the argument that thinking and writing about course content in two different ways lead to better learning outcomes. Figure 7. Comparisons of Initial Baseline Average versus the Final Comprehensive Essay Average COMPARISONS OF AVERAGE PERCENTAGE IMPROVEMENT ON ESSAY VS. BASELINE ACROSS SECTIONS CATEGORY
F2F
DISCUSSION
DISCUSSION/JOURNALS
DISCUSSION AND JOURNALS OVER DISCUSSIONS
DISCUSSIONS AND JOURNALS OVER F2F
Mechanics Basic Knowledge Thinking Historically Critical Thinking Composite
19.5 28.2
19.8 31.0
37.1 38.1
47 19
47 26
34.4
30.4
43.8
31
21
31.3 28.2
31.9 28.3
37.3 39.0
14 27
16 27
RESULTS – COMPARABILITY ACROSS PLATFORMS Appendix B contains the histogram comparisons for all course sections, showing course averages and the quantity of students in each grouping in each category for the final comprehensive essay. All course sections had average to above average scores in each of the four categories. In the mechanics category, the online sections utilizing journals and discussions ranked higher than their counterparts with discussions only (comparing 8 week USI with discussions to 8 week USI with both discussions and journals, etc.). The F2F class ranked first in all other categories, including the composite average. The online sections all scored above a 2 (which is the expected freshmen level) in these categories, but there was no other discernible pattern. Figure 8. Comparisons of the Final Essay Averages Across Platforms by Ranking of Averaged Scores. (D=DISCUSSION ONLY AND B=BOTH DISCUSSIONS AND JOURNALS)
RANK
MECHANICS
1 2
USI AP ONLINE – B USI F2F
3
USI AP ONLINE -‐ D
4
USII TRADITIONAL ONLINE -‐ B USII AP ONLINE -‐ B
5 6 7
USII AP ONLINE-‐D USII TRADITIONAL ONLINE – D
BASIC KNOWLEDGE
THINKING HISTORICALLY
CRITICAL THINKING
COMPOSITE AVERAGE
USI F2F
USI F2F
USI F2F
USI F2F
USI AP ONLINE – D USII TRADITIONAL ONLINE – D (TIE)
USII TRADITIONAL ONLINE – D USI AP ONLINE -‐ D
USI AP ONLINE -‐ D
USI AP ONLINE -‐ D
USII AP ONLINE -‐ B
USII TRADITIONAL ONLINE – B USII AP ONLINE –B (TIE)
USII TRADITIONAL ONLINE -‐ B USII TRADITIONAL ONLINE – D USII AP ONLINE –B USI AP ONLINE -‐ B
USII TRADITIONAL ONLINE – D USII TRADITIONAL ONLINE – B (TIE)
USII AP ONLINE -‐ D USII AP ONLINE -‐ D
USII AP ONLINE -‐ D
USII TRADITIONAL ONLINE – B USII AP ONLINE –D (TIE) USI AP ONLINE -‐ B
USI AP ONLINE -‐ B
USII AP ONLINE -‐ B USI AP ONLINE -‐ B
CONCLUSIONS The results indicate that the assumption based upon anecdotal evidence that journaling and discussions improved remedial and below average student performance over either discussions or journals alone is accurate. This is especially evident for the online sections of the course. It is also no surprise that courses utilizing both journals and discussions showed greater improvement in the mechanics of writing, since these students wrote more often and received more feedback on how to improve. In online classes, students who utilized both journals and discussions showed greater skill in thinking historically than their counterparts who only participated in discussions. These students had the advantage of looking at the course materials in two different ways, reinforcing content mastery. The results also indicate that there is no significant difference in the student learning outcomes for the various online versions of the course. The 8 week condensed and 15 week long semester formats resulted in average to above average performance on the final essay. Discussions alone and discussions coupled with journals both effectively taught key critical thinking and writing skills. The most interesting comparison is between the F2F class and all of the online courses with the “gold standard” final comprehensive essay. The F2F class had more true “just out of high school” freshmen, yet it outperformed the online classes in every category except mechanics, where it ranked second. This indicates that while the pedagogical design of the online course effectively teaches course content, critical thinking and writing, and the ability to think historically, the students in the F2F course have an advantage. The key difference, I believe, is the physical presence of the instructor and teaching assistants in the classroom and the synchronous nature of the course. The team based learning workshops and in class discussions allow immediate feedback and the ability to effectively reteach “fuzzy” concepts.
The next step taken to more closely align the F2F and online versions of the course should be finding a way to bring the benefits of the F2F workshop into an asynchronous environment. One option is to include a tutorial video diagraming the core components of an effective journal and comparing the results on the final essay between future F2F and online sections. Another possible solution would be to bring elements of Team-‐Based Learning into the asynchronous online environment. A follow-‐up study of the efficacy of these design changes is warranted.
APPENDIX A – TABLES Table 1. Discussion Scores Remedial LT 1.0
Mechanics Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Basic Knowledge Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Thinking Historically Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Thinking Critically Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Composite Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou
Below Average 1.0 -‐ 1.8
Average 1.8 -‐ 2.5
Above Average 2.5 -‐ 3.0
Average for Total Class
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Initial Score
1 1 3 5 1 2 0 3
+ 1.50 + 1.00 + 0.77 + 0.96 + 1.20 + 1.25
9 8 8 25 10 5 14 29
+ 0.46 + 0.60 + 0.48 + 0.51 + 0.46 + 0.44 + 0.49 + 0.47
5 10 8 23 1 11 5 17
+ 0.08 + 0.18 + 0.16 + 0.15 -‐ 0.50 + 0.14 + 0.32 + 0.15
1 0 1 2 0 0 0 0
-‐ 0.30
1.78 1.82 1.67 1.75 1.58 1.79 1.68 1.70
2.14 2.22 2.01 2.12 2.02 2.13 2.13 2.10
+ 0.36 + 0.40 + 0.34 + 0.36 + 0.44 + 0.34 + 0.44 + 0.41
20.0% 22.0% 20.1% 20.7% 28.0% 19.3% 26.3% 23.9%
2 0 3 5 1 9 0 10
+ 1.25
3 9 5 17 3 4 4 11
+ 0.33 + 0.94 + 0.50 + 0.71 + 0.50 + 0.75 + 1.13 + 0.82
7 7 10 24 7 5 14 26
+ 0.14 + 0.43 + 0.05 + 0.19 0.00 + 0.10 + 0.46 + 0.27
4 3 2 9 1 0 1 2
-‐ 0.50 -‐ 0.33 0.00 -‐ 0.33 0.00
2.09 1.92 1.83 1.94 1.88 1.28 2.00 1.70
2.25 2.47 2.15 2.29 2.08 2.14 2.55 2.29
0.16 0.55 0.33 0.35 0.21 0.86 0.55 0.58
7.5% 28.8% 17.8% 18.3% 11.1% 67.4% 27.6% 34.1%
1 1 5 7 2 6 0 8
+ 2.00 + 1.00 + 0.40 + 0.71 + 0.50 + 1.40
+ 0.36 + 0.51 + 0.16 + 0.35 + 0.70 + 0.49 + 0.61 + 0.59
2 6 6 14 4 1 4 9
+ 0.50 + 0.25 + 0.07 + 0.21 -‐ 0.15 0.00 + 0.23 + 0.03
4 1 0 5 0 0 0 0
-‐ 0.20 -‐ 0.40
+ 1.18
9 11 9 29 6 11 15 32
1.83 1.75 1.53 1.69 1.63 1.20 1.69 1.50
2.16 2.16 1.72 2.00 2.02 1.97 2.22 2.08
0.34 0.41 0.19 0.31 0.38 0.77 0.53 0.58
18.5% 23.1% 12.4% 18.2% 23.5% 63.9% 31.5% 38.9%
2 1 5 8 1 5 1 7
+ 1.65 + 1.00 + 0.58 + 0.90 + 1.00 + 1.38 + 1.30 + 1.31
9 11 9 29 4 12 10 26
+ 0.31 + 0.46 + 0.34 + 0.38 + 0.68 + 0.51 + 0.65 + 0.59
2 6 6 14 5 1 8 14
+ 0.15 + 0.28 + 0.25 + 0.25 + 0.02 -‐ 0.30 + 0.06 + 0.02
3 1 0 4 2 0 0 2
-‐ 0.20 -‐ 0.10
1.73 1.79 1.45 1.65 1.81 1.21 1.75 1.57
2.09 2.19 1.82 2.03 2.10 1.92 2.18 2.07
0.36 0.40 0.38 0.38 0.29 0.71 0.44 0.50
21.0% 22.5% 26.0% 23.2% 16.1% 58.3% 25.0% 31.9%
1 0 1 2 1 3 0 4
+ 1.95
9 13 12 34 6 12 15 33
+ 0.28 + 0.53 + 0.38 + 0.41 + 0.56 + 0.57 + 0.55 + 0.56
2 3 6 11 5 3 4 12
+ 0.53 + 0.75 + 0.13 + 0.37 -‐ 0.09 + 0.40 + 0.26 + 0.15
4 3 1 8 0 0 0 0
-‐ 0.17 -‐ 0.27 + 0.23 -‐ 0.16
1.86 1.82 1.62 1.76 1.72 1.37 1.78 1.62
2.16 2.26 1.92 2.11 2.05 2.04 2.27 2.13
0.30 0.44 0.31 0.35 0.33 0.67 0.49 0.52
16.3% 24.2% 18.9% 20.0% 19.2% 48.9% 27.6% 32.0%
+ 1.23
+ 1.17 + 1.20 + 1.00 + 1.33 + 1.30
+ 0.60 + 1.28 + 1.05 + 1.33 + 1.26
-‐ 0.70 -‐ 0.50
-‐ 0.50 -‐ 0.25
-‐ 0.24
-‐ 0.18 -‐ 0.15
-‐ 0.15
Final Score
Improvement
Table 2. Journal Scores Remedial LT 1.0
Mechanics Class 1 Jou Only Class 3 Class 5 Class 7 Dis + Jou Basic Knowledge Class 1 Jou Only Class 3 Class 5 Class 7 Dis + Jou Thinking Historically Class 1 Jou Only Class 3 Class 5 Class 7 Dis + Jou Thinking Critically Class 1 Jou Only Class 3 Class 5 Class 7 Dis + Jou Composite Class 1 Jou Only Class 3 Class 5 Class 7 Dis + Jou
Below Average 1.0 -‐ 1.8
Average 1.8 -‐ 2.5
Above Average 2.5 -‐ 3.0
Average for Total Class
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Qty Students
Change First -‐ Last
Initial Score
2 2 3 2 0 5
+ 0.40 + 0.40 + 0.77 + 0.75
+ 0.30 + 0.30 + 0.30 + 0.15 + 0.08 + 0.14
9 9 4 6 1 11
-‐ 0.20 -‐ 0.20 -‐ 0.45 -‐ 0.30 -‐ 1.80 -‐ 0.49
1 1 0 0 0 0
-‐ 0.50 -‐ 0.50
+ 0.76
5 5 5 10 18 33
1.96 1.96 1.61 1.63 1.59 1.61
1.96 1.96 2.10 2.03 1.94 2.01
0.00 0.00 0.49 0.39 0.34 0.40
0.0% 0.0% 30.6% 24.1% 21.5% 24.7%
1 1 3 5 1 9
+ 1.50 + 1.50 + 1.00 + 0.90 + 2.00 + 1.06
2 2 1 5 9 15
+ 0.50 + 0.50 0.00 + 0.60 + 1.11 + 0.87
10 10 8 8 9 25
+ 0.10 + 0.10 + 0.31 + 0.25 + 0.17 + 0.24
4 4 0 0 0 0
-‐ 0.50 -‐ 0.50
2.09 2.09 1.88 1.53 1.82 1.72
2.18 2.18 2.33 2.06 2.53 2.31
0.09 0.09 0.46 0.53 0.71 0.58
4.2% 4.2% 24.4% 34.5% 39.1% 33.7%
2 2 3 3 0 6
+ 0.80 + 0.80 + 0.87 + 0.47
4 4 6 12 18 36
+ 0.45 + 0.45 + 0.57 + 0.58 + 0.62 + 0.60
10 10 3 3 1 7
+ 0.08 + 0.08 + 0.20 -‐ 0.33 + 0.20 -‐ 0.03
1 1 0 0 0 0
-‐ 1.00 -‐ 1.00
1.85 1.85 1.53 1.52 1.56 1.54
2.04 2.04 2.08 1.93 2.16 2.06
0.19 0.19 0.55 0.41 0.60 0.52
10.2% 10.2% 35.9% 27.0% 38.5% 33.7%
2 2 3 3 0 6
+ 0.60 + 0.60 + 0.87 + 0.50
5 5 6 10 15 31
+ 0.40 + 0.40 + 0.52 + 0.64 + 0.49 + 0.54
8 8 2 5 4 11
-‐ 0.09 -‐ 0.09 + 0.45 -‐ 0.12 + 0.20 + 0.10
2 2 1 0 0 1
-‐ 0.55 -‐ 0.55 + 0.10
1.88 1.88 1.50 1.43 1.67 1.54
1.96 1.96 2.06 1.83 2.10 1.99
0.08 0.08 0.56 0.41 0.43 0.45
4.4% 4.4% 37.2% 28.4% 25.5% 29.3%
1 1 2 2 0 4
+ 1.20 + 1.20 + 1.04 + 0.36
4 4 4 12 14 30
+ 0.25 + 0.25 + 0.49 + 0.63 + 0.56 + 0.58
10 10 6 4 5 15
-‐ 0.01 -‐ 0.01 + 0.35 -‐ 0.12 + 0.42 + 0.25
2 2 0 0 0 0
-‐ 0.31 -‐ 0.31
1.94 1.94 1.63 1.53 1.66 1.60
2.03 2.03 2.14 1.96 2.18 2.09
0.09 0.09 0.51 0.43 0.52 0.49
4.6% 4.6% 31.6% 28.5% 31.3% 30.4%
+ 0.67
+ 0.68
+ 0.70
+ 0.10
Final Score
Improvement
Table 3. Final Essay Scores Remedial LT 1.0
Mechanics Class 1 Jou Only Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Basic Knowledge Class 1 Jou Only Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Thinking Historically Class 1 Jou Only Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Thinking Critically Class 1 Jou Only Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou Composite Class 1 Jou Only Class 2 Class 4 Class 6 Dis Only Class 3 Class 5 Class 7 Dis + Jou
Below Average 1.0 -‐ 1.8
Average 1.8 -‐ 2.5
Above Average 2.5 -‐ 3.0
Average for Total Class
Qty Students
Change First -‐ Essay
Qty Students
Change First -‐ Essay
Qty Students
Change First -‐ Essay
Qty Students
Change First -‐ Essay
First Score
2 2 1 1 3 5 1 1 0 2
+ 1.25 + 1.25 + 1.00 + 1.50 + 1.00 + 1.10 + 0.60 + 1.10
5 5 9 8 8 25 8 9 17 34
+ 0.60 + 0.60 + 0.27 + 0.71 + 0.60 + 0.52 + 0.80 + 0.97 + 0.44 + 0.66
9 9 5 10 8 23 3 8 2 13
+ 0.14 + 0.14 -‐ 0.04 + 0.14 + 0.02 + 0.06 + 0.20 + 0.45 + 0.77 + 0.44
1 1 1 0 1 2 0 0 0 0
-‐ 0.30 -‐ 0.30 0.00
1.96 1.96 1.78 1.82 1.67 1.75 1.59 1.71 1.64 1.65
2.34 2.34 1.98 2.27 2.04 2.10 2.23 2.46 2.12 2.27
0.38 0.38 0.20 0.45 0.36 0.35 0.63 0.74 0.48 0.61
19.5% 19.5% 11.2% 24.9% 21.9% 19.8% 39.8% 43.5% 29.1% 37.1%
1 1 2 0 3 5 0 4 0 4
+ 2.50 + 2.50 + 1.00
2 2 3 9 5 17 5 12 10 27
+ 1.50 + 1.50 + 1.50 + 1.39 + 0.90 + 1.26 + 0.60 + 0.79 + 0.75 + 0.74
10 10 7 7 10 24 7 2 9 18
+ 0.55 + 0.55 + 0.50 + 0.29 + 0.35 + 0.38 + 0.43 0.00 + 0.42 + 0.38
4 4 4 3 2 9 0 0 0 0
-‐ 0.25 -‐ 0.25 -‐ 0.38 -‐ 0.33 0.00 -‐ 0.28
2.09 2.09 2.09 1.92 1.83 1.94 1.88 1.40 1.91 1.71
2.68 2.68 2.63 2.63 2.38 2.54 2.38 2.22 2.50 2.37
0.59 0.59 0.53 0.71 0.55 0.60 0.50 0.82 0.59 0.65
28.2% 28.2% 25.4% 37.0% 30.1% 31.0% 26.7% 58.4% 31.0% 38.1%
2 2 1 1 5 7 1 3 0 4
+ 1.80 + 1.80 + 1.40 + 1.60 + 0.80 + 1.00 + 0.50 + 1.33
4 4 9 11 9 29 7 15 18 40
+ 0.90 + 0.90 + 0.69 + 0.64 + 0.58 + 0.63 + 0.81 + 0.65 + 0.58 + 0.65
10 10 2 6 6 14 4 0 1 5
+ 0.46 + 0.46 + 0.50 + 0.45 + 0.17 + 0.34 + 0.35
-‐ 1.00 -‐ 1.00 -‐ 0.25 -‐ 0.80
+ 0.75 + 0.43
1 1 4 1 0 5 0 0 0 0
1.85 1.85 1.83 1.75 1.53 1.69 1.58 1.36 1.62 1.52
2.48 2.48 2.30 2.31 2.04 2.21 2.22 2.12 2.22 2.18
0.64 0.64 0.48 0.55 0.51 0.51 0.63 0.76 0.59 0.66
34.4% 34.4% 26.0% 31.5% 33.3% 30.4% 40.0% 55.9% 36.5% 43.8%
2 2 2 1 5 8 1 2 0 3
+ 1.90 + 1.90 + 1.25 + 1.70 + 1.12 + 1.23 + 0.30 + 1.00
5 5 9 11 9 29 6 14 14 34
+ 0.78 + 0.78 + 0.54 + 0.65 + 0.56 + 0.59 + 0.78 + 0.75 + 0.53 + 0.66
8 8 2 6 6 14 5 2 5 12
+ 0.34 + 0.34 + 0.45 + 0.25 + 0.15 + 0.24 + 0.50 + 0.15 + 0.15 + 0.30
2 2 3 1 0 4 0 0 0 0
-‐ 0.20 -‐ 0.20 -‐ 0.30 -‐ 0.40
1.88 1.88 1.73 1.79 1.45 1.65 1.65 1.32 1.71 1.55
2.47 2.47 2.19 2.32 2.02 2.17 2.28 2.03 2.14 2.13
0.59 0.59 0.46 0.53 0.58 0.53 0.62 0.71 0.43 0.58
31.3% 31.3% 26.8% 29.4% 39.8% 31.9% 37.5% 54.1% 24.9% 37.3%
1 1 1 0 1 2 0 2 0 2
+ 2.15 + 2.15 + 1.13
4 4 9 13 12 34 6 14 15 35
+ 0.79 + 0.79 + 0.62 + 0.73 + 0.63 + 0.66 + 0.68 + 0.78 + 0.58 + 0.68
10 10 2 3 6 11 6 2 4 12
+ 0.41 + 0.41 + 0.44 + 0.62 + 0.13 + 0.32 + 0.52 + 0.38 + 0.29 + 0.42
2 2 4 3 1 8 0 0 0 0
-‐ 0.04 -‐ 0.04 -‐ 0.22 -‐ 0.21 + 0.20 -‐ 0.16
1.94 1.94 1.86 1.82 1.62 1.76 1.68 1.45 1.72 1.61
2.49 2.49 2.27 2.38 2.12 2.25 2.27 2.21 2.24 2.24
0.55 0.55 0.42 0.56 0.50 0.50 0.60 0.76 0.52 0.63
28.2% 28.2% 22.5% 30.8% 30.9% 28.3% 35.6% 52.4% 30.3% 39.0%
+ 0.85
+ 1.00 + 1.00 + 1.31 + 1.31
+ 1.13
+ 0.77
+ 1.48 + 1.30 + 1.03 + 1.03
-‐ 0.70 -‐ 0.35
-‐ 0.36
-‐ 0.33
Essay Score
Improvement
APPENDIX B -‐ HISTOGRAMS Histogram 1. Essay – Mechanics Course 1 Average 2.34
10
4
3
Remedial
Below Average
Course 2 Average 1.98
Average
Above Average
9 5
1 Remedial
1 Below Average
Average
Above Average
Course 3 Average 2.23 4
Remedial
Below Average
5
3
Average
Above Average
11
Course 4 Average 2.27
5 1
Remedial
2
Below Average
Average
Course 5 Average 2.46
Above Average
9 7 2
Remedial
Below Average
Average
Above Average
12
Course 6 Average 2.04 6
2 Remedial
Below Average
Course 7 Average 2.12
Average
Above Average
10 7 2
Remedial
Below Average
Average
Above Average
Histogram 2. Essay – Basic Knowledge 11
Course 1 Average 2.68 6
Remedial
Below Average
Average
Course 2 Average 2.63
Above A verage
9 7
Remedial
Below Average
Course 3 Average 2.38
Average
Above A verage
9
3
Remedial
Below Average
Course 4 Average 2.63
Remedial
Average
10
Below Average
Course 5 Average 2.22
Average
9
Above A verage
10 4
Remedial
Above A verage
Below Average
4
Average
Above A verage
14
Course 6 Average 2.38 4 2 Remedial
Below Average
Average
Above A verage
13
Course 7 Average 2.50 5
1 Remedial
Below Average
Average
Above A verage
Histogram 3. Essay – Thinking Historically Course 1 Average 2.48
10 7
Remedial
Below Average
Course 2 Average 2.30
Average
Above A verage
9 5 2
Remedial
Below Average
Average
Above A verage
Course 3 Average 2.22 5
3
Remedial
Below Average
Course 4 Average 2.31
Average
4
Above A verage
9
7 3
Remedial
Below Average
Average
Above A verage
Course 5 Average 2.12 6
Remedial
Course 6 Average 2.04
7 5
Below Average
Average
8
8
Above A verage
4
Remedial
Below Average
Average
Above A verage
13
Course 7 Average 2.22 3
Remedial
Below Average
3
Average
Above A verage
Histogram 4. Essay – Thinking Critically Course 1 Average 2.47
10 7
Remedial
Below Average
Course 2 Average 2.19
Average
Above A verage
10
4 2 Remedial
Below Average
Average
3
3
Below Average
Average
Above A verage
Course 3 Average 2.28 6
Remedial
Course 4 Average 2.32
9
Above A verage
8
2 Remedial
Below Average
Average
Above A verage
Course 5 Average 2.03 7 5
Remedial
Course 6 Average 2.02
Below Average
9
Average
6
Above A verage
8 3
Remedial
Below Average
Average
Above A verage
13
Course 7 Average 2.14 3
Remedial
Below Average
3
Average
Above A verage
Histogram 5. Essay – Composite Course 1 Average 2.49
10 7
Remedial
Below Average
Course 2 Average 2.27
Average
Above A verage
10 5 1
Remedial
Below Average
Average
Above A verage
5
5
Average
Above A verage
Course 3 Average 2.27 2 Remedial
Below Average
11
Course 4 Average 2.38
7 1 Remedial
Below Average
Average
Above A verage
Course 5 Average 2.21 7 5
Remedial
Below Average
Average
6
Above A verage
12
Course 6 Average 2.12 5
3
Remedial
Below Average
Average
Above A verage
12
Course 7 Average 2.24
5 2 Remedial
Below Average
Average
Above A verage
