Incorporating game mechanics into a network of online study groups Ari BADER-NATAL 1 Grockit, Inc.
[email protected] Abstract. The recent movement towards publishing open educational resources has increased the variety and quantity of learning materials available to students outside of the traditional classroom environment. Several core characteristics of the classroom environment, however, are difficult to offer through a web-based interface, including: (1) interaction and camaraderie among a cohort of peers, (2) the ability to get “real-time” answers to pressing questions, and (3) a motivating force to keep the student engaged over time. An online learning environment can approximate the value of peer cohorts and live question-answering by supporting (and encouraging) synchronous interactions among individuals studying a common topic. A learning system can motivate participation and collaboration by incorporating elements of game mechanics in the activity. We discuss Grockit, a recently-launched website that combines a virtual study group format with multi-player game dynamics to provide an engaging live collaborative learning environment for geographicallydispersed learners. Keywords. learning games, study groups, motivation, collaboration
Not all learning happens in classrooms. Self-motivated learners have a wealth of options, vastly enriched by widespread access to the internet. Now, in addition to local libraries, students have access to open educational resources such as MIT’s OpenCourseWare initiative [5] and educational podcasts such as those found on iTunes U [1]. But several key aspects of classroom learning are still not readily available, including (1) a peer group to learn with and from, (2) the opportunity to ask someone a question and get an immediate response, and (3) the structure and motivation that helps students remain committed over time. Synchronous communication among learners on an educational site could theoretically enable a virtual community of peer learners. The approach that we have been pursuing at Grockit (grockit.com) has been to leverage various game mechanics to affect how students choose to participate and interact with each other. With respect to AIED systems, the goal of the current approach is to reduce the traditional challenge of identifying and remediating learner misunderstandings to the much simpler task of enabling and motivating peers to do this for one another. As of June 2009, Grockit has built support for two groups of learners. The first is for students studying for the GMAT, a common business school entrance exam. The second is for students studying for the SAT, a national college entrance exam. These types of groups offer a good testbed for live online learning communities, as there are a large number of students who share the common learning goal of mastering the skills that 1 Corresponding
Author. Grockit, Inc. 777 Florida St. #300. San Francisco, CA 94110. http://grockit.com
Lobby
Friends
Profile
Feedback
Messages
Groups
Users
Content
Find a Tutor
Ari Bader-Natal
sign out
GMAT
Join a Game
Top GMAT Players
Play a Verbal Quest Challenge yourself with questions targeted to your ability level.
Praveen Gangasani 1900 XP earned
Start a Game Review Games
Quests Verbal Quest Available! Quantitative Quest Available!
Syed Basha 1691
Welcome to Grockit GMAT! Choose a game below to play now.
David Freeling
To play a game scheduled in the future, click RSVP to reserve a spot. You can also start your own game and play it now or schedule it for later.
John Doe 1032
Manishkumar Munot
Live Games
882
Pooja Shetty
GMAT Verbal Practice In Progress
1098
700
Abhinit Kulkarni
Scheduled by Komal Nadeem
496
rapa hoi 422
Quests become available as you answer practice questions. A quest is an opportunity to challenge yourself while you work through a finite set of questions.
Everyone on Grockit starts as a Beginner. You'll gain points as you answer questions correctly. Gain enough points and you'll move up to Intermediate, Advanced, and Expert.
Check out the Grockit blog for tips and announcements of new features to help you prepare for the GMAT.
5 Player Game | 2 Spots Available
Ely M 398
Cornell Carpenter
Upcoming Games
382
most XP in the last 7 days.
GMAT Verbal Practice Helpful & Friendly
Today 1:00pm Scheduled by Stephanie Wambach
David Freeling 14 GP earned
5 Player Game | 1 Spot Available
Syed Basha 8
ratnakar trivedi 7
GMAT Quantitative Practice
Anand Patel 7
Today 3:00pm Scheduled by Stephanie Wambach
Rosie Chawla 7
Figure 1. Each learning community has a “game lobby” which serves as an entry point, displaying Lucy Nin-progress 5 Player Game | 3 Spots Available and upcoming study sessions. (Note that all screenshots were captured in June, 2009). Leader-boards are dismanthena manoj kumar played on the left, and available quests are displayed on the right. Invite your friends! 6
6
GMAT Verbal Practice
.
Tomorrow 1:00pm Scheduled by Stephanie Wambach
Ashok Thiyagarajan 5
NN 5
S Shah 4
5 Player Game | 3 Spots Available
most GP in the last 30 days. XP: Experience Points are earned when you answer questions correctly.
GMAT Quantitative Practice Tomorrow 3:00pm Scheduled by Stephanie Wambach
GP: Grockit Points are earned when your helpful discussion messages receive a star. 5 Player Game | 3 Spots Available
GMAT Quantitative Practice Jun 12, '09 1:00pm Scheduled by Stephanie Wambach 5 Player Game | 3 Spots Available
GMAT Verbal Practice Jun 12, '09 3:00pm Scheduled by Stephanie Wambach 5 Player Game | 3 Spots Available
GMAT Verbal Practice Jun 12, '09 6:00pm
Anna Wlodarczyk
Scheduled by Anna Wlodarczyk
Figure 2. In each game, a series of questions are presented, such as the one above. After all players answer 5 Player Game | 2 Spots Available the question, they are provided with an opportunity to see, and discuss, the correct answer before moving on to a new question. The discussion includes GMATpane Verbal Practiceboth player discussion and system feedback (e.g. point awards). Jun 15, '09 Within each study group session (referred to as a “game”), discussion among students (“players”) takes place 1:00pm Scheduled by in a chat pane. Stephanie Wambach 5 Player Game | 3 Spots Available
GMAT Quantitative Practice Jun 15, '09 3:00pm Scheduled by Stephanie Wambach 5 Player Game | 3 Spots Available
GMAT Quantitative Practice
05/19/2009
with Stephanie
02/23/2009
s with Stephanie
02/19/2009
with Stephanie
Nitish Michael has played 0 questions with Stephanie
Abhinit Kulkarni has played 126 questions with Stephanie
Friends
550 Profile
GP
2
Level
5
02/19/2009 with Stephanie
02/19/2009
with Stephanie
Improving Sentences Start a Game
71%
Identifying Sentence Errors
76%
Feedback
Messages Find a Tutor Patrick Ewing has played 1 questions with Stephanie
Quests Response All Difficulties
All Time
12 questions Passage Based Reading
70%
0:43
100%
Verbal Quest Multiple Choice 100% answer 20 practice questions to unlock SAT Reading
Sentence Quantitative Completion Diagnostic
68%
0:50
78%
Arena Reed hasaplayed 13 questions with Stephanie Play Quantitative Diagnostic
33%
1:46
Quests become available as you GMAT Achievements answer practice questions. A quest 100 is anQuestions opportunity to Correct challenge yourself while you 04/20/09 work through a finite set of 1 GMAT Verbal Quest questions.
40%
0:40
47%
0:14
50%
0:37
53%
0:37
53%
Completed
0:36
62%
0:36
63%
0:36
63%
0:37
64%
0:54
65%
1:31
67%
0:35
67%
0:39
Verbal Diagnostic Everyone on Grockit starts as Completed a Beginner. You'll gain points as you answer questions Quantitative Diagnostic correctly. Gain enough points Completed and you'll move up to Intermediate, Advanced, and Expert.
Figure 3. Diagnostics Check out the Grockit blog for and Quests (top), and tips and announcements of new features to help you prepare for Badges (bottom). the GMAT.
69%
1:01
Andrea Johnson has played 4 questions with Stephanie
0:49
71%
0:49
72%
0:47
72%
0:47
Invite your friends!
Performance Analytics
GMAT In Progress Verbal
0:15
Scheduled by henrietta Aifah 605
expressions
Diction
Upcoming Games Tone form GMAT Verb Verbal Practice Parallelism Tomorrow 8:30am Rhetorical Scheduled by construction Manishkumar Munot Sentence Correction Logical predication Grammatical
3
68% 70%
0:31
33%
0:40
2
0:14
239 49
-
(no data)
-
(no data)
-
(no data)
-
(no data)
-
(no data)
-
(no data)
Improving TopSentences GMAT Players
Pooja Shetty
Student Produced 1166 Response
50%
Sentence Lucy N 964 Completion Manishkumar Munot
Passage Based 792 Reading
53%
0:44
0%
2:17
0%
1:29
0%
1:29
0%
0:33
0%
2:17
0%
Completed David Freeling
62%
0:36
63%
0:36
63%
5 Player Game | 4 Spots Available
0:37 Manishkumar Munot 26 0:54
11 GP earned
64%
Quantitative Diagnostic Anand Patel Completed 7 Rosie Chawla 7
ratnakar trivedi
65%
Critical Reasoning
209
0:49
71%
Argument evaluation Tomorrow Argument 1:00pm
88
0:49
72%
7
manthena manoj kumar 6
Lucy N 6
Ashok Thiyagarajan 5
NN 5
S Shah
101 0:47 72% Scheduled by of the day, there are generally several groups of construction are tested with the exam. At any time Stephanie Wambach Reading 157 0:47 80% students collaborating (and siteComprehension activity is quickly increasing). most GP in the last 30 days. Supporting Ideas 1:00 For the GMAT site, students connect from 19around the world, generally 84% from a home Modification 1 0:40 100% XP: Experience Points are GMAT Quantitative earned or shared computer. As aspiring business Practice school students are often several years out ofwhen you answer Logical structure 3 0:15 100% questions correctly. Tomorrow 100% school, they are less likely to have friends or co-workers the same Antecedent 1 2:57 currently studying for100% 3:00pm GP: Grockit Points are earned Scheduled by when your helpful discussion 100% Evaluation Stephanie Wambach exam. The characteristics of the standardized test-based groups (i.e. large numbers of receive a star. messages 100% Verb tense students independently studyingGrammar the same material) appear to be a good match for the live online study group, though future Grockit groups will- not be restricted to test-oriented Referents GMAT Structure Verbal Practice domains. Jun 11, '09 Comparison 1:00pm Over the past several months, we have incorporated into Grockit a variety of gameScheduled by Pronoun reference Stephanie Wambach like mechanisms to motivate students to productively engage in the learning community: 65% Quantitative 384 0:59 4
4
(no data) (no data)
5 Player Game | 4 Spots Available
0%
Angles 1 2:17 0% two point systems, which Points and Leaderboards Grockit currently incorporates GMAT Intersecting Quantitative Practice lines 3 1:29 0% serve as reputation-likeJunand indicators: Experience Points (XP) and Grockit Points angles 11, '09 3:00pm Parallel Lines 3 1:29 0% Scheduled by (GP). A player earns XP by answering questions. The point value for an answer is Stephanieon Wambach Operations 0% a function of the difficulty of the question1 (as 0:33 estimated by a one-parameter Item numbers measure in Response Theory model)Angle and the accuracy of2:17 the response. While this XP met1 0% degrees Operations with ric reflects the player’s and 6performance as a learner, the GP scale GMATparticipation Quantitative Practice 0:43 17% rational numbers Jun 12, '09 reflects the player’s activity as a teacher.4 When a peer finds a player’s comment 1:00pm Measurement 0:46 25% Scheduled by Estimation 3 0:47 opportunity 33% or explanation particularly helpful, the to award GP to that Stephanie Wambachthey have problem 3 0:27 33% player. In the right-hand Work column of the Game Lobby (illustrated in Figure 1), “reCircles 5 1:02 40% cent top player” leader-boards are displayed for GP earned, and players Counting methods 5 1:28 XP and 40% Verbal Practice Simple coordinate frequently compete GMAT to appear on these lists. 11 1:14 45% Jun 12, '09 geometry 3:00pm Rectangularto solids Performance Statistics In addition XP and GP points achieved, student’s Scheduled by displaying 4 1:08 50% Stephanie and cylindersWambach can view (and opt to share) their performance statistics (see Figure 4). This data Rounding 2 0:53 50% provides students with test-, section-, and2 skill-grained feedback on their perforLines 0:39 50% Concepts of sets 50% mance (and can be GMAT restricted to include20data 0:56 from a specified time and/or diffiVerbal Practice Anna Wlodarczyk Absolute value 2 0:52 50% Jun 12,select '09 Scheduled by of skills from the list, then review detailed culty range). Students can a set Data Sufficiency 145 0:57 54% 6:00pm Anna Wlodarczyk Decimals tagged with9these 1:02 56% explanations of the questions skills. 5 Player Game | 4 Spots Available
17% 25%
0:47
33%
0:27
33%
5 Player Game | 4 Spots Available
1:02
40%
1:28
40% 45%
1:08
50%
0:53
50%
0:39
50%
0:56
50%
0:52
50%
5 Player Game | 4 Spots Available
54%
1:02
56%
1:10
56%
Triangles
25
Solving two linear equations with two
1:10
5 Player Game | 3 Spots Available
56%
54
1:02
59%
55
0:53
60%
15
0:59
60%
0:57
60%
unknowns
1:02
59%
0:53
60%
0:59
60%
GMAT Verbal Practice Properties of Jun 15, '09 Numbers 1:00pm
Scheduled by Rate problem Stephanie Wambach Arithmetic operations
10
37%
Cornell Carpenter
Helpful &Verbal Friendly Diagnostic
0:36
0:44
78%
GMAT most XP in1the last Verbal 7 days.Quest Completed
53%
55
1
100%
David Freeling
100 Questions Correct elon wolf 04/20/09 440
47%
Tomorrow Figure 4. Player performance statistics are reported 67% at the test, sec9:30am Main Idea 12 1:31 Scheduled by tion, and skill levels. Color-coded bars help students67% focus their atManishkumar Munot Idiom 94 0:35 tention (for future games and reviews) on the skills 69% for which the Agreement 42 0:39 5 Player Game | 4 Spots Available student had percentage Evaluation of a the plan lowest 17 1:01 accuracy. 71%
Distance problem
100%
SAT Reading 982
(no data)
65%
76%
Praveen Gangasani 1466 SAT Math
(no data)
(no data)
71%
Syed Basha
Identifying 1567 XP earned Sentence Errors
(no data) 5 Player Game | 4 Spots Available (no data)
0:59
0%
Paragraphs
542
40%
49 0:37 Manishkumar Munot 43 0:37 34
questions left 40 Improving
GMAT Achievements Abhinit Kulkarni
5 Player Game | 4 Spots Available
17
5
sign out
546
GMAT construction Quantitative Practice Inference
All Time
0:43
1:46
a b
John Doe
0:50
5
2
Level
(no data)
2:57
0:57
989
550
GP
1032 Multiple Choice
All Difficulties
Live Games
GMAT Quantitative Practice
SAT
Praveen Gangasani
84%
0:40
53%
5 Player Game | 4 Spots Available
80%
1:00
02/19/2009
Welcome to Grockit GMAT! Choose a game below to play now.
GMAT Verbal Practice
71%
02/19/2009
for all Diagnostic the great work that youyour do and The Thanks Quantitative will assess skill.being such a great member of the Grockit team!!!!
Evaluating
37%
71%
Data Sufficiency
SAT Writing
I am so thankful that you accepted the story that led to this Testimonial being possible. Gold star, all the way. ++
GMAT
Application
0:31
1:14
02/19/2009
hey stephanie - thanks for your help today! To play a game scheduled in the future, click RSVP to reserve a spot. You can also start your own game and play it now or schedule it for later.
Review Games
SAT Math Student Produced
or your help today!
0%
Problem Solving
XP
Hey Stephanie, thanks on behalf of all Grockiteers for helping us with our GMAT prep
Improving Join a Game Paragraphs
work that you do and being such a great member of the
0:46
02/23/2009
SAT Writing
u accepted the story that led to this Testimonial being he way. ++
0:43
05/19/2009
Thanks!!!!
SAT XP Lobby
n behalf of all Grockiteers for helping us with our GMAT
-
Comprehension Edit Photos GMAT Quantitative Practice Send a Message Problem Solving 71% Write a Testimonial Data Sufficiency 53% Remove friend
5 Player Game | 4 Spots Available
Subsequent activity (Person count) subsequently participated in a game subsequently participated in a game with others subsequently logged in again subsequently participated in game discussion subsequently reviewed questions
Always available (86) 54.7% (47) 31.4% (27) 43% (37) 24.4% (21) 66.3% (57)
Every 20 questions (100) 69% (69) 55% (55) 59% (59) 41% (41) 72% (72)
Every 5 questions (73) 61.6% (45) 42.5% (31) 47.9% (35) 27.4% (20) 68.5% (50)
Never available (101) 60.4% (61) 42.6% (43) 38.6% (39) 24.8% (25) 71.3% (72)
p-value 0.25 0.01 * 0.03 * 0.03 * 0.83
Table 1. Testing the effect of Quest availability on subsequent participation. The null hypothesis is that the four populations of students have the same true proportions, and the alternative is that the proportion is different in at least one of the populations. Starred p-values indicate significance at the α = 0.05 level, two-tailed.
Diagnostics, Quests, and Badges By answering questions in collaborative games, players work towards “unlocking” certain fixed-duration solo activities (seen in Figure 3a). A Diagnostic includes a fixed set of questions designed to generate an initial assessment of the student’s current ability level. A Rasch Model [3] is used to construct and evaluate this assessment. After completing a Diagnostic, Quests can be unlocked through subsequent collaborative game-play. Quests are designed to provide students with targeted practice, including questions for which the Rasch Model predicts a probability of response accuracy near p = 0.5. “Medium” difficulty challenges such as these have been found to increase intrinsic motivation [4] and to facilitate a “flow” experience [2]. Students are awarded badges (Figure 3b) for various accomplishments, such as the completion of Diagnostics and Quests. One interesting recent finding came from a small randomized experiment run over a two week period in June, 2009 (shown in Table 1). Students who completed a Diagnostic were randomly assigned to one of four groups, determining if and when a Quest was available to them (“unlocked”). For three of the five outcome measures of interest, there was a statistically significant difference among the groups, with the group for which Quests were unlocked every 20 questions outperforming the others (visible in Figure 4a). Perhaps the added question spacing introduces an attainable goal that motivates increased participation. Game mechanics, such as this one and those outlined above, offer a rich toolset for motivating students in learning and collaboration, and help increase engagement in the virtual study group environment. Acknowledgements Writing about the game-like aspects of Grockit has been complicated by the rapid pace of system development. Many thanks to Farbood Nivi and the entire Grockit team for this added challenge. References [1] [2] [3]
Apple, Inc. iTunes U. http://www.apple.com/education/mobile-learning/. Mihaly Csikszentmihalyi. Flow: The Psychology of Optimal Experience. Harper&Row, New York, 1990. Gerhard H. Fischer and Ivo W. Molenaar, editors. Rasch Models: Foundations, Recent Developments, and Applications. Springer-Verlag, New York, 1995. [4] Mark R. Lepper. Motivational considerations in the study of instruction. Cognition and Instruction, 5(4):289–309, 1988. [5] Massachusetts Institute of Technology. MIT OpenCourseWare. http://www.mit.edu/ocw.
