Reimagining Leaderboards: Towards Gamifying ... - ACM Digital Library

Stratford, Ontario, Canada, October 2-4, 2013

Gamification 2013 | PROCEEDINGS

Reimagining Leaderboards: Towards Gamifying Competency Models through Social Game Mechanics Katie Seaborn University of Toronto 5 King’s College Road Toronto, ON, M5S 3G8 [email protected]

Peter Pennefather University of Toronto 144 College Street Toronto, ON, M5S 3M2 [email protected]

ABSTRACT

Deborah I. Fels Ryerson University 350 Victoria Street Toronto, ON, M5B 2K3 [email protected]

learning, qualitative assessment, and student-centrism; the result is an educational continuum that embraces competencies as pedagogical constructs that are multi-dimensional, dynamic, developmental and contextual [2,5]. However, critics have brought attention to issues in current implementations of competency-based assessment, including a reliance on legislated standards, a lack of empirical validation, and the difficulty of accurate self-assessment [5,11,16].

Competency models have been widely employed within education, training and development contexts. In particular, medical education programs have become a platform of exploration around innovations in competency-based assessment. Approaches that employ dynamic, flexible models in a social context are now being sought. In this paper, we present the design of a deliberately gamifiable competency model system that encourages pre-professional development through collaborative peer appraisals. We propose a framework for incorporating social game mechanics into a competency modelling system and draw attention to how a collaborative approach to evidencing progress is a potentially fruitful solution to the threat of demotivation associated with the competitive slant of traditional leaderboards.

This paper advances an approach that is currently being developed to address these concerns. The BERGE framework of competency model-based education allows participants to track their responses to competency challenges posed by the curriculum and creatively represent emergent competencies within curricular constraints. The competency model visualization is interactively linked to portfolio items submitted by the learner, their peers, and mentors for self and peer evaluation as they progress through the program. Participants have the opportunity to visualize their progress with respect to near peers in a social gamified format.

Author Keywords

Social gamification; leaderboards; competency models. ACM Classification Keywords

D.2.2. Software engineering: Design tools and techniques: User interfaces; H.5.2. User interfaces: Graphical user interfaces (GUI).

The social nature of the proposed BERGE framework invokes the notion of a leaderboard. We know of no research that has considered how a cohort of dynamic competency model visualizations can operate as a motivating leaderboard nor the implications for learners. Further, leaderboards are traditionally framed as a quantitative, competitive game element, and whether or not they are motivational for all types of learners has been questioned [1,3]. A potential way forward is to reimagine the leaderboard as a collaborative, qualitative game element that invites prosocial interactions. Prosocial interactions occur when individuals willingly act in the interest of others [4]. In contrast to competitive approaches, prosocial setups promote positive social relationships with peers and greater overall achievement [12]. Whether this transfers to a social gamified environment has yet to be explored.

General Terms

Design; Human Factors. INTRODUCTION

Competency-based assessment has swiftly gained prominence in education, training and development. Competency models provide a standardized structure for the development of specific abilities, knowledge and behaviours that impact performance in a given domain [10]. Recent years have seen the rise of competency-based assessment in medical education [5]; in Canada, the CanMEDS “daisy” is a widely adopted standard [17]. The competency modelbased approach values demonstrable ability, transferable Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In this paper, we outline the design of a prosocial leaderboard system envisioned for BERGE using cohort-based medical education as an example. We discuss how a leaderboard setup founded on social capital is primed to address concerns about the demotivational outcomes linked to conventional leaderboards. We end with the design of an empirical study to explore the effects of social game mechanics with human subjects.

Gamification '13, October 02-04 2013, Stratford, ON, Canada Copyright 2013 ACM 978-1-4503-2815-9/13/10 $15.00. http://dx.doi.org/10.1145/2583008.2583026

107



BACKGROUND

oped through social interaction [14]. It is embodied in altruism, sharing, reciprocity and gratitude. Previous research suggests that prosocial games inspire prosocial behaviour [6,18], but whether this transfers to a social gamified setup is unknown. A preliminary study on a gratitude system showed positive results for receivers, but contributors were affected only when a leaderboard was involved [8]. This suggests that prosocial interactions may need to be represented in a concrete form to ensure that all users benefit.

Social gamification is an emerging subgenre of gamified systems that use mechanics and elements from social games, which feature interactions designed for close peers and direct ties to social networking systems, in which they are often embedded [7,15]. Simões, Redondo and Vilas developed a social gamification framework for educational systems that was implemented for a series of elementary schools in Portugal [15]. Game elements were drawn directly from social games; for example, peer appraisal, including the use of a “like” button, giving and sharing rewards, and teamwork-based goals. The in-progress Mission-Oriented Social Gaming and Learning (MOSGAL) platform by Lin et al. uses social media analytics to generate “people-likeme” recommendations in preparation for social gamification interactions [9]. We seek to extend this initial work on social gamification by focusing on how one element—the leaderboard—can be reimagined as a prosocial tool.

SYSTEM DESIGN

The BERGE—Bounded emERGEnce—framework of competency model-based assessment is designed around the notion of bounded emergence, where users are able to develop novel competencies while still satisfying regulated standards. As a dynamic competency modelling system, BERGE empowers the learner to demonstrate a diverse set of competencies in a collaborative, social portfolio context.

Leaderboards are a performance comparison game element. Employed in single and multiplayer contexts, leaderboards are used to map progress and incite action, usually by inspiring a competitive disposition through ranking and comparison. Additionally, comparison metrics are generally, if not exclusively, quantitative. In multiplayer settings, leaderboards display rankings of near-performing peers and/or rankings of highest performing peers, where the player may not be represented if their score is not high enough. Single player leaderboards compare the player’s latest score to their previous scores. An example of a traditional quantitative leaderboard can be found in Figure 1.

A description of the core elements and social functionality is provided here. The running example features a persona named Kevin, a fourth-year medical student who recently returned from military training, during which time he gained field experience as a combat medic. BERGE begins with an expandable rubric based on a standardized model of essential competencies that allows for the addition of novel competencies (Figure 2). The student is placed at the centre of the model, which is made up of two layers: the top layer is comprised of standard competencies while the bottom layer represents additional competencies. Each layer is divided into competency “slices” inspired by the CanMEDS model [17]. The concentric tree ring-like circles act as a visual cue of progress for the entire model, allowing comparisons to be drawn between competencies.

Figure 1. Example of a traditional multiplayer leaderboard.

Despite the prevalence of leaderboards, little research has examined how they affect different kinds of people, in what ways, and why [3]. This gap has caused at least one researcher to recommend using badges instead of leaderboards [3]. In his study on how leaderboards impact players’ experience of fun, Butler compared three types of leaderboards: single-player, multiplayer with higher scores, and multiplayer with lower scores [1]. His findings indicate that rank comparisons do affect players’ perceptions of fun, and that winning or losing polarizes responses. This raises the question of how to design a multiplayer leaderboard to ensure that all participants benefit; the possibility of a prosocial setup is explored here.

Figure 2. Competency model for a single user. © Katie Seaborn.

Here, Kevin has developed all of his core competencies as well as six emergent competencies; moreover, these additional competencies, which were gained during his service

Prosocial behaviour is an aspect of social capital, or the knowledge, resources and relationships shared and devel-

108



as a combat medic, have a greater number of portfolio items in evidence of his practical knowledge and ability.

leaderboard is provided in addition to individual competency models and offers a gestalt of the overall competency of the group. The competency petals include all emergent competencies as well as the items and individuals associated with each. Unlike traditional leaderboards, this setup avoids ranking contributions, instead providing a qualitative overview of the competency makeup of the cohort. Kevin and his peers are able to see how they contribute to the group model and in what specific ways they can develop individual competencies by reviewing what types of activities their peers have engaged in. The ability to contact peers provides a way for closer peer relationships to develop; for instance, if Kevin sees that a peer has taken an external course to develop her “Technologist” competency, he can contact the student for more details about her experience.

Selecting a competency slice reveals a portfolio of items in evidence of the student’s competency in that area. The student, their peers, and other users, such as teachers and mentors, can add items and review the details associated with each item, including other competencies for which the item acts as evidence. The student has some degree of control over what information they share with peers; however, the item’s type (e.g. report, case study, exam results, etc.) and a short, descriptive title are always publicized. Prosocial Leaderboard Approach

Social game mechanics can be applied to encourage prosocial behaviours. Drawing from the mechanics outlined in [13], we suggest the following prosocial leaderboard setup. Status

All prosocial interactions are recognized. Collaborators are given a special status indicating the degree of their collaborative output. For example, after five collaborative actions, the user is given the status of “Collaborator.” After another five, this status will be upgraded to “Social Collaborator.” Status is displayed every time a user’s profile is invoked, including within other participants’ portfolios next to contributed items. Status provides meaningful pleasure through positive recognition.

Figure 3. The qualitative group leaderboard over time. © Katie Seaborn.

In Figure 4, a core competency required by the curriculum—the “Collaborator” competency—is selected. The competency is briefly described and the portfolio items associated with it are shown. Each item can be selected to reveal view and download links, the competencies associated with the item, and the contributor. In this case, an email correspondence has been contributed by Dr. Shoemaker to Kevin’s portfolio for the “Collaborator” competency, and is also associated with the “Technologist” and “Communicator” competencies. The attachment can be downloaded.

Scarcity

The ability to add new items to one’s portfolio is modulated by the system. Insertions are unlocked as individuals contribute to each others’ portfolios. For example, if Kevin contributes an item to another student’s portfolio, a space for him to add an item to his portfolio will be unlocked. Development is locked if one or more users lag or are ignored, prompting the group to help these users build up their portfolio. The intent is to facilitate reciprocal interaction. The result is a set of rich portfolios augmented by peer assessment. Karma Points

Karma points are freely given to others based on the perceived quality of their contributions. If Kevin appreciates a comment, contribution or other social gesture, he can give karma points to the person responsible. As a user’s karma points accumulate, they contribute to the status of the user by signalling their ability to provide appreciated contributions and engage in quality interactions. As a meaningful action, the pleasure is in the giving as well as the receiving. Group Leaderboard

The main contribution of this paper is a novel approach to leaderboards: using a qualitative group leaderboard based on the single player model of self-referential motivation over time (Figure 3). A cumulative, cohort-wide competency model shifts the goal from individual success to group success: “When one of us wins, we all win.” This group

Figure 4. A single user’s contribution to a core competency. © Katie Seaborn. Limitations

The described prosocial setup requires users to be invested in the overall goal of developing models of their and others’

109



competencies from the start. Peers may not be able to assess or contribute to others’ models; however, this is unlikely in a cohort where peers are attending most of the same classes and are given similar opportunities for practice, if not shared lab and volunteer work. Furthermore, the sharing of portfolio items gives underperforming peers concrete, practice-based ideas on how to develop specific competencies.

5. Frank, J.R., Snell, L.S., Cate, O.T., et al. Competencybased medical education: Theory to practice. Medical Teacher 32, 8 (2010), 638–645. 6. Greitemeyer, T. and Osswald, S. Playing prosocial video games increases the accessibility of prosocial thoughts. The Journal of Social Psychology 151, 2 (2011), 121– 128. 7. Järvinen, A. Game design for social networks: Interaction design for playful dispositions. Proc. 2009 ACM SIGGRAPH Symposium on Video Games, ACM (2009), 95–102.

USER STUDY

The prosocial leaderboard design will be evaluated in a mixed methods empirical study that will compare two configurations: with social mechanics (the proposed design) and without (the present design). Usage data will be recorded by the system and users will be solicited for feedback.

8. Kim, Y., Kato, D., Kunieda, K., and Yamada, K. Preliminary user study for gratitude and reciprocity in a Q&A system. Proc. CSCW 2012, ACM (2013), 169–174.

CONCLUSION

9. Lin, R.J., Chang, H., Zheng, K.-Q., and Chou, S.C.T. Technologies and innovation matrix for behaviormotivation-embedded Web 2.0 services. Proc. ICCM 2012, IEEE (2012), 863–866.

In this paper, we have outlined the design of a competency model application that uses social game mechanics to encourage collaborative portfolio development. This novel approach to leaderboards addresses concerns about the negative effects of competition on learners while upholding the benefits of this game element for the context of medical education. Future work will involve the creation of a prototype and execution of the study described in this paper. The results will serve to clarify the role of social game mechanics in gamified systems and validate the outlined prosocial leaderboard game element for non-competitive uses.

10. Lucia, A.D. and Lepsinger, R. The Art and Science of Competency Models. Pfeiffer, Hoboken, NJ, 1999. 11. Lurie, S.J. History and practice of competency-based assessment. Medical Education 46, 1 (2012), 49–57. 12. Roseth, C.J., Johnson, D.W., and Johnson, R.T. Promoting early adolescents’ achievement and peer relationships: the effects of cooperative, competitive, and individualistic goal structures. Psychological Bulletin 134, 2 (2008), 223–246.

ACKNOWLEDGMENTS

Funding for this research was provided through GRAND NCE and the University of Toronto. I thank Marie Rocchi and my KMD1001 classmates for their early involvement in the design of BERGE. Photos by Steven Depolo and Joi Ito were used, cropped and resized, in Figures 2 and 4, and sourced from Flickr under CC by 2.0 Creative Commons licenses: http://www.creativecommons.org/licenses/by/2.0. Icons by Jan Kovařík at http://www.glyphicons.com were sourced under the CC by 3.0 Creative Commons license: http://creativecommons.org/licenses/by/3.0.

13. Ross, T.L. Eight mechanics to enhance cooperation in multiplayer games. Motivate. Play., 2011. http://www.motivateplay.com/2011/07/eight-mechanicsto-enhance-cooperation-in-multiplayer-games/. 14. Sheng, M. and Hartono, R. An exploratory study of knowledge creation and sharing in online community: A social capital perspective. Total Quality Management & Business Excellence, (2013), 1–15. 15. Simões, J., Redondo, R.D., and Vilas, A.F. A social gamification framework for a K-6 learning platform. Advanced Human-Computer Interaction 29, 2 (2013), 345–353.

REFERENCES

1. Butler, C. The effect of leaderboard ranking on players’ perception of gaming fun. Proc. OCSC/HCII 2013, Springer (2013), 129–136.

16. Talbot, M. Monkey see, monkey do: A critique of the competency model in graduate medical education. Medical Education 38, 6 (2004), 587–592.

2. Dannefer, E.F. and Henson, L.C. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Academic Medicine 82, 5 (2007), 493–502.

17. Whitehead, C.R., Austin, Z., and Hodges, B.D. Flower power: The armoured expert in the CanMEDS competency framework? Advances in Health Science Education 16, 5 (2011), 681–694.

3. DuBravac, S. Game mechanics for classroom engagement. Cutting-edge Technologies in Higher Education 6, C (2012), 67–94.

18. Wohn, D.Y., Lee, Y., Sung, J., and Bjornrud, T. Building common ground and reciprocity through social network games. Ext. Abstracts CHI ’10, ACM (2010), 4423–4428.

4. Eisenberg, N., Fabes, R.A., and Spinrad, T.L. Prosocial development. In Handbook of Child Psychology. John Wiley & Sons, Inc., 2007.

110