Social Game Fliplife: Digging for talent – an analysis Heinrich Söbke, Christiane Hadlich, Naira Müller, Tobias Hesse, Christoph Hennig, Sascha Schneider, Mario Aubel, Oliver Kornadt Bauhaus-Universität Weimar, Faculty of Civil Engineering, InnoProfile: Intelligentes Lernen (Intelligent Learning), Weimar, Germany
[email protected] [email protected] [email protected]
Abstract In the context of the increasing spread of the internet, the importance of e-assessment has gained momentum in the last years. E-assessment contributes to the efficiency of recruitment processes through computer-supported pre-selection of candidates. The set of e-assessment tools also includes video games, which allow observation of candidates in informal contexts. This paper presents a case study of the social online game Fliplife and its theoretical role as a tool in the recruitment process. As we have no access to the internals of this game, we describe a theoretically feasible approach to compiling a list of job candidates. We also identify restrictions of this method and suggest solutions to the problems identified. We draw the conclusion that video games can be valuable tools in the area of recruitment.
Keywords e-assessment, Fliplife, recruitment, third place, social gaming
1 Introduction 1
The term e-assessment defines the process of computer-based measurement of a person’s skills and competencies required by a certain role, task or profession (Kupka 2009). In recruitment processes it is used as an effort-saving tool to preselect suitable candidates (Keller 2009). The advantages of e-assessment include (C. Hagmann & J. Hagmann 2011; Lippold 2010; Golembowski 2002; Steiner 2009; Kupka 2009): •
cost saving effects
•
reduction of workload
•
acceleration of recruitment processes
•
automated data redirection/processing
•
location and time independence
•
high accessibility.
In traditional e-assessment environments, the user knows about the exam situation he is exposed to: the only purpose of the software is assessment. However, video games offer a rich set of goals which the player can aim toward. Assessment here can take place by observing the player as he or she tries to reach the games’ goals. From the player’s point of view, assessment is not the main purpose of the game, and often he does not even know about it: the assessment takes place in an informal context. Given the fact that games often require and foster the development of skills needed in real life (Steinkuehler & Duncan 2008), video games could be a useful assessment tool in the recruitment 1
Other terms are Online-Assessment, Computer-assisted Assessment or Computer-based Assessment
process. Fliplife is a social online game in which players can act out a virtual life. The virtual life's components are work, education, and leisure time (i.e. sports and parties). Every player has to choose a profession and to work his way up the job ladder (Fliplife 2012). Fliplife is a browser game. This implies a high accessibility, i.e. the potential player has to overcome only low technical hurdles, which is one prerequisite for reaching diverse groups of players. Also Fliplife has a very simple and straightforward appearance (implemented in HTML5) and simple game mechanics. Compared to traditional video games the development of Fliplife is supposed to be less costly. Nevertheless it seems that many players devote a such huge amount of their spare time to playing this game, that it has become a third place (Steinkuehler & Williams 2006; Oldenburg 1999) for them. Therefore Fliplife offers the opportunity of long-term observation of players in informal contexts without the pressure of an exam situation. Fliplife is seen by its producer as a storytelling platform (Fliplife.com 2012): A few real-life German companies use this game as a platform for public relations and employer branding. Additionally it is reported that the German company Bayer AG may invite Fliplife players for job interviews (Meyer 2011).
Scenario Although it is not the main purpose of Fliplife, the hypothetical chance of getting an job interview invitation based on ones performance in the game has led us to the question of how qualified job candidates can be found with the help of Fliplife. As we have no access to the inside mechanics of this game, we created the following scenario: We work in the HR department of a company and have to fill specified vacancies. The only possibility to reach candidates is via Fliplife. We can access all the data Fliplife can collect. Our goal is to identify candidates for job interviews based on their Fliplife performance.
Solution approach The first step in our approach was to identify what types of data Fliplife could collect. Then we deduced required skills from the specified job descriptions. The third step was to match collectable data and needed skills: Given a skill - by which Fliplife-generated data types can the grade of this skill be measured? Thereafter we applied a two phase approach: In the first phase we used a fully automated filtering process to produce a short list of candidates. Then collected information (e.g. chat logs) of this group of players was examined manually to produce a list of invitations for the job interview (Müller et al. 2012).
Basic game mechanics In this section we want to explain the basic game mechanics of Fliplife as far as it is necessary to understand this text. 2
The main formal goal of Fliplife is levelling up . Projects allow the player to collect coins and experience points (XPs). XPs are the measurement for the current level of the player. Project A project is the main metaphor used in the job-related part of the game: It has a name, a description, a level of effort and a certain amount of gain. A predefined number of players are needed 2
Levels are a numerical classification for players based on their reached sum of XPs. A higher level implies more XPs.
to complete the project successfully. A player registers in the project as a co-worker, and after a fixed amount of time the project is finished and the player can collect his reward. If all co-workers claim their reward in time, an extra bonus is issued. Materials and Tools If a player joins a project, he or she can add materials and tools to increase the possible gain of the project. He receives these objects, which are subject to wear (and are thus “consumables”), as a special bonus for completing projects. A well-considered use of materials and tools means that their efficiency can be optimized. Department A player can join a department – this is a hierarchical organisation of players. They consist of an owner, several leaders and common members. Departments compete in achieving most XPs with their projects – thus each department has the same goal as each individual player has individually. Departments are the location where most of the communication takes place: since usually players belong to a department for a long time, they end up knowing each other very well. Skill Virtual skills improve the gain for the player. These can be trained in stages in exchange for time and coins. The player has to make a decision, which skill he or she wants to improve first and take the action accordingly. Award Fliplife awards special achievements with prizes. The prerequisites for receiving the awards are published so players can explicitly strive for specific awards. Energy Taking part in projects costs energy, which is a time-based renewable resource. The scarcity of energy forces the player to consider his or her usage of it carefully.
Collectable data A main assumption is that we have access to all the data which can be generated implicitly by the game play in Fliplife. Thus the recruitment process does not require conscious participation of players necessary, as would be the case, for example, in the completion of online surveys. The players would be observed without their knowledge. We consider all the data which can be collected, regardless if it is collected in reality: It must be emphasized that we have no insights into the actual technical implementation of the game. We differentiate three classes of collectable data: Player statistics and achievements This data is related to the game mechanics and the goals of the game. Table 1 lists some examples – only a few among many – showing that the game offers a rich set of measurements of the player’s behaviour and performance. The column “Potential indicator of” hints at the potential importance of this data. The named abilities do not follow a categorization within the game, but demonstrate the linkage between game play and real world skills (Steinkuehler & Duncan 2008). Data
Description
Potential indicator of
XPs
Main measurement of progress in the game.
Goal-orientation
Awards
Awards reward special player performances.
Status awareness, goalorientation
Skill levels
Virtual skills improve the ability to progress in the game.
Strategic thinking
Success Rate
Percentage of projects collected in time
Reliability, organisational capacity
Department state
Is the player member of a department and which role does he take?
Capacity for teamwork, leadership
Table 1: Player statistics as potential indicators of real-life player capabilities Behavioural data This data is derived from the behaviour of the players as it can be observed by the game without any additional data-recording facilities – just by the software of the game as it is. Data
Description
Potential indicator for
Online time
Login times can be used for creation of an online/offline profile. How long is the player online?
Efficiency of game play, discipline, organisational capacity, time management
Number of cancelled projects
How many projects does the player abandon before it starts?
Decision-making ability
Number of consecutive login-days
Is there a pattern of logins?
Reliability, endurance, patience
Number of page hits prior to a decision
Which data does the player look at before he makes a decision, e.g. whether to take part in a project?
Good judgement
Order of skill development
In which order does the player improve his skills?
Goal-orientation, strategic thinking, decision behaviour
Table 2: Examples of behavioural data Chat protocols Fliplife offers a chat and messaging component. Players can communicate via department chat and project chat and also by e-mail-like messages. A substantial part of the chat communication consists of social banter and serious real-life problem discussions. In this sense Fliplife and especially the departments have the function of a third place . This makes the game attractive as a recruitment tool: Being a third place the game offers the possibility to observe potential candidates in informal contexts – unbiased by the pressure of an exam situation. A difficulty is that assessment of chat protocols cannot be automated completely. The result of this activity was a table of game-collectable data types with preliminary assignments of measurable skills (Examples in Table 1 and Table 2).
Data analysis Player statistics and achievements as well as behavioural data can widely be analyzed automatically, because data is collected in a semantic context, e.g. if a project is started and finished these events are registered by the game as such, as semantic actions. We differentiate between stored attributes and derived attributes. Stored attributes are saved in the database of the game. It requires only minimal effort to select data records with certain values of these attributes (“filtering”). Examples for this kind of attributes are level, department membership and department role. Derived attributes, on the other hand, must be calculated, which is not always a trivial task and which may need sophisticated algorithms and a greater effort. An example is the calculation of an order of skill training. After this calculation, the results have to be evaluated using an appropriate strategy. Chat data is stored without any semantics. Automatic interpretation of this data would require methods from computer linguistics (Jurafsky & Martin 2008). Anderka et al.(2011a, 2011b) offer a heuristic approach to derive attributes from text chunks which are connected with a certain probability to text characteristics (here: technical quality of Wikipedia articles). Thus it seems possible to evaluate
chat protocols based on text attributes as length, number of words, and categories of used words. These automatic evaluations would require that a statistical sample be manually taken and evaluated beforehand: Chat texts are categorized (e.g. in categories as social banter, technical support, coordination of activities, directions), attributes are calculated and those attributes are identified which are the most suitable indicator for a certain category.
Skills Among the variety of skill classifications, we have adopted the categories of Diercks et al. (2005) and Gmürr & Thommen (2011). This classification was originally created for assessments of job candidates and can be used in online environments – these are the characteristics of our setting. Figure 1 shows a skill tree based on this classification. Not all of the skills are seen as measurable online: for example Motivation depends on context – the context of an online game differs from the context of a real life job. Technical expertise is also not expected to be measured by the game, because the game does not require technical knowledge.
Figure 1 Skill tree (cf. Diercks et al. 2005 and Gmürr & Thommen (2011))
A main resource of traditional recruitment processes has been job advertisements in newspapers and – only emerging in the last years – online job boards (HRM_Guide 2012). These advertisements list normally required skills for the offered positions. So we have chosen specific online job offerings and have extracted a set of required skills and competencies, based on the chosen classification we present above. For each offered position we ranked these skills according to their importance regarding the specified position. Then we took the eight topmost rated skills of each position. The measurements for these skills have been used – according to their ranking order – to determine a list of candidates for each position.
Mapping required skills and available data: Finding valid measurements Having identified relevant skills and types of Fliplife-generated data we needed to assign to each skill
those data types which are a measurement for that skill. This mapping is demonstrated in the following section by the examples of Leadership and Organisational Capacity.
Measurement of Leadership Attributes •
Department role: Leader and Owner
•
Department founders: Who took the initiative?
•
Initiator of department skill training: Who decides which skills get trained?
•
Invitation leader for further department members
•
Donations: Who donates coins and thus cares for the development of the department?
Derived attributes •
Number of initiated department projects
•
Quantity of chat entries o
with the department interns providing mentorship
o
with department members (Providing leadership through communication)
o
with other players
•
Number of removed members, number of invitations: active department management: are underperformers removed from the department?
•
Average project gain, maximum project gain: Which projects are completed in the department? Is there a project strategy? (Note: There are some techniques in high-gain projects which require players to delay rewards)
•
Feeling of fellowship: average length of stay of members in a department; change of career and department in groups
Chat analysis •
Reaction to failed projects
•
Communication with underperformers
Measurement of Organisational Capacity Attributes •
Success rate of projects: Who is able to plan and collects the reward for the project he joins in time?
•
Reaching higher levels of the game: Since it becomes harder and harder to level up, players who reach the highest levels are required to play very efficiently
Derived attributes •
•
How efficiently does the player use the login times? o
Analysis of the login schedule: Is there a pattern of login times?
o
Relation between login time and game progress (i.e. XPs)
Functional tasks in common projects, e.g. collection of materials for high gain projects
•
Number of materials and tools received by bartering
•
Real efficiency of materials and tools
Chat analysis •
Sending messages with links to projects to acquire co-workers.
•
Organisation of projects
Application of the selection algorithm Having determined measurements for the required skills of a certain position, our toolkit is complete to allow us to filter a list of candidates for this position from a population of Fliplife players. Figure 2 demonstrates an algorithm for the position of a construction manager. As already pointed out, not all steps are completely automatable (e.g. deep chat analysis). To reduce time and effort, we decided to first reduce the number of candidates by applying the automated selection steps and then sift through their chat logs manually. This is called automated preprocessing and manual post-processing.
Figure 2: Algorithm Example (Construction Manager)
Restrictions Our approach seems technically feasible. Nevertheless there are some restrictions which have to be eliminated before it can be put into practice. Pre-selection bias: Every game attracts not all, but certain types of players (Bartle 1996; Cummings 2011). There is the high probability that the community of players is not a valid representation of society as a whole. Candidates for certain kinds of positions therefore may be heavily underrepresented in this game, thus this game may not be a good recruitment source for certain types of positions. There seems to be no solution to this problem, only a consequence: one must accept that not all positions can be staffed using this game. Further research could help to identify the types of player and those positions which can be filled by them for which this game would be appropriate. Privacy protection and ethics The use of game data linked to a person will raise problems of privacy protection. The observation of players without their knowledge must also be rejected for ethical reasons. The opportunity seen in the approach suggested in this article is that explicit player knowledge about observation of their play would bias their behaviour and would remove the un-self-
consciousness of the informal game context. Therefore this approach is not realizable in the way we have described. A way to overcome these restrictions may be the employment of a trusted and neutral third party which would hold the link between the game account and the real identity of a player. If an employer were interested in contacting a given player, it would ask the game supplier to ask the player for permission to be contacted by the employer, a step which would only be completed if the player accepted the request. Demanding automation algorithms The algorithms which are needed to determine derived attributes can turn out to be very complex. This problem could be faced by saving additional data during game-play. Also for an automated, resilient chat log analysis, further research may be necessary. Loose coupling to real-world tasks The game mechanics of Fliplife are not tightly connected to the tasks which might be needed in a real-life job. This stands in contrast to some research topics, e.g. the concept of Epistemic Games, which are based explicitly on a tight integration of game-related tasks and real-world usable learning goals (Shaffer 2005). This could lead to the question of how the data of this (generic) game can be a valid measurement for the ability of players to fulfil a certain job. To deal with this restriction, it first must be said that the suggested approach only produces a list of candidates. A further review of the candidates must be done by humans in job interviews so that mistakes in the e-assessment procedure can be recognized and corrected. Furthermore, the existing video game can be amended step-by-step – since it is a browser-based game - by game mechanics which simulate real-world procedures more closely. A third point is that the game already currently measures skills which are not specific to certain tasks but are needed in general to meet job requirements (Steinkuehler & Duncan 2008). This may be already a good foundation for making hiring decisions. Costs Although producing social online games is cheap compared to traditional games, the effort needed to create a video game must not be neglected. Used in recruitment processes, the costs of such games have to compete with other tools and resources of recruitment, unless they possess some unique characteristic. One such characteristic could be the new channel of recruitment with access to a broad group of candidates who play these games. Regarding cost, advancing technology could allow for advances such as powerful frameworks and easy customizations: Game development would become more affordable.
Conclusion Advantages of e-assessment include automation of selection process, lowering the work load on personnel, independence of time and location, and acceleration of recruitment processes. Using voluntarily played video games allows for an authentic view of candidates, because they are observed in informal contexts, i.e. without having the pressure of an exam. In this thought experiment we documented a way how the social online game Fliplife could be used as an e-assessment tool. This tool allows for the automated generation of a list of candidates for certain positions. At the same time we identified some limitations of such an approach. Among them are pre-selection bias and privacy protection, as well as ethical concerns. This case study demonstrated that the use of less costly and high accessible social games in recruitment processes is possible. Further work may contribute to lowering the impacts of the aforementioned restrictions.
References Anderka, M., Stein, B. & Lipka, N., 2011a. Detection of text quality flaws as a one-class classification problem. In Proceedings of the 20th ACM international conference on Information and knowledge management. New York, NY, USA: ACM, pp. 2313-2316. Anderka, M., Stein, B. & Lipka, N., 2011b. Towards automatic quality assurance in Wikipedia. In Proceedings of the 20th international conference companion on World wide web. New York, NY, USA: ACM, pp. 5-6. Bartle, R.A., 1996. Hearts, Clubs, Diamonds, Spades: Players Who Suit MUDs. The Journal of Virtual Environments, 1(1). Cummings, J., 2011. To Bartle or Not to Bartle?: Capturing Player Behavior in Social Games | Motivate. Play. Motivate Play. Available at: http://www.motivateplay.com/2011/06/tobartle-or-not-to-bartle-capturing-player-behavior-in-social-games/ [Accessed June 4, 2012]. Diercks, J., Hansen, M. & Weber, A., 2005. ONLINE ASSESSMENT CENTER - EINE NEUE OPTION FÜR HIGH POTENTIALS. In S. Albers, V. Haßmann, & T. Tomczak, eds. Digitale Fachbibliothek “Verkauf.”Symposion Publishing, p. 5965. Fliplife, 2012. Fliplife. Available at: http://fliplife.com/ [Accessed April 26, 2012]. Fliplife.com, 2012. Was ist Fliplife? Fliplife for Business. Available at: http://business.fliplife.com/lerne-fliplife-kennen/was-ist-fliplife/ [Accessed May 21, 2012]. Gmürr, M. & Thommen, J.-P., 2011. Human Resource Management 3., überar., Versus. Golembowski, K., 2002. Reichweiten und Grenzen von e-Recruitment: Eine kritische Analyse unter besonderer Berücksichtigung von eignungsdiagnostischen Online-Verfahren und deren Akzeptanz am Markt (German Edition), Diplomica Verlag. HRM_Guide, 2012. Recruitment Sources. SimpleHRGuide.com. Available at: http://www.simplehrguide.com/recruitment-sources.html [Accessed May 18, 2012]. Hagmann, C. & Hagmann, J., 2011. Assessment Center, Haufe-Lexware. Jurafsky, D. & Martin, J.H., 2008. Speech and Language Processing, Prentice Hall International. Keller, M., 2009. Fachlexikon für das Human Resource Management, Praxium; 1. Auflage. Kupka, K., 2009. E-Assessment : Entwicklung und Güteprüfung von zwei internetbasierten Simulationsverfahren zur Messung der Planungs- und Problemlöseleistung von zukünftigen (pädagogischen) Führungskräften. Sozialarbeit / Sozialpädagogik. Lippold, S., 2010. Systematische Personalauswahl - Bestandsaufnahme und innovative Ansätze zur Weiterentwicklung (German Edition), GRIN Verlag.
Meyer, M., 2011. Per Spiel zum Traumjob? FORUM - Das Wochenmagazin. Available at: http://www.magazin-forum.de/per-spiel-zum-traumjob/ [Accessed May 21, 2012]. Müller, N. et al., 2012. FlipLife als Mitarbeiterrekrutierungsquelle. Bauhaus-Universität Weimar. Oldenburg, R., 1999. The Great Good Place: Cafes, Coffee Shops, Bookstores, Bars, Hair Salons, and Other Hangouts at the Heart of a Community, Da Capo Press. Shaffer, D.W., 2005. Epistemic games. International Journal of Educational Research, 1(6), p.49-61. Steiner, H., 2009. Online-Assessment: Grundlagen und Anwendung von Online-Tests in der Unternehmenspraxis (German Edition), Springer. Steinkuehler, C.A. & Duncan, S., 2008. Scientific Habits of Mind in Virtual Worlds. Journal of Science Education and Technology, 17(6), p.530-543. Steinkuehler, C.A. & Williams, D., 2006. Where Everybody Knows Your (Screen) Name: Online Games as “Third Places.” Journal of Computer-Mediated Communication, 11(4), p.885-909.
