2011 IEEE International Conference on Privacy, Security, Risk, and Trust, and IEEE International Conference on Social Computing
An Exploratory Study of Player and Team Performance in Multiplayer First-Person-Shooter Games Kyong Jin Shim Department of Computer Science & Engineering University of Minnesota Minneapolis, MN, USA
[email protected]
Kuo-Wei Hsu Department of Computer Science National Chengchi University Taipei City, Taiwan
[email protected]
Samarth Damania Breck School 123 Ottawa Avenue North Minneapolis, MN, USA
[email protected]
Colin DeLong, Jaideep Srivastava Department of Computer Science & Engineering University of Minnesota Minneapolis, MN, USA {delong, srivasta}@cs.umn.edu from Bungie Corporation. The HaloFit project analyzes data collected during professional gaming tournaments in order to understand behaviors of players and teams. Specific questions being studied include refinement of skill, performance assessment, team dynamics, and assembly of highperformance teams through the use of data mining. We use the database from the HaloFit project to gather relevant player and team statistics from the 2008 and 2009 seasons for professional Halo 3 games to determine impact of change in team composition on player and team performance. Numerous prior studies [8-11] in social sciences and management have investigated how team composition and changes in team composition can affect team performance. However, little is understood about combat player and team performance and factors contributing to it in competitive shooter games. The specific goals of this study are: 1) to determine how variability in the number of different teams on which a player plays affects player performance, and 2) to determine how variability in team composition throughout a season affects team performance. The rest of this paper is organized as follows: Section II introduces the dataset. Sections III and IV discuss player and team performance, respectively. Section V discusses player performance and team changing. Sections VI and VII discuss player and team performance prediction, respectively. Section VIII concludes this paper.
Abstract— In this paper, we report findings from an exploratory study of player and team performance in Halo 3, a popular FirstPerson-Shooter game developed by Bungie. In the study, we first analyze player and team statistics obtained from the 2008 and 2009 seasons for professional Halo 3 games in order to investigate the impact of change in team composition on player and team performance. We then examine the impact of past performance on future performance of players and teams. Performing a largescale experiment on a real-world dataset, we observe that player and team performance can be predicted with fairly high accuracy and that information about change in team composition can further improve the prediction results. Keywords: player performance, team performance, team changing, team composition, video games, multiplayer games, first person shooter
I.
INTRODUCTION
In recent years, a significant rise in computing power and storage devices has generated enormous interest in data-driven approaches to studying all kinds of phenomena. Understanding and explaining social behavior can fundamentally alter the functioning of professional organizations, battlefield scenarios, and multi-player professional teams. The rapid advancement of data-driven techniques, combined with greater availability of data and greater capabilities in data storage have now allowed computers to better analyze and model social phenomena. This study analyzes player and team performance in professional Halo 3 1gaming. The study uses data from the Halo 3 professional leagues [4]. This study is an extension of the HaloFit [1] project and also an extended study of player and team performance in massively multiplayer online role-playing games [5-7]. The HaloFit project aims to understand various aspects of player and team performance in professional Halo 3 games, whose game data were collected 1
II.
A Halo 3 team is composed of four players and is differentiated as "professional" or "amateur". Teams play in up to five tournaments throughout a one-year Major League Gaming (MLG) season. Within each tournament, a team plays a best-of-five or best-of-seven games in a number of series during a gaming season. At any point in the seven or eight series held in a tournament, teams that win advance to a
www.halo3.com
978-0-7695-4578-3/11 $26.00 © 2011 IEEE DOI
DATASET
617
activity as “team changing”. We examine how team changing affects player performance. One or more tournaments can occur in a given season, hence, we count the number of distinct teams and compute the average across all tournaments in each season. Next, we perform linear regression on the average number of distinct teams against player K/D ratio. There is a slight decrease in player performance with an increase in team changing for both 2008 (R2 = 0.2722) and 2009 (R2 = 0.2133) tournaments. Also, as team changing increased, player K/D ratio was generally more varied. The team changing has a slightly negative correlation with player performance, in a given gaming season.
"winner’s bracket" while teams that lose move to the "loser’s bracket". Teams that lose in the "loser’s bracket" are out of the tournament. Team rank is calculated by cumulative performance of the team in five tournaments held in various locations in the United States and Canada. Teams that place 116 are ranked as professional while teams that place 17 and below are ranked as amateur [4]. We extracted data for professional teams for all five tournaments held in the 2008 and 2009 seasons [4]. This study examines 122 players and 24 teams over 6 tournaments in 2008, 5 tournaments in 2009, 230 series in 2008 and 216 series in 2009. III.
PLAYER PERFORMANCE
B. Team Performance Next, we examine how change in team composition can affect team performance. We say that a “change in team composition” occurred for a given team, when one or more players get replaced by new players from one game to another in a given time unit (i.e. season, series, tournament). For each team, we count the number of distinct players it has had. Because one or more tournaments can occur in a given season, we compute the average across all tournaments in each season. Next, we perform linear regression on the average number of distinct players against: 1) Team K/D ratio, 2) Team Win/Loss ratio. There is a slight decrease in team K/D ratio as the change in team composition increased for 2008 (R² = 0.1851) and 2009 (R² = 0.2034) tournaments. Also, there is a moderate decrease in team Win/Loss ratio as the change in team composition increased for 2008 (R² = 0.4632) and 2009 (R² = 0.4496) tournaments. One interesting finding is that the impact of “change” (i.e. players joining and moving on to different teams) is greater for teams than it is for players, as shown in our regression analyses. While there is a slight decrease in player performance for those players that move from one team to another in a given gaming season, it is noted that the teams left behind (and having to find one or more new players to fill in) experience greater negative impact in terms of Win/Loss ratio. The results from this analysis also show that the change in team composition has a greater (negative) impact on team’s Win/Loss ratio than it has for the team’s K/D ratio. As discussed in Section IV, team K/D ratio is an aggregate of all kills and deaths sustained by all team members. This is a particularly interesting finding given the heavily team coordination-oriented nature of this game that suggests that when it comes to winning or losing a game, there is much more to it than just performances of individual players put together. Given these interesting findings, in the next two sections, we set out to build predictive models for player performance and team performance.
A. Kill/Death/Assist We measure player performance in terms of number of kills (killing players from the opponent team), deaths (dying as a result of getting shot by one or more players from the opponent team), and assists (assisting one or more teammates). First, for each player, we track the total number of kills, deaths, assists occurring in each game he/she participates in, and then we compute the average across all games. Second, we compute the percentage of each of these metrics that each player is responsible for across the team. For instance, if Player X made 15 kills out of the total 60 kills performed by his team, his “kill” participation ratio is 0.25. We perform the same for the number of deaths and assists. Next, we compute the average across all games. This set of metrics can provide information indicative of what role a given player played in team plays (i.e. main slayer, support, objective). B. Kill/Death (K/D) Ratio This performance metric measures the overall number of kills divided by deaths. The higher the number, the more successful a given player is at shooting and killing. IV.
TEAM PERFORMANCE
A. Kill/Death/Assist Similarly to Player Performance, we compute the total number of kills, deaths, and assists generated by each team in each game. We then take the average across all games. B. Kill/Death (K/D) Ratio This performance metric computes total number of kills divided by total number of deaths, generated by each team. We then take the average across all games. C. Win/Loss (W/L) Ratio Another team performance metric used here is Win/Loss ratio, which computes the total number of game wins divided by the total number of losses, generated by each team. V.
PLAYER PERFORMANCE AND TEAM CHANGING
A. Player Performance In a given time unit (i.e. season, series, tournament), when a player switches from one team to another, we refer to this
VI.
PLAYER PERFORMANCE PREDICTION
This experiment investigates whether player’s past performance is indicative of his/her future performance. We
618
contributes to the prediction of player K/D ratio in most of the algorithms even though it is minimally contributing. In Tables 3 and 4, due to space limitation, we show in parentheses the prediction F-measure in the case of excluding player “team changing” information in the feature representation. Table 3 – Prediction of Kill/Death/Assist Ratios F-measure Kill Death Assist Algorithm Ratio Ratio Ratio 88.6 88.0 89.7 Bayesian Network (84.2) (85.0) (84.8) 88.3 87.8 93.5 SVM (Radial) (81.3) (77.2) (82.7) 87.3 84.0 85.6 Neural Network (66.7) (64.9) (69.9) 97.1 97.7 96.9 JRip (96.0) (96.6) (96.7) 98.4 97.9 98.4 J48 (97.6) (97.5) (97.3) 98.3 97.7 98.3 Random Tree (97.4) (98.2) (97.4) 96.6 96.7 96.6 REP Tree (95.4) (96.0) (95.8)
define performance in the year 2008 as “past” and performance in the year 2009 as “future”. Suppose that for an upcoming season, a team is looking to acquire a new player to join. What information about players can a team leverage to decide which players might be the top picks? This experiment seeks to build predictive models which take into consideration individual level in-game statistics (i.e. kills, deaths, assists) as well as player’s past team changing behavior. In this exploratory study, we categorize players based on their performance (i.e. K/D ratio, etc.) into four buckets: top 25% performing players, the next 25% best performing players and so on. The reasoning behind this approach is that given a large pool of available players (seeking to join a team), a good starting approach might be to narrow down the search space by looking at, for instance, the top 25% best performing players. Given this reasoning, we formulate this player performance prediction problem as a four-class classification problem. We take each player performance metric from Section III, and compute mean (μ) and standard deviation (ı). The four classes are generated in the following manner: a) Class 1 μ + 1ı, b) μ < Class 2 < μ + 1ı, c) μ - 1ı < Class 3 < μ, d) Class 4 μ - 1ı. In each case, all of the other player performance metrics not selected to be the dependent variable are considered to be the independent variables. We use Weka [2] to run various classification algorithms. We use standard 10-fold crossvalidation. The SMOTE algorithm [3], a widely employed technique for imbalanced classification, is used to over-sample the comparatively rare classes. Table 2 – Prediction of K/D Ratio F-measure All except team Algorithm All changing Bayesian Network 88.3 83.0 SVM (Radial) 87.0 78.5 Neural Network 86.9 72.5 JRip 97.2 96.0 J48 97.4 97.3 Random Tree 97.6 97.6 REP Tree 96.0 96.3 Table 2 shows the prediction results where player’s K/D ratio is the dependent variable. Our first finding is that player K/D ratio can be accurately predicted using several of these algorithms, most notably J48 algorithm. While it is important to be able to predict with a high level of confidence, from certain perspectives, it is equally (if not less) important to be able to interpret the prediction results and be able to provide contextual explanation. The advantage of J48 is that the resulting tree can be further pruned and meaningful rules can be extracted. Which independent variables and their value ranges best predict which performance bucket can be a valuable piece of information for analyzing player performance. The present exploratory study focuses on constructing predictive models with a goal of helping narrow down the search space of game players (for the reason stated earlier in this section), whilst a future extension includes an investigation of this contextual understanding. Our second finding is that player’s “team changing” information positively
Table 4 – Prediction of Kill/Death/Assist Counts F-measure Kill Death Assist Algorithm Count Count Count 89.1 89.4 84.3 Bayesian Network (84.9) (86.7) (81.7) 92.8 91.4 90.4 SVM (Radial) (79.8) (81.6) (82.0) 86.6 91.2 82.2 Neural Network (65.6) (80.8) (66.4) 96.5 97.2 96.4 JRip (95.3) (96.4) (95.3) 98.4 97.5 96.6 J48 (97.1) (97.6) (95.8) 98.2 97.6 97.3 Random Tree (97.5) (97.7) (96.8) 97.1 95.8 95.4 REP Tree (95.6) (95.7) (93.6) Tables 3 and 4 show findings similar to those found in Table 2. Kill/death/assist ratio as well as counts can be highly accurately predicted using several of the algorithms. Although the impact is minimal, including the “team changing” information helps the prediction by slightly increasing the prediction f-measure. In summary, we report that various player performance metrics for Halo 3 players can be highly accurately predicted using several algorithms. We also report that when predicting player performance, including players’ past “team changing” behavior can further improve the prediction results. Lastly, a few of the several best performing predictive algorithms yield prediction results in a form of tree or rule set, based on which a future extension of this exploratory study will attempt to attain contextual understanding of exactly which of the various player performance metrics and their value ranges map to high or low performance in the future.
619
VII.
change in team composition on team performance. Our linear regression analyses show that the impact of “change” is greater for teams than it is for players. When a player switches from one team to another, the performance reduction is less than that sustained by teams left behind. While the teams experiencing change in team composition sustain performance reduction in terms of K/D ratio that is similar to the performance reduction experienced by players (switching teams), the team’s Win/Loss ratio is more negatively impacted. This suggests that in this heavily team coordinationbased first-person-shooter game, for a team to win, it takes more than just individual-level player performance aggregated together.
TEAM PERFORMANCE PREDICTION
This experiment investigates whether team’s past performance is indicative of its future performance. We define performance in the year 2008 as “past” and performance in the year 2009 as “future”. Suppose that a player is looking to join a team. What information can the player leverage in order to decide which team to join? This experiment seeks to build predictive models which take into consideration team’s in-game statistics as well as its past change in “team composition”. In this exploratory study, we categorize teams based on their performance (i.e. K/D ratio, Win/Loss ratio) into four buckets: top 25% performing teams, the next 25% best performing teams and so on. The reasoning behind this approach is that given a large pool of available teams (seeking to recruit players), a good starting approach might be to narrow down the search space by looking at, for instance, the top 25% best performing teams. Given this reasoning, we formulate this team performance prediction problem as a four-class classification problem. We take each team performance metric from Section IV and create four classes as discussed in Section VI. In Table 5, due to space limitation, we show in parentheses the prediction F-measure in the case of excluding information about change in “team composition” in the feature representation. Table 5 – Prediction of K/D and Win/Loss Ratios F-measure Algorithm K/D Ratio Win/Loss Ratio Bayesian Network 98.6 (91.5) 95.6 (89.1) SVM (Radial) 97.2 (81.7) 89.7 (75.7) Neural Network 99.5 (81.2) 98.6 (81.1) JRip 99.4 (97.1) 98.0 (93.1) J48 99.7 (97.5) 98.8 (93.5) Random Tree 99.6 (97.3) 97.7 (94.1) REP Tree 99.6 (97.2) 97.1 (93.6) Table 5 shows the prediction results where team K/D ratio and Win/Loss ratio, respectively, is the dependent variable. In both cases, our first finding is that team performance can be accurately predicted using several of these algorithms. Especially, some of the rule-based algorithms and tree-based algorithms perform very well, which sets a good foundation for our future direction of exploring the underlying contextual findings as to what attributes to good team performance. Our second finding is that the information about past change in “team composition” positively contributes to the prediction of future team performance. The f-measure difference between when including the information and when excluding it is much greater across most of the algorithms, in comparison to the prediction results from Section VI (Player Performance Prediction). The f-measure difference is greater in the case of Win/Loss ratio prediction, and this finding aligns well with the findings we earlier reported in Section V – B based on linear regression analysis. VIII.
ACKNOWLEDGMENTS We would like to thank the Data Analysis and Management Research group at the University of Minnesota, as well as the reviewers, for their feedback and suggestions. We would also like to thank Major League Gaming for making their 20082009 tournament data available. We would also like to thank anonymous reviewers for their constructive feedback. REFERENCES [1]
DeLong, C., Erickson, K., Perrino, E., Shim, K., Pathak, N. (2009). Project HaloFit Data Repository [http://halofit.org/resources.php#datasets]. Minneapolis, MN: University of Minnesota, Department of Computer Science. [2] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H. Witten (2009); The WEKA Data Mining Software: An Update; SIGKDD Explorations, Volume 11, Issue 1. [3] Wang, J., Xu, M., Wang, H., Zhang, J. "Classification of Imbalanced Data by Using the SMOTE Algorithm and Locally Linear Embedding," In Proceedings of the 8th international Conference on Signal Processing (2006). [4] "Halo 3 (4v4 Competition Format)." MLGPro.com. Major League Gaming. Web. July 2010. [5] Shim, K. J., Ahmad, M., Pathak, N., Srivastava, J., "Inferring Player Rating from Performance Data in Massively Multiplayer Online RolePlaying Games (MMORPGs)," cse, vol. 4, pp.1199-1204, 2009 International Conference on Computational Science and Engineering, 2009. [6] Shim, K. J., Sharan, R., Srivastava, J., "Player Performance Prediction in Massively Multiplayer Online Role-Playing Games (MMORPGs)," 14th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2010. Hyderabad, India, June 21 - June 24, 2010, Proceedings. [7] Shim, K. J., Srivastava, J. "Team Performance Prediction in Massively Multiplayer Online Role-Playing Games (MMORPGs)," Proceedings of the IEEE Social Computing (SocialCom-10). Minneapolis, Minnesota, USA, August 20-22, 2010. [8] Hellerstedt, K., Aldrich, H. E. "The impact of initial team composition and performance on team dynamics and survival," Academy of Management Proceedings (2008), Issue: 1984, Publisher: Academy of Management, Pages: 6. [9] Trower, J. K., Moore, K. K. "A study of the impact of individual goals and team composition variables on team performance, " in Proceedings of the 1996 ACM SIGCPRSIGMIS conference on Computer personnel research SIGCPR 96 (1996). [10] Autrey, R. L. (2005). Team Synergy , Team Composition and Performance Measures. [11] Spotts, Harlan E "Evaluating the Effects of Team Composition and Performance Environment on Team Performance". Journal of Behavioral and Applied Management. FindArticles.com. 29 Mar, 2011. http://findarticles.com/p/articles/mi_qa5335/is_200501/ai_n21364013/
CONCLUSIONS
In this exploratory study of player performance and team performance in professional Halo 3 game, we investigate the impact of 1) team changing on player performance and 2)
620
