movement primitives to locations in the game environment. Experimental results ... machine learning to produce life-like artificial characters. On the other hand, ...
Synthesizing Movements for Computer Game Characters Christian Thurau, Christian Bauckhage, and Gerhard Sagerer Faculty of Technology, Applied Computer Science Bielefeld University, P.O. Box 100131, 33501 Bielefeld, Germany cthurau,cbauckha,sagerer @techfak.uni-bielefeld.de
Abstract. Recent findings in biological neuroscience suggest that the brain learns body movements as sequences of motor primitives. Simultaneously, this principle is gaining popularity in robotics, computer graphics and computer vision: movement primitives were successfully applied to robotic control tasks as well as to render or to recognize human behavior. In this paper, we demonstrate that movement primitives can also be applied to the problem of implementing lifelike computer game characters. We present an approach to behavior modeling and learning that integrates several pattern recognition and machine learning techniques: trained with data from recorded multiplayer computer games, neural gas networks learn topological representation of virtual worlds; PCA is used to identify elementary movements the human players repeatedly executed during a match and complex behaviors are represented as probability functions mapping movement primitives to locations in the game environment. Experimental results underline that this framework produces game characters with humanlike skills.
1 Motivation and Overview Computer games have become an enormous business; just recently, its annual sales figures even surpassed those of the global film industry [3]. While it seems fair to say that this success boosted developments in fields like computer graphics and networking, commercial game programming and modern artificial intelligence or pattern recognition hardly influenced each other. However, this situation is about to change. On the one hand, the game industry is beginning to fathom the potential of pattern recognition and machine learning to produce life-like artificial characters. On the other hand, the AI and pattern recognition communities and even roboticists discover computer games as a testbed in behavior learning and action recognition (cf. e.g. [1, 2, 10, 13]). This paper belongs to the latter category. Following an idea discussed in [2], we report on analyzing the network traffic of multiplayer games in order to realize game agents that show human-like movement skills. From a computer game perspective, this is an interesting problem because many games require the player to navigate through virtual worlds (also called maps). Practical experience shows that skilled human players do this more efficiently than their computer controlled counterparts. They make use of shortcuts or perform movements which artificial agents cannot perform simply because their programmers did not think of it1 . 1
An example is the rocket jump in ID Software’s (in)famous game Quake II. Shortly after the game was released, players discovered that they can jump higher if they make use of the recoil
An intuitive idea to address this problem is to identify elementary building blocks of movements and to learn how they have to be sequenced in order to produce the desired complex movement behavior. In fact, as recent results from biological neuroscience suggest, this seems to be the way the human brain constructs body movements [7, 16]. This observation is substantiated by psychological experiments on imitation learning which suggest that infants devote much of their time to the learning of elementary limb movements and of how to combine them to reach a certain goal [14]. Not surprisingly, movement primitives are thus becoming popular in robotics and computer vision, too. Schaal et al. [15] describe how nonlinear differential equations may be used as dynamic movement primitives in skill learning for humanoid robots. Given motion sensor data, Mataric et al. [4] apply PCA and linearly superimpose resulting elementary movements to reproduce arm movements demonstrated by human subjects. Using spatio-temporal morphable models, Ilg and Giese [9] and Giese et al. [8] linearly combine primitive movements extracted from data recorded with motion capture systems to produce high quality renderings of karate moves and facial expressions, respectively. Galata et al. [6] apply variable length Markov models trained with vectors describing elementary 2D or 3D motions to synthesize or recognize complex activities of humans. However, the preconditions for moving a character through a virtual world differ from the ones in the cited contributions. While these focus on movements of limbs or facial muscles and neglect the environment, a game character moves as a whole and appropriate movement sequences will depend on its current location and surroundings. For instance, movements leading the agent into a virtual perils like abysses or seas of lava would be fatal and must be avoided. We are thus in need of a methodology that except for elementary moves also learns a representation of the environment and generates movement sequences with respect to the current spatial context. As the next section will show, neural gases are well suited to learn topological representations of game environments. In section 3, we shall discuss the extraction of movement primitives from network traffic of multiplayer games and how to relate them to the neural gas representation of the environment. Section 4 will present experiments carried out within this framework and a conclusion will close this contribution.
2 Neural Gas for Learning Topological Representations For a representation of the virtual world we are learning a topology using a Neural Gas [12] algorithm, which is a cluster algorithm showing good performance when it comes to topology learning [11]. The training data used for Neural Gas learning consists of all locations a human player visited during various plan executions, thus staying very close to the actual human movement paths. Application of a Neural Gas algorithm to the player’s positions results in a number of prototypical positions, which are interconnected only by the player’s actions. Thereby non-reachable (or at times unimportant) world positions are excluded in advance (in theory any point in the gaming world could be reached by using game exploits such as the rocket jump,
of a fired rocket. I.e. if they fired a rocket to the ground immediately after they jumped, their avatar was able to reach heights never planned by the programmers.
0.05 0.04 0.03 0.02 0.25
0.01 0.2 0 0.15
0.15 0.1
0.1 0.05
0.05 0
0 −0.05
−0.05 −0.1
−0.1
Fig. 1. A simple map and its corresponding topological representation
therefore topology learning reveals a more accurate discretization of the 3D gaming world). Since a sufficient number of cluster centres varies among maps, we were also using the Growing Neural Gas [5] to determine the number of cluster centres to reach a given error value. Thereby we can choose proportional number of cluster centres for each map, once a suitable error value is found. The experiments were carried out using an mean squared error value of and resulted in to cluster centres for larger maps, and about to centres for smaller maps. For our approach the interconnections between nodes are not needed, therefore we can safely skip the edge learning. Figure 1 shows a small map and its corresponding topological representation, edges were drawn for clarification reasons, they are of no further use in the presented approach. By assigning training samples of recorded player actions to cluster centres in the topological map, small sets of localized training samples are generated. Each separated training set defines the legal actions for a specific region, not only in the topological map, but also in the simulated 3D world. However, for a further investigation of the player’s behavior we first have to introduce movement and action primitives.
3 Movement Primitive Extraction Evidence form neuroscience indicates that complex movements in humans/animals are built up by combinations of simpler motor or movement primitives [7]. For a more lifelike appearance of computer game character motions, a biological approach utilizing movement primitives seems promising. To identify the underlying set of basic movements, PCA is applied to the training samples. A training sample set consists of a number of eight dimensional motion vectors
! "# $%& (')*+,- /.+)0)10,23 0/.4 0, 5
completely defining an action the player executed (a motion vector corresponds to a human player’s mouse and keyboard inputs). These motion vectors can be directly send to a server, no further mapping on agent motor commands is needed. The resulting eigenvectors provide the elementary movements, of which a motion vector can be constructed. No dimension reduction is applied at this stage, PCA is just used for computing an optimal representation. Thereby a projection of the observed motion vectors onto the eigenvectors can be viewed as a reconstruction of player movements by their movement primitives. Frequently executed player movements can be grouped to gain action primitives, which are of course built up using movement primitives. To acquire a set of action primitives, the training samples are projected onto the eigenmovement space, in which they are clustered using a k-means algorithm (similar to the primitive derivation in [4]). This results in a set of cluster centers, each of which representing a single action primitive. The number of action primitives does have an influence on the overall smoothness of the later motion sequences, we achieved good results by choosing to cluster centers. The right number of cluster centres depends on the number of training samples and on the variety of observed motions. However, even the representation of a large number of training samples could not be further improved by choosing a higher number of cluster centres. This indicates, that there might be a fixed number of action primitives, which guarantee a smooth execution of motion sequences. Sequentially executed action primitives lead to complex behaviors (in fact all human behaviors can be interpreted as sequences of action primitives). In order to generate human-like motions, the action primitives need to be executed in a convincing manner, based on the training set motion vectors. Since the actual player’s movement depends on the surrounding and his position on the map, the probability of executing a specific action primitive can be denoted as
(1)
where denotes a certain node in the topological map. The acquisition of the conditional probabilities is fairly easy, each movement vector can be assigned to a node in the topological representation and it can be assigned to an action primitive. Counting the evidences of action primitives in all nodes results in a matrix, where denotes the number of nodes in the topological map and denotes the number of action primi tives. A matrix entry at position ( denotes the probability of executing an action primitive for node number . However, in a sequence of action primitives, not every primitive can be executed as a successor of any primitive. On the one hand humans tend to move in a smooth way, at least compared to what would be possible for an artificial player, on the other hand humans are bound to physical limitations of their hand motion, besides, some player’s might have certain habits, they tend to jump for no reason or make other kinds of useless, yet very human movements. To reflect those aspects, the probability of executing a primitive as a successor of a primitive should be incorporated. It can be denoted as
(2)
Fig. 2. The artificial game character, observed while executing one of the experiment movements - a jump to an otherwise not reachable item
The probabilities can be extracted from the training samples by inspecting the observed action primitive sequence, resulting in a transition matrix, where denotes the number of action primitives. and are independent, Assuming the conditional probabilities the overall probability for the execution of a primitive can now be denoted as
(
(3)
More conditional probabilities, expressing a greater variety of dependencies, could be incorporated at this point. For example, action primitive selection based on an enemy player’s relative position or based on the current internal state of the player. In the presented approach we wanted to concentrate on movements for handling environmental difficulties, while still creating the impression of a human player, therefore ignoring further possibilities of the presented approach. When placed in the game world, the next action for the artifical game character is chosen randomly using a roulette wheel selection according to the .
4 Experiments To test the presented approach, we carried out a set of eight smaller experiments. Each experiment consisted of a separate training sample set, in which (at times) complicated movement sequences were executed several times by a human player. The observable motions varied from simple ground movements to more complex jump or shooting maneuvers (or combinations of both). In addition, a larger training sample set, this time a real match between two human players, was used. In all but one experiments the observed movement could be reproduced in a convincing manner. In one experiment the artifical player usually ended in front of wall and stopped moving. More training samples, containing hints for mastering the described situation, could have led to a smarter impression and a better recreation of the observed motion. Although staying very close to the training set movements, an exact reproduction almost never occurred because of the randomness in sequence generation. While this
0.24 0.178
0.235 0.23
0.176
0.225
0.174
0.22
0.172
0.215
0.17
0.21
0.168
0.205
0.166
0.2
0.164 0.3
0.195 0.3 0.28 0.26
0.35
0.24 0.22 0.2 0.18
0.42
0.4
0.38
0.36
0.34
0.32
0.3
0.28
0.26
0.24
0.4
−0.04
−0.02
0
0.02
0.04
0.06
0.08
0.1
Fig. 3. Comparison of the artificial player’s (blue) movement trajectories in 3D and the trajectories of a human player (red). The left picture shows a combination of different jumps to get to reach an item located on a platform. The right image shows a repeated jump through a window.
might seem like a disadvantage, it definitely adds to a more human-like impression. Figure 3 and Figure 4 show the trajectories of the artificial player compared to a human test player in 3D. Besides the basic reconstruction of certain motions, the artificial player’s motions themselves, his way of turning, jumping and running, were naturally looking - creating the illusion of a human player (this applies to it’s motion, not the tactical/strategic decisions). Even complicated actions, for example the famous rocket jump (an impossible maneuver for an inexperienced human player) could be learned and executed. The more realistic training set, two human players competing on a smaller map, finally resulted in a very good imitation of a broader repertoire of a human player’s motion, indicating, that our approach is suitable for larger scale problems. The strong coupling between the game characters position in the topological representation and the selection of motion primitives made the character appear smart, by acting in an appropriate way to the architecture of the 3D game world - jumping over cliffs or standing still when using an elevator. In addition the approach preserved certain player habits by (in one case) executing senseless jumps from time to time. Since no information about the enemy player was introduced during live play, some kind of shadow fighting could be observed, as if an enemy would be present. A further incorporation in already developed approaches [17] for strategic movement should be established. The human motion generation does not hinder the agent to occasionally walk in circles or do other kinds of human looking but nevertheless stupid movements, after all the approach pays attention to realistic motions, goal oriented movements were not intended.
5 Conclusion and Future Work In order to create life-like motion for computer game characters, we decided for a biologically inspired approach by using movement primitives as the basic building blocks
0.15
0.24
0.22 0.14
0.2 0.13
0.18
0.16
0.12
0.14 0.11
0.12 0.4 0.1 0.3
0.35 0.25
0.3 0.2
0.15
0.1
0.25 0.05
0.2
0.1 0.5 0.45 0.4 0.35 0.3
0.24
0.22
0.2
0.18
0.16
0.14
0.12
Fig. 4. Comparison of the artificial player’s (blue) movement trajectories in 3D and the trajectories of a human player (red). The experiment underlying the left image contained an elevator usage and a long jump to a platform. While the right image displays the trajectories extracted from the experimental results illustrated in Figure 2
of motion. PCA applied to observed human motions revealed the eigenmoves, or in our case movement primitives. Prototypical motion vectors were extracted using k-means clustering in the eigenspace of the used training samples. Finally conditional probabilities of the execution of a specific motion primitive were computed, dependent on the position in a topological map and on the last executed action primitive. When playing, the artificial player selects its next action based on the conditional probabilities, thus favoring more common sequences of action. And indeed, our experiments show a good performance by reproducing -even complicated- training movement sequences. The artificial player’s motion appears surprisingly realistic and preserves certain human habits, which of course adds to a life-like impression. Besides the topological map position, there might be other features on which motion primitive selection may depend. For example, an enemy player’s movement might be of importance. Integration of such conditional probabilities should be possible and could provide improvements. Besides a further development of the described approach, an integration in available approaches for strategical and reactive imitation of human players would be desirable. After all life-like motion for computer game characters is only one aspect for the imitation of a human player, though of great importance. It also is of interest, if our approach could be applied to other domains in the field of action recognition/simulation for humans as well as for animals. First ideas have already been discussed with biologists, who are investigating sequences in the courtship behavior of zebra finches.
6 Acknowledgments This work was supported by the German Research Foundation (DFG) within the graduate program “Strategies & Optimization of Behavior”.
References 1. E. Amir and P. Doyle. Adventure games: A challenge for cognitive robotics. In Proc. Int. Cognitive Robotics Workshop, Edmonton, Canada, 2002. 2. C. Bauckhage, C. Thurau, and G. Sagerer. Learning Human-like Opponent Behavior for Interactive Computer Games. In B. Michaelis and G. Krell, editors, Pattern Recognition, Lecturenotes in Computer Science 2781, pages 148–155. Springer-Verlag, 2003. 3. S. Cass. Mind games. IEEE Spectrum, pages 40–44, December 2002. 4. A. Fod, M.J. Mataric, and O.C. Jenkins. Automated Derivation of Primitives for Movement Classification. Autonomous Robots, 12(1):39–54, 2002. 5. B. Fritzke. A growing neural gas network learns topologies. In G. Tesauro, D. S. Touretzky, and T. K. Leen, editors, Advances in Neural Information Processing Systems 7, pages 625– 632. MIT Press, Cambridge MA, 1995. 6. A. Galata, N. Johnson, and D. Hogg. Learning Variable-Length Markov Models of Behaviour. Computer Visiosn and Image Understanding, 81(3):398–413, 2001. 7. Z. Gharamani. Building blocks of movement. Nature, 407:682–683, October 2000. 8. M.A. Giese, B. Knappmeyer, and H.H. B¨ulthoff. Automatic synthesis of sequences of human movements by linear combination of learned example patterns. In H.H. B¨ulthoff, S.W. Lee, T. Poggio, and C. Walraven, editors, Biologically Motivated Computer Vision, volume 2525 of LNCS, pages 538–547. Springer, 2002. 9. W. Ilg and M.A. Giese. Modeling of movement sequences based on hierarchical spatiotemporal correspondences of movement primitives. In H.H. B¨ulthoff, S.W. Lee, T. Poggio, and C. Walraven, editors, Biologically Motivated Computer Vision, volume 2525 of LNCS, pages 528–537. Springer, 2002. 10. J. E. Laird and M. v. Lent. Interactice Computer Games: Human-Level AI’s Killer Application. In Proc. AAAI, pages 1171–1178, 2000. 11. T. Martinez and K. Schulten. A neural gas network learns topologies. In Artificial Neural Networks. Elseviers Science Publishers B.V, 1991. 12. T.M. Martinez, S.G. Berkovich, and K.J. Schulten. Neural gas network for vector quantization and its application to time-series prediction. IEEE Trans. on Neural Networks, 4(4):558– 569, 1993. 13. A. Naraeyek. Computer Games – Boon or Bane for AI Research. K¨unstliche Intelligenz, pages 43–44, February 2004. 14. R.P.N. Rao and A.N. Meltzoff. Imitation learning in infoants and robots: Towards probabilistic computational models. In Proc. AISB 2003 Convention: Cognition in Machines and Animals, Aberystwyth, UK, 2003. 15. S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primitives. In Proc. Int. Symposium on Robotics Research, Siena, Italy, 2003. 16. K.A. Thoroughman and R. Shadmehr. Learning of action through adaptive combination of motor primitives. Nature, 407:742–747, October 2000. 17. C. Thurau, C. Bauckhage, and G. Sagerer. Learning Human-Like Movement Behavior for Computer Games. In Proc. 8th Int. Conf. on the Simulation of Adaptive Behavior (SAB’04), 2004. to appear.
