Procedural Content Generation via Machine Learning (PCGML)

1

Procedural Content Generation via Machine Learning (PCGML) Adam Summerville1 , Sam Snodgrass2 , Matthew Guzdial3 , Christoffer Holmgård4 , Amy K. Hoover5 , Aaron Isaksen6 , Andy Nealen6 , and Julian Togelius6 , 1 Department

of Computational Media, University of California, Santa Cruz, CA 95064, USA of Computing and Informatics, Drexel University, Philadelpia, PA 19104, USA 3 School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA 4 Duck and Cover Games ApS, 1311 Copenhagen K, Denmark 5 College of Arts, Media and Design, Northeastern University, Boston, MA 02115, USA 6 Department of Computer Science and Engineering, New York University, Brooklyn, NY 11201, USA emails: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

arXiv:1702.00539v1 [cs.AI] 2 Feb 2017

2 College

This survey explores Procedural Content Generation via Machine Learning (PCGML), defined as the generation of game content using machine learning models trained on existing content. As the importance of PCG for game development increases, researchers explore new avenues for generating high-quality content with or without human involvement; this paper addresses the relatively new paradigm of using machine learning (in contrast with search-based, solver-based, and constructive methods). We focus on what is most often considered functional game content such as platformer levels, game maps, interactive fiction stories, and cards in collectible card games, as opposed to cosmetic content such as sprites and sound effects. In addition to using PCG for autonomous generation, co-creativity, mixed-initiative design, and compression, PCGML is suited for repair, critique, and content analysis because of its focus on modeling existing content. We discuss various data sources and representations that affect the resulting generated content. Multiple PCGML methods are covered, including neural networks, long short-term memory (LSTM) networks, autoencoders, and deep convolutional networks; Markov models, n-grams, and multi-dimensional Markov chains; clustering; and matrix factorization. Finally, we discuss open problems in the application of PCGML, including learning from small datasets, lack of training data, multi-layered learning, style-transfer, parameter tuning, and PCG as a game mechanic. Index Terms—Computational and artificial intelligence, Machine learning, Procedural Content Generation, Knowledge representation, Pattern analysis, Electronic design methodology, Design tools

I. I NTRODUCTION

What these methods have in common is that the algorithms, parameters, constraints, and objectives Procedural content generation (PCG), the creation that create the content are in general hand-crafted by of game content through algorithmic means, has designers or researchers. While common to examine become increasingly prominent within both game existing game content for inspiration, so far machine development and technical games research. It is emlearning methods have far less commonly been used ployed to increase replay value, reduce production to extract data from existing game content in order cost and effort, to save storage space, or simply as an to create more content. aesthetic in itself. Academic PCG research addresses Concurrently, there has been an explosion in these challenges, but also explores how PCG can the use of machine learning to train models based enable new types of game experiences, including on datasets [4]. In particular, the resurgence of games that can adapt to the player. Researchers neural networks under the name deep learning has also addresses challenges in computational creativity precipitated a massive increase in the capabilities and ways of increasing our understanding of game and application of methods for learning models from design through building formal models [1]. big data [5], [6]. Deep learning has been used for a In the games industry, many applications of PCG variety of tasks in machine learning, including the are what could be called “constructive” methods, generation of content. using grammars or noise-based algorithms to create content in a pipeline without evaluation. Many For example, generative adversarial networks techniques use either search-based methods [2] (for have been applied to generating artifacts such as example using evolutionary algorithms) or solver- images, music, and speech [7]. But many other based methods [3] to generate content in settings machine learning methods can also be utilized that maximize objectives and/or preserve constraints. in a generative role, including n-grams, Markov

2

models, autoencoders, and others [8], [9], [10]. The basic idea is to train a model on instances sampled from some distribution, and then use this model to produce new samples. This paper is about the nascent idea and practice of generating game content from machine-learned models. We define Procedural Content Generation via Machine Learning (abbreviated PCGML) as the generation of game content using models that have been trained on existing game content. These models could be of many different kinds and trained using very different training algorithms, including neural networks, probabilistic models, decision trees, and others. The generation could be partial or complete, autonomous, interactive, or guided. The content could be almost anything in a game, such as levels, maps, items, weapons, quests, characters, rules, etc. This paper is focused on game content that is directly related to game mechanics. In other words, we focus on functional rather than cosmetic game content. We define functional content as artifacts that, if they were changed, could alter the in-game effects of a sequence of player actions. The main types of cosmetic game content that we exclude are textures and sound, as those do not directly impact the effects of in-game actions the way levels or rules do in most games, and there is already much research on the generation of such content outside of games [11], [12]. This not a value judgment, and cosmetic content is extremely important in games; however, it is not the focus of this paper. It is important to note a key difference between game content generation and procedural generation in many other domains: most game content has strict structural constraints to ensure playability. These constraints differ from the structural constraints of text or music because of the need to play games in order to experience them. Where images, sounds, and in many ways also text can be consumed statically, games are dynamic and must be evaluated through interaction that requires non-trivial effort—in Aarseth’s terminology, games are ergodic media [13]. A level that structurally prevents players from finishing it is not a good level, even if it’s visually attractive; a strategy game map with a strategy-breaking shortcut will not be played even if it has interesting features; a gamebreaking card in a collectible card game is merely a curiosity; and so on. Thus, the domain of game content generation poses different challenges from that of other generative domains. Of course, there are many other types of content in other domains which pose different, and in some sense more difficult challenges, such as lifelike and beautiful images or evocative musical pieces; however, in this paper we focus on the challenges posed by game content by virtue of its necessity for interaction. The remainder of this paper is structured as fol-

lows. Section II describes the various use cases for PCG via machine learning, including various types of generation and uses of the learned models for purposes that are not strictly generative. Section III discusses the key problem of data acquisition and the recurring problem of small datasets. Section IV includes a large number of examples of PCGML approaches. As we will see, there is already a large diversity of methodological approaches, but only a limited number of domains have been attempted. In Section V, we outline a number of important open problems in research and application of PCGML. II. U SE C ASES FOR PCGML Procedural Content Generation via Machine Learning shares many uses with other forms of PCG: in particular, autonomous generation, cocreation/mixed initiative design, and data compression. However, because it has been trained on existing content, it can also extend into new use areas, such as repair and critique/analysis of new content. A. Autonomous Generation The most straightforward application of PCGML is autonomous PCG: the generation of complete game artifacts without human input at the time of generation. Autonomous generation is particularly useful when online content generation is needed, such as in rogue-like games which require runtime level generation. PCGML is well-suited for autonomous generation because the input to the system can be examples of representative content specified in the content domain. With search-based PCG using a generateand-test framework, a programmer must specify an algorithm for generating the content and an evaluation function that can validate the fitness of the new artifact [14]. However, these are rarely coded in the same space as the output artifact, requiring the designer to use a different domain to express their ideas. With PCGML, a designer can create a set of representative artifacts in the target domain as a model for the generator, and then the algorithm can generate new content in this style. PCGML avoids the complicated step of experts having to design specific algorithms to create a particular type of content. B. Co-creative and Mixed-initiative Design A more compelling use case for PCGML is AIassisted design, where a human designer and an algorithm work together to create content. This approach has previously been explored with other methods such as constraint satisfaction algorithms and evolutionary algorithms [15], [16], [17].

3

Again, because the designer can train the machine- the aesthetics of the game and repair broken parts, learning algorithm by providing examples in the could be very useful. Such models could also be target domain, the designer is “speaking the same used to automatically evaluate game content, as is language” the algorithm requires for input and already done within many applications of searchoutput. This has the potential to reduce frustration, based PCG, and potentially be used together with user error, user training time, and lower the barrier other generative methods. to entry because a programming language is not required to specify generation or acceptance criteria. E. Data Compression Because PCGML algorithms are provided with One of the original motivations for PCG, particexample data, they are suited to auto-complete ularly in early games such as Elite [23], was data game content that is partially specified by the compression. There was simply not enough space designer. Within the image domain, we have seen on disk for the whole game universe. The same is work on image inpainting, where a neural network true for some of today’s games such as No Man’s is trained to complete images where parts are Sky [24]. The compression of game data into fewer missing [18]. Similarly, machine learning methods dimensions through machine learning could allow could be trained to complete game content with more efficient game content storage. By exploitparts missing. ing the regularities of a large number of content instances, we can store the distinctive features of C. Repair each more cheaply. Unsupervised learning methods With a library of existing representative content, such as autoencoders might be particularly well PCGML algorithms can identify areas that are not suited to this. playable (e.g. if an unplayable level or impossible III. DATA S OURCES AND R EPRESENTATIONS rule set has been specified) and also offer suggestions for how to fix them. Summerville and In this section we investigate how training data, Mateas [19] use a special tile that represents where varying in source, quality, and representation, can an AI would choose to move the player in their impact PCGML work. training set, so that the algorithm only generates playable content; the system inherently has learned A. Data Sources the difference between passable and impassable A commonly held view in the machine learning terrain. Jain et. al. [20] used a sliding window and community is that more data is better. Halevy, an autoencoder to repair illegal level segments – et al. [25] make the case that having access to because they did not appear in the training set, the more data, not better algorithms or better data, is autoencoder replaced them with a nearby window the key difference between why some problems it had seen during training. are able to be solved in an effective manner with machine learning while others are not (e.g., language D. Recognition, Critique, and Analysis translation is a much harder problem than document A use case for PCGML that sets it apart from classification, but more data exists for the translation other PCG approaches is its capacity for recognition, task). For some types of machine learning a surfeit analysis and critique of game content. Given the of data exists enabling certain techniques, such as basic idea of PCGML is to train some kind of model images for generative adversarial networks [26] or on sets of existing game content, these models could text for Long Short-Term Memory recurrent neural be applied to analyzing other game content, whether networks (LSTMs) [27]; However, video games created by an algorithm, players, or designers. do not have this luxury. While many games exist, For example, by supervised training on sets of they typically share no common data structures. game content artifacts and player response data, Additionally, even if they do they typically do not machine learning models can be trained to classify share semantics (e.g., sprites in the NES share game content into various types, such as broken or similarity in how they are displayed but sprite 0x01 whole, underworld or overworld, etc. They can also is unlikely to have similar semantics between Super be trained to predict quantities such as difficulty, Mario Bros. [28] and Metroid [29], despite the fact completion time, or emotional tenor [21], [22]. that both are platformers). As such, data sources They also have the potential to identify uniqueness, are often limited to within a game series [30], [19] for example by noting how common a particular and more commonly within individual games [31], pattern appears in the training set, or judging how [32], [33]. This in turn means that data scarcity is typical a complete content artifact (such as a level) likely to plague PCGML for the foreseeable future, is with regard to an existing set. Being able to as there are a small, finite number of official maps identify which user-created levels are truly novel, for most games. However, for many games there and then enhance them further so as to fit with are vibrant fan communities producing hundreds,

4

thousands, and even millions of maps [34], which original two-dimensional level must be transformed could conceivably provide much larger data sources. according to some function. Most simply this involves iterating over the data Recently, Summerville et al. created the video game level corpus (VGLC) [35], a collection of from left-to-right then top-to-bottom [20], however video game levels represented in several easily depending on the technique, this can obfuscate parseable formats. The corpus contains 428 levels spatial relationships. This loss can be mitigated, for from 12 games in 3 different file formats. This example [19] utilizes a “snaking” path while [36] effort marks an initial step in creating a shared data represents vertical and horizontal relations by splitsource that can be used for commonality across ting two-dimensional maps into one-dimensional PCGML research. That being said, the repository sequences of columns. Super Mario Bros. stores its levels in a grid of is limited in scale (12 games and 428 levels being small on the standard machine learning front) and “sprites,” a predefined set of 16 pixel square images the representations are lossy (e.g., in Mario, levels that can be tiled to create larger shapes. Many of only contain a single Enemy type mapping for all these sprites represent the same in-game function Goombas, Koopa Troopas, etc.), so additional effort such as “solid blocks” or “enemies,” which tileis needed to increase the scope and fidelity of the based representations take advantage of. In a tilebased representation, the low-level sprites of the representations. Assuming bountiful data existed for video games, game are classified into a higher order representation bigger questions abound: 1) How should the data based on gameplay function (e.g. all enemies clasbe represented? 2) What is a training instance? For sified as an “enemy" type). This improves learning certain classes of PCGML, such as the aforemen- by cutting down on variance, which is important tioned image generation, obvious data structures in cases of PCGML due to the limited size of the exist (e.g., RGB matrices), but there are no common training set. However this representation requires structures for games. Games are a combination hand-authored rules to transform from higher order of some amount of content (levels, images, 3-D tiles to sprites after generation. While tile-based representations have been the models, text, boards, pieces, etc.) and the rules that most popular thus far, we highlight two alternatives. govern them (What happens when input x is held Voices-based representations as in [36] similarly down in state y? What happens if z is clicked on? separate sprites by function, with similar drawbacks, Can player α move piece p? etc.). Furthermore, the but rely on a musical metaphor to categorize sprites content is often dynamic during playtime, so even according to voices based on gameplay function. if the content were to capture with perfect fidelity Shape-based representations as in [33] instead use (e.g., a Goomba starts at tile X, Y ) how it will affect the game’s original sprites as their atomic unit, gameplay requires play (e.g., by the time the player 0 0 categorizing the various shapes they are arranged can see the Goomba it has moved to tile X , Y ). into, which therefore requires no processing after Thus, issues of representation serve as a second generation. However this representation limits the major challenge, after access to sufficient quantities types of machine learning approaches that can be of quality data. applied. PCGML has been largely limited to the representation of video game levels thus far, impacting the B. Data Representation Example: Mario Levels types of data source and representation approaches We next look at Super Mario Bros. levels as a that we’ve seen. However, we note that PCGML case study in the importance of data representation has promise to address many other elements of in PCGML. Super Mario Bros. levels have been one video games, along with entire games themselves. of the most successful venues for PCGML research, We expect that future PCGML approaches will with many of these prior approaches having distinct still struggle with issues of data sources and data representations, with individual affordances and representations, which will in turn restrict the limitations. These can be divided according to two underlying machine learning approaches. features: how the units of the representation are arranged (one or two-dimensional representations) C. Non-level Content and the atomic unit of the representation (tiles, voices, shapes). The majority of PCGML work thus far, and the Two-dimensional representations of Super Mario majority covered in this paper, focuses on generBros. levels parallel the two-dimensional spatial rela- ating individual game levels. However, PCGML tionships in the game. However, a two-dimensional can apply to all types of game content, including representation limits the types of machine learn- mechanics, stories, quests, NPCS, and even full ing techniques that can be applied. Therefore the games. We note two examples covered in this majority of PCGML approaches structure levels paper: Magic/Hearthstone card generation [37] as one-dimensional sequences, meaning that the and interactive narrative/story generation [38]. The

5

former shares a framework with much of the level generation work, relying on a fan-made corpus of game content. The latter relies on crowdsourcing its training data, given the lack of a general, formal story representation. We expect either of these strategies could find success in other types of PCGML content generation. IV. M ETHODS OF PCGML We classify procedural content generation approaches that employ machine learning techniques (PCGML) using two axes: method, the machine learning approach that is used; and content, the type of content that is generated. In this section, we discuss various PCGML techniques, grouped by method. We start with neural networks, because of the recent uptake in PCGML approaches employing neural networks. We then discuss Markov models, which have been explored in the context of platformer game level generation. Next we cover clustering-based approaches that have been used to model and generate platformer levels as well as interactive stories. Afterwards, we discuss matrix factorization techniques for generating platformer and action role-playing game levels. We close the section with an explanation of a wave function collapse approach. A. Artificial Neural Networks Recently neural networks have been used in the context of generating images [40], [41] and learning to play games [42], [43], [44] with great success. Unsurprisingly, neural networks are also being explored in PCG for level and map generation as well as cooperative trading card generation. 1) Function Scaffolding for Mario Levels Hoover et al. [36] generate levels for Super Mario Bros. by extending a representation called functional scaffolding for musical composition (FSMC) that was originally developed to compose music. The original FSMC representation posits 1) music can be represented as a function of time and 2) musical voices in a given piece are functionally related [45]. Through a method for evolving artificial neural networks (ANNs) called NeuroEvolution of Augmenting Topologies (NEAT) [39], additional musical voices are then evolved to be played simultaneously with an original human-composed voice. To extend this musical metaphor and represent Super Mario Bros. levels as functions of time, each level is broken down into a sequence of columns with a tile-sized width. The height of each column extends the height of the screen. While FSMC represents a unit of time by the length of an eighthnote, a unit of time in this approach is width of each column. At each unit of time, the system queries the ANN to decide at what height to place a tile. FSMC inputs

a musical note’s pitch and duration to the ANNs, this approach translates pitch to the height at which a tile is placed and duration to the number of times a tile-type is repeated at that height. For a given tile-type or musical voice, this information is then input to an ANN that is trained on two-thirds of the existing human-authored levels to predict the value of a tile-type at each column (as shown in Figure 1). The idea is that the ANN will learn hidden relationships between the tile-types in the humanauthored levels that can then help humans construct entire levels from as little starting information as the layout of a single tile-type. Like in FSMC, this approach combines a minimal amount of human-authored content with the output from previous iterations. The output of the network is added to the newly created level and fed back as input into the generation of new tile layouts. By acknowledging and iterating on the relationships between design pieces inherent in a human-produced level, this method can generate maps that both adhere to some important aspects of level design while deviating from others in novel ways. 2) LSTMs for Mario Levels Summerville and Mateas [19] used Long ShortTerm Memory Recurrent Neural Networks (LSTM RNNs) [46] to generate levels learned from a tile representation of Super Mario Bros. and Super Mario Bros. 2 (JP) [47] levels. LSTMs are a variant of RNNs that represent the current state-of-the-art for sequence based tasks, able to hold information for 100’s and 1000’s of time steps, unlike the 5-6 of standard of RNNs. Summerville and Matteas used a tile representation like the one used by Snodgrass and Ontañón [48], but instead of representing levels as 2-D arrays they represent them as a linear string with 3 different representations used for experimentation. They also included simulated player path information in the input data, forcing the generator to generate exemplar paths in addition to the level geometry. Finally, they included information about how deep into the string the level geometry was, causing the generator to learn both level progression and when a level should end. Summerville et al. [49] extended this work to incorporate actual player paths extracted from 4 different YouTube playthroughs. The incorporation of a specific player’s paths biased the generators to generate content suited to that player (e.g., the player that went out of their way to collect every coin and question-mark block had more coins and question-marks in their generated levels). 3) Autoencoders for Mario Levels In [20] Jain, Isaksen, Holmgård and Togelius show how autoencoders [50] may be trained to reproduce levels from the original Super Mario Bros. game. The autoencoders are trained on series of vertical level windows and compress the typical

6

Outputs

Brick

Bias

Ground (t)

Ground (t-1)

Ground (t-2)

Output (t-1)

Output (t-2)

Output(t-3)

Inputs Figure 1: The ANN Representation. At each column in a given level, an ANN inputs visual assets (i.e., musical voices) from at least one tile type of Super Mario Bros. and outputs a different type of visual asset. This example figure shows ground tiles being input to the ANN while the brick tile placements are output from predictions made through NeuroEvolution of Augmenting Topologies [39]. To best capture the regularities in a level, each column is also provided information about the most recently encountered inputs (i.e., input values for the three previous columns). The inputs and outputs are then fed back into the ANN at each subsequent tick. Once trained, ANNs can potentially suggest reasonable placements for new human composed in-game tiles.

(a) Original

(b) Unplayable

(c) Repaired

Figure 3: The original window is overwritten with a vertical wall making the game unplayable. The autoencoder is able to repair the window to make it playable again, although it chooses a different solution to the problem. Figure 2: Example output of the LSTM approach, including generated exemplar player path. features of Mario levels into a more general representation. They experimented with the width of the level windows and found that four tiles seems to work best for Mario levels. They proceeded to use these networks to discriminate generated levels from original levels, and to generate new levels as transformation from noise. They also demonstrated how a trained autoencoder may be used to repair unplayable levels, changing tile-level features to make levels playable, by inputting a broken/unplayable level to the autoencoder and receiving a repaired one as output (see Figure 3). 4) Deep Nets for StarCraft II Lee, Isaksen, Holmgård and Togelius [51] use convolutional neural networks to predict resource locations in maps for StarCraft II [52]. Resource

locations are sparsely represented in StarCraft II maps, but are decisive in shaping gameplay. They are usually placed to match the topology of maps. Exploiting this fact, they transform StarCraft II maps into heightmaps, downsample them, and train deep neural networks to predict resource locations. The neural networks are shown to perform well in some cases, but struggled in other cases, most likely due to overfitting owing to a small training dataset. By adding a number of postprocessing steps to the network, the authors created a tool that allows map designers to vary the amount of resource locations in an automatically decorated map, varying the frequency of mineral placements, shown in Figure 4. 5) LSTM Generation of Magic Cards A rare example of game content generation that is not a level generator is Magic: The Gathering [53] card generation. The first to use a machine learning approach for this is Morgan Milewicz

7

piece of the card, addressing one of the limitations of @RoboRosewater. This work shows multiple different paradigms for PCGML, showing how it can be used on different game content and as a co-creative design assistant, as in Hoover et al.’s [36] previously mentioned work.

(a) A StarCraft II heightmap

Figure 5: Example partial card specification and its resulting output.

B. Markov Models Markov models are stochastic models that model changes in a system over time, where the current state of the system depends on the previous state. This type of model lends itself easily to modeling levels for games that have a directional trend, such as Super Mario Bros. and Kid Icarus [56], where the “time” of the system relates to the position in the level. 1) n-grams for Mario Levels (b) Varying resource amounts. Perhaps the simplest possible Markov models are Figure 4: A heightmap and generated resource n-gram models. An n-gram model (or n-gram for locations for StarCraft II short) is simply an n-dimensional array, where the probability of each character is determined by the n characters that precede it. Training an n-gram with @RoboRosewater [54] a twitter bot that uses consists of scanning a set of strings and calculating LSTMs to generate cards. Trained on the entirety the conditional probability of each character. of the corpus of Magic cards it generates cards, Dahlskog et al. trained n-gram models on the represented as a sequence of text fields (e.g., Cost, levels of the original Super Mario Bros. game, and Name, Type, etc.). A limitation of this representation used these models to generate new levels [31]. As and technique is that by treating cards as a sequence n-gram models are fundamentally one-dimensional, there is no way to condition generation of cards on these levels needed to be converted to strings in fields that occur later in the sequence (e.g., Cost order for n-grams to be applicable. This was done occurs near the end of the sequence and as such through dividing the levels into vertical “slices,” can not be used to condition the generation unless where most slices recur many times throughout the all previous fields are specified). level [57]. This representational trick is dependent 6) Sequence-to-Sequence Magic Card Creation on there being a large amount of redundancy in the Building on the work of @RoboRosewater is level design, something that is true in many games. Mystical Tutor by Summerville and Mateas [37]. Models were trained using various levels of n, and Using sequence-to-sequence learning [55], wherein it was observed that while n = 0 create essentially an encoding LSTM encodes an input sequence into random structures and n = 1 create barely playable a fixed length vector which is then decoded by a levels, n = 2 and n = 3 create rather well-shaped decoder LSTM, they trained on the corpus of Magic: levels. See Figure 6 for examples of this. 2) Multi-dimensional Markov Chains for Levels The Gathering. They corrupted the input sequences by replacing lines of text with a MISSING token In [48] Snodgrass and Ontañón present an apand then tried to reproduce the original cards as proach to level generation using multi-dimensional the output sequence. The corruption of the input Markov chains (MdMCs) [58]. An MdMC differs along with the sequence-to-sequence architecture from a standard Markov chain in that it allows for deallows the generator to be conditioned on any pendencies in multiple directions and from multiple

8

(a) n = 1

(b) n = 2

(c) n = 3

Figure 6: Mario levels reconstructed by n-grams with n set to 1, 2, and 3 respectively.

states, whereas a standard Markov chain only allows for dependence on the previous state alone. In their work, Snodgrass and Ontañón represent video game levels as 2-D arrays of tiles representing features in the levels. For example, in Super Mario Bros. they use tiles representing the ground, enemies, and ?-blocks, etc. These tile types are used as the states in the MdMC. That is, the type of the next tile is dependent upon the types of surrounding tiles, given the network structure of the MdMC (i.e., the states that the current state’s value depends on). In order to generate levels, first the model must be trained. They train an MdMC by building a probability table according to the frequency of the tiles in training data, given the network structure of the MdMC, the set of training levels, and the set of tile types. A new level is then sampled one tile at a time by probabilistically choosing the next tile based upon the types of the previous tiles and the learned probability table. In addition to their standard MdMC approach, Snodgrass and Ontañón have explored hierarchical [32] and constrained [59] extensions to MdMCs in order to capture higher level structures and ensure usability of the sampled levels, respectively. They have also developed a Markov random field approach (MRF) [60] that performed better than the standard MdMC model in Kid Icarus, a domain where platform placement is pivotal to playability. Figure 7 shows a section of a Super Mario Bros. level (top) and a section of a Kid Icarus level (bottom-left) sampled using a constrained MdMC approach, and a section of Kid Icarus level (bottomright) sampled using the MRF approach. 3) Wave Function Collapse A recent approach by Gumin [61] leverages concepts from quantum mechanics in order to generate images and levels from a representative example tile set. Specifically, he created a wave function collapse algorithm that takes a set of representative level sections and generates new levels that are locally similar to the input. That is, each N × N window in the output appears in the

Figure 7: Sections from a Super Mario Bros. level (top) and a Kid Icarus level (bottom left) generated using the constrained MdMC approach, and a Kid Icarus level generated using an MRF approach (bottom right).

input, and the distribution of each N ×N window in the output is similar to the distribution in the input, where N is chosen prior to generation, typically based on the patterns present in the training data. This wave function collapse algorithm is particularly exciting, because it is able to produce high-quality results with a single training example. The algorithm works by initializing a level to superpositions of possible N × N windows from the input. It then chooses a position among those with the lowest entropy, and collapses it (i.e., sets it probabilistically according to the distribution of windows in the input and the possible windows for that position). The entropy for the rest of the level is updated accordingly. This process is repeated until all of the positions in the output have been collapsed. This approach is a variant of MdMC, except instead of sampling based on a geometric pattern (e.g. leftto-right, bottom-to-top), samples are chosen via the

9

aforementioned entropy methodology. This approach was initially explored for bitmap generation, but has since been expanded for use with 3-D tile sets as well as for level generation [62], [63]. The source code and examples of the bitmap project can be found online [61].

novel chunks of level. After this point generating a new level requires generating novel level chunks in sequences derived from the gameplay videos. An example screen of generated content using this approach is shown in Figure 8.

Figure 9: Example of Scheherazade-IF being played. 2) Scheherazade-IF: General Interactive Fiction PCGML has focused on graphical games, and Figure 8: Example level section from the clustering particularly the levels of graphical games. However, approach. there exists some work in the field of generating interactive fiction, text-based games like choose your own adventure stories. Guzdial et al. adapted Scheherazade [66], a system for automatically learnC. Clustering ing to generate stories, into Scheherazade-IF [38], Clustering is a category of machine learning which can derive entire interactive fiction games approaches that attempt to group similar elements from a dataset of stories. together in order to learn their relationships. Below Both Scheherazades rely on exemplar stories we discuss an approach that uses clustering to learn crowdsourced from Amazon Mechanical Turk grammars and graphs for level and story generation, (AMT). The reliance on crowdsourcing rather than respectively. finding stories "in the wild" allows Scheherazade 1) Learning Mario Levels from Video to collect linguistically simple stories and stories In [33] Guzdial and Riedl utilized gameplay related to a particular genre or setting (e.g., a bank video of individuals playing through Super Mario robbery, a movie date, etc). Bros. to generate new levels. They accomplished Scheherazade-IF’s structures its model in the form this by parsing each Super Mario Bros. gameplay of plot graphs [67], directed graphs with events as video frame-by-frame with OpenCV [64] and a fan- vertices and sequentiality information encoded as authored spritesheet, a collection of each image edges. The system derives events from the individual that appears in the game (referred to as sprites). sentences of the story, learning what events can Individual parsed frames could then combine to occur via an adaption of the OPTICS clustering form chunks of level geometry, which served as the algorithm [68]. The ordering of these primitive input to the model construction process. In total events can then be derived from the exemplar Guzdial and Riedl made use of nine gameplay stories, which then enables the construction of a plot videos for their work with Super Mario Bros., graph for each type of story (e.g., bank robbery). roughly 4 hours of gameplay in total. Scheherazade-IF uses these learned plot graphs as Guzdial and Riedl’s model structure was adapted the basis for an interactive narrative experience, from [65], a graph structure meant to encode styles allowing players to take the role of any of the of shapes and their probabilistic relationships. The characters in the plot graph (robber or bank teller shapes in this case refer to collections of identical in bank robbery) and making choices based on the sprites tiled over space in different configurations. possible sequences of learned events. For further details please see [33], but it can be understood as a learned shape grammar, identifying D. Matrix Factorization individual shapes and probabilistic rules on how to Matrix factorization is a way of extracting latent combine them. To train this model Guzdial and Riedl features from high-dimensional data. These techused a hierarchical clustering approach utilizing niques have been explored (below) in the context K-means clustering with automatic K estimation. of learning from level data and learning from other First the chunks of level geometry were clustered to level generators. derive styles of level chunks, then the shapes within 1) Bayes Nets and PCA for Zelda Levels that chunk were clustered again to derive styles of shapes, and lastly the styles of shapes were clustered While platformers, specifically Super Mario Bros., to determine how they could be combined to form have dominated the field, Summerville et al. [30]

10

[69] have generated levels learned from The Legend V. O PEN P ROBLEMS AND O UTLOOK of Zelda [70] series. Combining different approaches In this section, we describe some of the major for the content, they split the generation process into issues facing PCGML and open areas for active two domains. The first is the dungeon level wherein research. Because games as a ML research domain the base topology of a dungeon is generated in a are considerably different from other ML domains, graph structure. The room-to-room structure of a such as vision and audio learning, the problems that dungeon is learned along with high level parameters arise are commensurately different. In particular, such as dungeon size and length of the optimal game datasets are typically much smaller than player path using a Bayes Net [71], a graphical other domains’ datasets, they are dynamic systems structure of the joint probability distribution. that rely on interaction, and the availability of After generating the topological structure of high-quality clean data is limited. In the following a dungeon, they then use Principal Component sections we discuss the issue of limited data as Analysis (PCA) to generate the low-level tile rep- well as underexplored applications of PCGML for resentations of the individual rooms. PCA finds games includes style transfer, parameter tuning, and a compressed representation of the original 2-D potential PCGML game mechanics. room arrays by taking the eigenvectors of the original high dimensional space and only keeping A. Learning from Small Datasets the most informative, in this work only keeping As previously mentioned, it is a commonly held the top 20 most informative accounting for 95% tenet that more data is better; However, games are of all variation. These compressed representations likely to always be data-constrained. The English are then represented as weight vectors that can be Gigaword corpus [75] which has been used in over interpolated between to generate new room content 450 papers at the time of writing is approximately 27 as seen in Figure 10. GB. The entire library of games for the Nintendo 2) Matrix Factorization for Level Generation Entertainment System is approximately 237 MB, Many generators (both machine and human) creover 2 orders of magnitude smaller. The most comate in a particular identifiable style with a limited exmon genre, platformers, makes up approximately pressive range [72]. Shaker and Abou-Zleikha [73] 14% of the NES library (and is the most common use Non-Negative Matrix Factorization (NNMF) to genre for PCG research) which is roughly another learn patterns from 5 unique non-ML-based unique order of magnitude smaller. For specific series it is generators of Super Mario Bros. levels (Notch, even worse, with the largest series (Mega Man [76]) Parameterized, Grammatical Evolution, Launchpad, making up 0.8% of the NES library, for 4 orders of and Hopper), to create a wider variety of content magnitude smaller than a standard language corpus. than any of the single methods. To their knowledge, Standard image corpora are even larger than the this is the first time that NNMF has been used Gigaword corpus, such as the ImageNet corpus [77] to generate content. Because NNMF requires only at 155GB. All told, work in this area is always going non-negative values in its matrix decomposition, to be constrained by the amount of data, even if the basis representations are localized and purely more work is put into creating larger, better corpora. additive, and they are therefore more intuitive to While most machine learning is focused on using analyze than PCA. large corpora, there is work in the field focused Their system begins by generating 1000 levels on One-Shot Learning/Generalization [78]. Humans of length 200 with each of the G = 5 unique have the ability to generalize very quickly after generators. These 5000 levels in the training set being shown a single (or very small set, e.g., n < 5) are represented as vectors recording the locations object to other objects of the same type, and Oneor amount of T = 5 content types at each level Shot Learning is similarly concerned with learning column: Platforms, Hills, Gaps, Items, and Enemies. from a very small dataset. Most of the work has These vectors are combined into T matrices, one for been focused on image classification, but games each level content type. The algorithm then uses a are a natural testbed for such techniques due to the multiplicative update NNMF algorithm [74] to factor paucity of data. the data matrices into T approximate “part matrices” which represent level patterns and T coefficient B. Learning on Different Levels of Abstraction matrices corresponding to weights for each pattern that appear in the training set. The part matrices can As described above, one key challenge that sets be examined to see what types of patterns appear PCGML for games apart from PCGML for domains globally and uniquely in different generators, and such as images or sound is the fact that games multiplying these part matrices by novel coefficient are multi-modal dynamic systems. This means that vectors can be used to generate new levels. This content generated will have interactions: generated allows the NNMF method to explore far outside the rules operate on generated levels which in turn expressive range of the original generators. change the consequences of those rules and so on.

11

Figure 10: Example of interpolation between two Zelda rooms (the leftmost and rightmost rooms).

A useful high-level formal model for understanding the games, but not the training data necessary for games as dynamic systems that create experiences machine learning approaches. is the Mechanics, Dynamics, Aesthetics (MDA) In addition to competitions, there has been some framework by Hunicke, LeBlanc and Zubek [79]. work in developing metric-based benchmarks for “Mechanics describes the particular components comparing techniques. Horn et al. [72] proposed a of the game, at the level of data representation benchmark for Super Mario Bros. level generators and algorithms. Dynamics describes the run-time in a large comparative study of multiple generbehavior of the mechanics acting on player inputs ators, which provided several evaluation metrics and each other’s outputs over time. Aesthetics that are commonly used now, but only for level describes the desirable emotional responses evoked generators for platforming games. In recent years, in the player, when she interacts with the game Canossa and Smith [82] presented several general system.” [79] If the end goal for PCGML for games evaluation metrics for procedurally generated levels, is to generate content or even complete games and which could allow for cross-techniques comparisons. with good results at the aesthetic level, the problem Again, however, these only apply to level generation is inherently a hierarchical one. It is possible that and not other generatable content. Other machine learning from data will need to happen at all of these learning domains, such as image processing, have three levels simultaneously in order to successfully benchmarks in place [83] for testing new techniques generate content for games. and comparing various techniques against each other. For the field of PCGML to grow, we need to be able to compare approaches in a meaningful way. C. Datasets and Benchmarks Procedural content generation, and specifically D. Style Transfer PCG via machine learning, is a developing field of Style and concept transfer is the idea that inforresearch. As such, there are not many publicly availmation learned from one domain can enhance or able large datasets and no standardized evaluation supplement knowledge in another domain. Style benchmarks that are widely used. transfer has most notably been explored for image Publicly available, large datasets are important, as modeling and generation [84], [85]. For example, in they will reduce the barrier for entry into PCGML, recent work Gatys et al. [41] used neural networks which will allow the community to grow more to transfer the style of an artist onto different quickly. As previously mentioned, the VGLC [35] images. Additionally, Deep Dream [40] trains neural is an important step in creating widely available networks on images, and then generates new images datasets for PCGML. However, it currently only that excite particular layers or nodes of the network. provides level data. Other data sets exist for ob1 2 This approach could be adapted to learn from game ject models and gameplay mechanics , but they content (e.g., levels, play data, etc.) and used to have not been utilized by the PCGML community. generate content that excites layers associated with By bringing these data sets to the attention of different elements from the training data. More researchers, we hope to promote the exploration traditional forms of blending domain knowledge and development of approaches for generating those have been applied sparingly to games, such as types of content. Additionally, creating, improving, game creature blending [86] and interactive fiction and growing both new and existing data sets is blending [87]. However, until very recently no one necessary for refining and expanding the field. has applied this machine learning-oriented style Widespread benchmarks allow for the comparison of various techniques in a standardized way. transfer to procedural content generation. Recently Snodgrass and Ontañón [88] explored Previously, competitions have been used to compare domain transfer in level generation across platformvarious level generation approaches [80], [81]. Howing games, and Guzdial et al. [89] used concept ever, many of the competitors in the Super Mario blending to meld different level designs from Super Bros. competition included hard coded rules, and Mario Bros. together (e.g., castle, underwater, and the GVG-AI competition provides descriptions of overworld levels). These approaches transfer and 1 opengameart.org blend level styles, but do not attempt to address 2 http://www.squidi.net/three/index.php the game mechanics explicitly; both approaches

12

ensure playable levels, but do not attempt transfer or blending between different mechanics. Gow and Corneli [90] proposed a framework for blending two games together into a new game, including mechanics, but no implementation yet exists. The above approaches are important first steps, but style transfer needs to be explored more fully in PCG. Specifically, approaches to transferring more than just aesthetics are needed, including game mechanic transfer. Additionally, the above approaches only apply transfer to game levels, and there are many other areas where style transfer can be applied. Character models, stories, and quests are a few such areas.

F. Using PCGML as a Game Mechanic

Most current work in PCGML focuses on replicating designed content in order to provide the player infinite or novel variations on gameplay, informed by past examples. Another possibility is to use PCGML as the main mechanic of a game, presenting the PCGML system as an adversary or toy for the player to engage with. Designs could include enticing the player to, for example, generate content that is significantly similar to or different from the corpus the system was trained on, identify content examples that are outliers or typical examples of the system, or train PCGML systems to generate examples that possess certain qualities or fulfill certain objective functions, teaching the player to operate an opaque or transparent model by feeding it examples that shape its output in one direction or the other. This would allow a PCGML system to take on several E. Exposing and Exploring the Generative Space design pattern roles, including AI as Role-Model, One avenue that has yet to be deeply explored Trainee, Editable, Guided, Co-Creator, Adversary, in PCGML is opening the systems up to allow and Spectacle [93]. for designer input. Most systems are trained on a Role-model: a PCGML system replicates content source of data and the editorial control over what generated by players of various levels of skill or data is fed in is the main interaction for a designer, generates content suitable for players of certain but it is easy to imagine that a designer would skill levels. New players are trained by having them want to control things such as difficulty, complexity, replicate the content or by playing the generated or theming (e.g., the difference between an above content in a form of generative tutorial. ground level and below ground level in Super Mario Trainee: the player needs to train a PCGML system Bros.). The generative approach of Summerville et al. to generate a piece of content necessary as part of [30] allows a designer to set some high-level design a puzzle or a piece of level geometry to proceed in parameters (number of rooms in a dungeon, length the game. of optimal player path), but this approach only works Editable: rather than training the AI to generate if all design factors are known at training time the missing puzzle piece via examples, the player and a Bayes Net is a suitable technique. Similarly, changes the internal model’s values until acceptable Snodgrass and Ontañón [59] allow the user to define content is generated. constraints (e.g., number of enemies, distance of Guided: the player corrects PCG system output in the longest gap, etc.), but require hand crafted order to fulfill increasingly difficult requirements. constraint checkers for each constraint used, and The AI, in turn, learns from the player’s corrections, for the domain to be able to be modeled with multi- following the player’s guidance. Co-creator: the player and a PCGML system take dimensional Markov chains. The nature of Generative Adversarial Networks turns in creating content, moving toward some has allowed for the discovery of latent factors external requirement. The PCGML system learns that hold specific semantics (e.g., subtracting the from the player’s examples. latent space image of blank-faced person from a Adversary: the player produces content that the smiling one is the smile vector) [92]. Interpolating PCGML system must replicate by generation to and extrapolating along these latent dimensions survive or vice versa in a “call and response” style allows a user to generate content while freely battle. tuning the parameters they see fit. We envision Spectacle: the PCGML system is trained to replithat this type of approach could lead to similar cate patterns that are sensorically impressive or results in the PCG domain (e.g., subtract Mario cognitively interesting. 1-1 from Mario 1-2 for the underground dimension, VI. C ONCLUSION subtract Mario 1-1 from Mario 8-3 for the difficulty dimension). Furthermore, if multiple games are In this survey paper, we give an overview of an used as training input, it is theoretically possible emerging machine learning approach to Proceduthat one could interpolate between two games ral Content Generation, including describing and (e.g., find the half-way point between Mario and contrasting the existing examples of work taking Sonic) or find other more esoteric combinations this approach and outlining a number of challenges (Contra − M ario + Zelda =???). and opportunities for future research. We intend

13

Table I: A summary of the PCGML approaches discussed in this article. Note, the C/R column specifies if the output data is Categorical (Cat.) or Real valued, and the Data column specifies the format of the data (e.g., Seq. = Sequence). The methods are listed in the order they are discussed in Section IV. Paper

C/R

Data

Content

Methods

Task

Hoover, et. al. [36]: Functional Scaffolding for Mario

Cat.

Seq.

Mario Levels

Summerville, Mateas [19]: LSTMs for Mario

Cat.

Seq.

Mario Levels

Autonomous generation, Domain transfer Autonomous generation

Jain, et. al. [20]: Autoencoders for Mario

Cat.

Seq.

Mario Levels

Lee, et. al. [51]: Deep nets for StarCraft

Real

Grid

Starcraft II Maps

@RoboRosewater [54]: LSTM generation of Magic cards

Cat.

Seq.

Magic Cards

Feed Forward NN, NEAT, FSMC Recurrent Neural Networks (RNNs) Neural Network Autoencoders Convolutional Neural Networks (CNNs) RNNs

Summerville, Mateas [37]: Seq.-to-seq. learning for Magic

Cat.

Seq.

Magic Cards

Dahlskog, et. al. [31]: n-Grams for Mario

Cat.

Snodgrass, Ontañón [48]: Multi-dim. Markov Ch. for Levels

Cat.

Snodgrass, Ontañón [32]: Hierachical Multi-dim. Markov Ch.

Cat.

Snodgrass, Ontañón [59]: Constrained Multi-dim. Markov Ch.

Cat.

Snodgrass, Ontañón [60]: Markov Random Fields for Levels

Cat.

Gumin, M. [61]: Wave Function Collapse

Cat.

Guzdial and Riedl [33]: Learning Mario Levels from Video

Cat., Real

Guzdial et. al. [38]: Scheherazade-IF: Interactive Fiction

Cat. Graph

Interactive Fiction

OPTICS Clustering

Autonomous generation, Co-creative generation Autonomous generation, Co-creative generation, Domain transfer Autonomous generation

Summerville et. al. [91]: Bayes Nets and PCA for Zelda

Cat., Graph Real Real Seq.

Zelda Dungeons

Bayes Nets, Graphical Models Non-Negative Matrix Factorization (NNMF)

Autonomous generation, Co-creative generation Design Pattern Modeling, Autonomous generation

Shaker, Abou-Zleikha [73]: Matrix Factorization for Level Gen.

Sequence to Sequence Neural Networks Seq. Mario Levels Markov Chains, n-Grams Seq. Mario, Loderunner, Multi-dimensional Kid Icarus Levels Markov Chains (MdMCs) Seq. Mario, Loderunner, Hierarchical MdMCs, Kid Icarus Levels Clustering Seq. Mario, Loderunner, Constrained MdMCs Kid Icarus Levels Grid Mario, Loderunner, Markov Random Fields, Kid Icarus Levels Metropolis-Hastings Grid Tile Levels Wave Function Collapse Graph Mario Levels K-Means Clustering

Mario Levels

the paper to play a similar role as the SearchBased Procedural Content Generation paper [2], which pointed out existing research as well as work that was yet to be done. Much research that was proposed in that paper was subsequently carried out by various authors. In Table I, we present a summary the Procedural Content Generation via Machine Learning papers discussed throughout this paper. As can be seen from this table, and indeed from the rest of the paper, there is much work left to do. The vast majority of work has so far concerned twodimensional levels, in particular Super Mario Bros. levels. Plenty of work remains in applying these methods to other domains, including rulesets, items, characters, and 3-D levels. There is also very rapid progress within machine learning in general in the moment, and in particular within the deep learning field and in methods with generative capabilities such as Generative Adversarial Networks. There is plenty of interesting and rewarding work to do in exploring how these new capabilities can be adapted to function with the particular constraints and affordances of game content.

Autonomous generation Repair, Recognition Co-creative generation Autonomous generation Co-creative generation Autonomous generation Autonomous generation Autonomous generation Autonomous generation, Domain transfer Autonomous generation

R EFERENCES [1] N. Shaker, J. Togelius, and M. J. Nelson, Procedural Content Generation in Games: A Textbook and an Overview of Current Research. Springer, 2016. [2] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search-based procedural content generation: A taxonomy and survey,” Computational Intelligence and AI in Games, IEEE Transactions on, vol. 3, no. 3, pp. 172–186, 2011. [3] A. M. Smith and M. Mateas, “Answer set programming for procedural content generation: A design space approach,” Computational Intelligence and AI in Games, IEEE Transactions on, vol. 3, no. 3, pp. 187–200, 2011. [4] J. Montgomery, “Machine learning in 2016: what does nips2016 tell us about the field today?” 2016, https://blogs.royalsociety.org/inverba/2016/12/22/machine-learning-in-2016-what-doesnips2016-tell-us-about-the-field-today/. [5] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015. [6] I. Goodfellow, Y. Bengio, and A. Courville, “Deep learning,” 2016, book in preparation for MIT Press. [Online]. Available: http://www.deeplearningbook.org [7] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, 2014, pp. 2672–2680. [8] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Modeling temporal dependencies in high-dimensional

14

[9]

[10]

[11]

[12]

[13] [14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23] [24] [25]

[26]

[27]

[28] [29] [30]

[31]

[32]

[33]

[34]

sequences: Application to polyphonic music generation and transcription,” arXiv preprint arXiv:1206.6392, 2012. S. Fine, Y. Singer, and N. Tishby, “The hierarchical hidden markov model: Analysis and applications,” Machine learning, vol. 32, no. 1, pp. 41–62, 1998. K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, “Draw: A recurrent neural network for image generation,” arXiv preprint arXiv:1502.04623, 2015. L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk, “State of the art in example-based texture synthesis,” in Eurographics 2009, State of the Art Report, EG-STAR. Eurographics Association, 2009, pp. 93–117. Z. Ren, H. Yeh, and M. C. Lin, “Example-guided physically based modal sound synthesis,” ACM Transactions on Graphics (TOG), vol. 32, no. 1, p. 1, 2013. E. J. Aarseth, Cybertext: Perspectives on ergodic literature. JHU Press, 1997. J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search-Based Procedural Content Generation,” in Applications of Evolutionary Computation. Springer, 2010, pp. 141–150. N. Shaker, M. Shaker, and J. Togelius, “Ropossum: An authoring tool for designing, optimizing and solving cut the rope levels.” 2013. A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient sketchbook: Computer-aided game level authoring.” in FDG, 2013, pp. 213–220. G. Smith, J. Whitehead, and M. Mateas, “Tanagra: Reactive planning and constraint solving for mixed-initiative level design,” IEEE Transactions on Computational Intelligence and AI in Games, no. 99, 2011. C. Guillemot and O. Le Meur, “Image inpainting: Overview and recent advances,” IEEE signal processing magazine, vol. 31, no. 1, pp. 127–144, 2014. A. Summerville and M. Mateas, “Super Mario as a string: Platformer level generation via LSTMs,” Proceedings of 1st International Joint Conference of DiGRA and FDG, 2016. R. Jain, A. Isaksen, C. Holmgård, and J. Togelius, “Autoencoders for Level Generation, Repair, and Recognition,” in ICCC, 2016. G. N. Yannakakis, P. Spronck, D. Loiacono, and E. André, “Player modeling,” in Dagstuhl Follow-Ups, vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2013. M. Guzdial, N. Sturtevant, and B. Li, “Deep static and dynamic level analysis: A study on infinite mario,” in Third Experimental AI in Games Workshop, vol. 3, 2016. D. Braben and I. Bell, “Elite,” 1984. Hello Games, “No Man’s Sky,” 2016. F. Pereira, P. Norvig, and A. Halevy, “The unreasonable effectiveness of data,” IEEE Intelligent Systems, vol. 24, no. undefined, pp. 8–12, 2009. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-toimage translation with conditional adversarial networks,” 2016. A. Karpathy, “The unreasonable effectiveness of recurrent neural networks,” 2015. [Online]. Available: http://karpathy. github.io/2015/05/21/rnn-effectiveness/ S. Miyamoto and T. Tezuka, “Super Mario Bros.” 1985. G. Yokoi and Y. Sakamoto, “Metroid,” 1986. A. Summerville and M. Mateas, “Sampling hyrule: Sampling probabilistic machine learning for level generation,” 2015. S. Dahlskog, J. Togelius, and M. J. Nelson, “Linear levels through n-grams,” in Proceedings of the 18th International Academic MindTrek Conference, 2014. S. Snodgrass and S. Ontañón, “A hierarchical MdMC approach to 2D video game map generation,” in Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference, 2015. M. Guzdial and M. Riedl, “Game level generation from gameplay videos,” in Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference, 2016. D. McFerran, “More than a million super mario maker levels have been uploaded in a week,” 2015. [Online]. Available: http://www.nintendolife.com/news/2015/09/

[35] [36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45]

[46] [47] [48]

[49]

[50]

[51]

[52] [53] [54] [55] [56] [57] [58]

[59]

more_than_a_million_super_mario_maker_levels_have_ been_uploaded_in_a_week A. J. Summerville, S. Snodgrass, M. Mateas, and Ontañón, “The VGLC: The video game level corpus.” A. K. Hoover, J. Togelius, and G. N. Yannakis, “Composing video game levels with music metaphors through functional scaffolding.” A. Summerville and M. Mateas, “Mystical tutor: A magic: The gathering design assistant via denoising sequence-tosequence learning,” 2016. [Online]. Available: http://www. aaai.org/ocs/index.php/AIIDE/AIIDE16/paper/view/13980 M. Guzdial, B. Harrison, B. Li, and M. O. Riedl, “Crowdsourcing open interactive narrative,” in the 10th International Conference on the Foundations of Digital Games. Pacific Grove, CA, 2015. K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” Evolutionary Computation, vol. 10, pp. 99–127, 2002. A. Mordvintsev, C. Olah, and M. Tyka, “Inceptionism: Going deeper into neural networks,” Google Research Blog. Retrieved June, vol. 20, 2015. L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” arXiv preprint arXiv:1508.06576, 2015. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016. S. Samothrakis, D. Perez-Liebana, S. M. Lucas, and M. Fasli, “Neuroevolution for general video game playing,” in Computational Intelligence and Games (CIG), 2015 IEEE Conference on. IEEE, 2015, pp. 200–207. A. K. Hoover, P. A. Szerlip, and K. O. Stanley, “Functional scaffolding for composing additional musical voices,” Computer Music Journal, vol. 38, no. 4, pp. 80–99, 2014. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computing, 1997. S. Miyamoto and T. Tezuka, “Super mario bros. 2,” 1986. S. Snodgrass and S. Ontañón, “Experiments in map generation using Markov chains,” in Proceedings of the 9th International Conference on Foundations of Digital Games, ser. FDG, vol. 14, 2014. A. Summerville, M. Guzdial, M. Mateas, and M. Riedl, “Learning player tailored content from observation: Platformer level generation from video traces using lstms,” in AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2016. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Intl. Conf. on Machine Learning. ACM, 2008, pp. 1096–1103. S. Lee, A. Isaksen, C. Holmgård, and J. Togelius, “Predicting Resource Locations in Game Maps Using Deep Convolutional Neural Networks,” in The Twelfth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 2016. B. Entertainment, “StarCraft II,” 2010. W. of the Coast, “Magic: The Gathering,” 1993. M. Milewicz, RoboRosewater. https://twitter.com/roborosewater?lang=en, 2016. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in CoRR, 2014. T. Osawa and Y. Sakamoto, “Kid icarus,” 1986. S. Dahlskog and J. Togelius, “Patterns as objectives for level generation,” in Proceedings of EvoApplications, 2013. W. Ching, S. Zhang, and M. Ng, “On multi-dimensional markov chain models,” Pacific Journal of Optimization, vol. 3, no. 2, 2007. S. Snodgrass and S. Ontañón, “Controllable procedural content generation via constrained multi-dimensional Markov chain sampling,” in Proceedings of the Twenty-Fifth

15

[60]

[61] [62] [63] [64]

[65]

[66]

[67]

[68]

[69]

[70] [71] [72]

[73]

[74]

[75] [76] [77]

[78]

[79]

[80]

[81]

[82]

[83] [84]

[85]

International Joint Conference on Artificial Intelligence, IJCAI, 2016, pp. 780–786. S. Snodgrass and S. Ontanon, “Learning to generate video game maps using Markov models,” IEEE Transactions on Computational Intelligence and AI in Games, 2016. M. Gumin, “Wavefunctioncollapse,” GitHub repository, 2016. Freehold Games, “The Caves of Qud,” 2016. Arcadia-Clojure, “Proc Skater 2016,” 2016. K. Pulli, A. Baksheev, K. Kornyakov, and V. Eruhimov, “Real-time computer vision with opencv,” Commun. ACM, vol. 55, no. 6, pp. 61–69, Jun. 2012. E. Kalogerakis, S. Chaudhuri, D. Koller, and V. Koltun, “A probabilistic model for component-based shape synthesis,” ACM Transactions on Graphics, vol. 31, no. 4, pp. 55:1– 55:11, Jul. 2014. B. Li, “Learning knowledge to support domain-independent narrative intelligence,” Ph.D. dissertation, Georgia Institute of Technology, 2015. P. Weyhrauch, “Guiding interactive fiction,” Ph.D. dissertation, Ph. D. Dissertation, Carnegie Mellon University, 1997. M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander, “Optics: ordering points to identify the clustering structure,” in ACM Sigmod Record, vol. 28, no. 2. ACM, 1999, pp. 49–60. A. J. Summerville, M. Behrooz, M. Mateas, and A. Jhala, “The learning of zelda: Data-driven learning of level topology.” Nintendo, “The Legend of Zelda,” 1986. S. Wright, “Correlation and causation,” 1921. B. Horn, S. Dahlskog, N. Shaker, G. Smith, and J. Togelius, “A comparative evaluation of procedural level generators in the mario ai framework,” 2014. N. Shaker and M. Abou-Zleikha, “Alone we can do so little, together we can do so much: A combinatorial approach for generating game content.” AIIDE, vol. 14, pp. 167–173, 2014. D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, 1999. D. Graff, J. Kong, K. Chen, and K. Maeda, “English gigaword,” Linguistic Data Consortium, Philadelphia, 2003. Capcom, “Mega Man,” 1987. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009, pp. 248–255. F.-F. Li, R. Fergus, and P. Perona, “One-shot learning of object categories,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594–611, 2006. R. Hunicke, M. LeBlanc, and R. Zubek, “Mda: A formal approach to game design and game research,” in Proceedings of the AAAI Workshop on Challenges in Game AI, vol. 4, 2004, p. 1. J. Togelius, N. Shaker, S. Karakovskiy, and G. N. Yannakakis, “The Mario AI Championship 2009-2012,” AI Magazine, vol. 34, no. 3, pp. 89–92, 2013. S. L. Ahmed Khalifa, Diego Perez-Liebana and J. Togelius, “General video game level generation,” in Proceedings of the Genetic and Evolutionary Computation Conference, 2016. A. Canossa and G. Smith, “Towards a procedural evaluation technique: Metrics for level design,” development, vol. 7, p. 8, 2015. Y. LeCun, C. Cortes, and C. J. Burges, “The MNIST database of handwritten digits,” 1998. S. Bruckner and M. E. Gröller, “Style transfer functions for illustrative volume rendering,” in Computer Graphics Forum, vol. 26, no. 3. Wiley Online Library, 2007, pp. 715–724. A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, “Image analogies,” in Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 2001, pp. 327–340.

[86] P. Ribeiro, F. C. Pereira, B. Marques, B. Leitao, and A. Cardoso, “A model for creativity in creature generation.” in Proceedings of the 4th GAME-ON Conference, 2003, p. 175. [87] J. Permar and B. Magerko, “A conceptual blending approach to the generation of cognitive scripts for interactive narrative,” in Proceedings of the 9th AIIDE Conference, 2013. [88] S. Snodgrass and S. Ontañón, “An approach to domain transfer in procedural content generation of two-dimensional videogame levels,” in Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference, 2016. [89] M. Guzdial and M. Riedl, “Learning to blend computer game levels,” arXiv preprint arXiv:1603.02738, 2016. [90] J. Gow and J. Corneli, “Towards generating novel games using conceptual blending,” in Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference, 2015. [91] A. J. Summerville, M. Behrooz, M. Mateas, and A. Jhala, “The learning of Zelda: Data-driven learning of level topology.” [92] T. White, “Sampling generative networks: Notes on a few effective techniques,” arXiv preprint arXiv:1609.04468, 2016. [93] M. Treanor, A. Zook, M. P. Eladhari, J. Togelius, G. Smith, M. Cook, T. Thompson, B. Magerko, J. Levine, and A. Smith, “AI-based game design patterns,” in Proceedings of the 10 International Conference on Foundations of Digital Games, FDG, 2015.