Houston, we have a problem...: A Survey of Actual Problems in Computer Games Development Fábio Petrillo, Marcelo Pimenta, Francisco Trindade, Carlos Dietrich Institute of Informatics - Federal University of Rio Grande do Sul (UFRGS) 9500 Bento Gonçalves Avenue Porto Alegre, Brazil
[email protected],{mpimenta,fmtrindade,cadietrich}@inf.ufrgs.br ABSTRACT This paper presents a survey of problems found in the development process of electronic games. These problems were collected mainly from game postmortems and specialized litterature on game development, allowing a comparison with respect to well-known problems in the traditional software industry.
Categories and Subject Descriptors K.8.0 [Personal Computing]: Games; D.2.9 [Software Engineering]: Management; K.6.3 [Management of Computing and Information Systems]: Software Management
this information in order to allow further analyze and discussion. This paper is structured as follows: after this introduction, section 2 summarizes some traditional software industry problems. In section 3 we present the actual problems cited in the specialized literature about game development and collected from the postmortems analysis, quoting examples that illustrate its relevance. Then, we discuss and describe the results obtained in the problems survey, comparing the results obtained in the game industry with data from traditional software industry, searching for similarities and idiosyncrasies. Finally, section 4 presents the conclusions.
2. General Terms
SOFTWARE INDUSTRY PROBLEMS
Although the academic community has being discussing for almost 40 years the problems that afflict the development of computational systems, we still have difficulties in elaborating robust and correct software which meets its specification, satisfies customers and also meets estimates, as much in time as in budget [1, 14], and the current status of software development can be seen as far from an ideal situation. The research made by the Standish Group [16] shows that more than 30% of the projects are cancelled before being completed and that 53% of the projects have a cost exceeded in 189%. Only 16% of the projects are completed in the stated period and budget, with all the initially specified functionalities [16]. In this scenario, Sommerville [15] suggests that the software engineering is not really in a crisis, but in a state of chronic agony. As cited in The Chaos Report [16], many errors in software projects are not deeply investigated and the same errors continue to be committed. The Standish Group, in its research of the problems of software projects, quantified the main factors that cause damages or cancellations of projects. In this research, made with IT executives, the opinion requested was which would be the main factors that cause problems in the projects. Thus, 13.1% of the executives have cited incomplete requirements and 12.4% the lack of user involvement. From these surveys and from the analysis of the references made by Yourdon [17] and Charette [7], it is possible to group the problems in four main categories:
Management, Human Factors
Keywords Electronic games, game development, problems in game development, survey, postmortems
1. INTRODUCTION The game industry has an annual income of billions of dollars, exceeding even the movie industry profits (since 2003), and it emerges as one of the most powerful, exciting and influential industry in the world of the arts [11]. Great game projects count on highly specialized multidisciplinary teams, having simultaneously software developers, designers, musicians, scriptwriters and many other professionals. However, game development is an extremely complex activity [11] and a much harder task than one can initially imagine. Surprisingly very little attention has been paid to collect information about problems that afflict the electronic games industry. The goal of this paper is to collect and organize
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC’08 March 16-20, 2008, Fortaleza, Cear´a, Brazil Copyright 2008 ACM 978-1-59593-753-7/08/0003 ...$5.00.
1. schedule problems, due to bad estimates or even optimism; 2. budget problems, by demanding investments above the estimated;
707
3. quality problems, due to a poor quality control;
The postmortem is closed with a final message from the author, commonly followed by a project technical brief, which includes the number of full- and partial-time developers, length of development, release date, target platforms, and the hardware and software used in the development. The information contained in the postmortems constitute knowledge base that can be reused by any development team, which includes examples and real life development experiences. They can help in knowledge sharing, and can be very useful for planning future projects.
4. management and business related problems, including bad communication, lack of investment in the team’s training, and the lack of project inspection in regular intervals [7]. In fact, IT projects rarely fail for only one or two reasons. More frequently a combination of these problems is found, being associated with one another [7]. It is not our intention to discuss here these problems: we will use these categories as a basis for the comparison between the traditional and the electronic games industry in section 3.4.
3.2
According to Flood [9], all game development postmortems say the same things: the project was delivered behind schedule; it contained many defects; the functionalities were not the ones that had originally been projected; a lot of pressure and an immense amount of development hours in order to complete the project. The specialized literature in electronic games development is also effective in describing the problems of this industry. For Bethke [2], the electronic games industry adopts, in general, a poor methodology, if any, causing projects to have longer duration in comparison to what they were supposed to, exceeding the budget and tending to be irrationally full of defects. This scenario is responsible for declarations as this, of Ian Lane, of Mad Doc Software [11]: “It’s rare a project where there is not some form of agony. I am trying to think of one that wasn’t miserable in some way. ” and of Scott Campbell, president and founder of Incognito Studio[11]: “Every projects has a moment where you are totally in the hell, especially when you are making new intellectual property or trying to create the “fun” and core game identity.” In order to see these problems in a clearer way, we will discuss some of them, based on the causes of imperfection referred by Flynt and Salem [10] and others reports from electronic games development specialized literature.
3. THE ELECTRONIC GAMES INDUSTRY PROBLEMS In this section we will discuss the problems of the electronic games industry. We will initially explain what postmortems are and will show the problems which are cited by the game development specialized literature, demonstrating its characteristics and pointing out the criteria that were used to define this situations as problems. Finally, we will compare the collected results with the data obtained in the traditional software industry.
3.1
Problems found in specialized literature
The Game Postmortems
The term postmortem designates a document which summarizes the project development experiences, with a strong emphasis on positive and negative aspects of the development cycle [12]. It is commonly done right after the project finishes, by managers or senior project participants [6]. In a software engineering viewpoint, postmortems are important tools for knowledge management [3], from which the group can learn from its own experiences and plan future projects. In fact, the postmortem analysis can be so revealing that some authors [3] argue that any project should be finished without it’s own postmortem. Postmortems are much used in the game industry. Many game websites devote entire sections to present these documents, such as Gamasutra (http://www.gamasutra.com) and Gamedev (http://www.gamedev.net). It is also very interesting to note the variety of development teams profiles and projects behind these documents, varying from few developers in small projects to dozens of developers in fiveyear-long projects. The postmortems published by Gamasutra mainly follow the structure proposed by the Open Letter Template [13], which is composed by three sections. The first section summarizes the project and presents some important aspects of the development. The next two sections, however, discuss the most interesting aspects to the game developers:
3.2.1
Scope problems
According to Flynt and Salem [10], the biggest reason of games project imperfection is the failure in clearly establishing the project scope. If a project does not have a well established target, emergent requirements can cause significant structural changes in the system’s architecture, causing serious problems [6]. The development teams lose themselves in the scope when some difficulties arise: problems with the exaggerated size and complexity of the project, in addition to facing highly specific requirements of the games domain [11, 4]. However, the main cause of scope problems is the common situation where new functionalities are added during the development phase, increasing the project’s size. This practice is known in the industry of games as feature creep. The late addition of features occurs when developer, managers or customers add requirements to the project after its scope has already been defined. With the intention to create the best possible game, there is the habit of adding more elements everytime, creating a probably more interesting game (even without any guarantee) but a certainly bigger one [11]. Typical reasons for feature creep are :
• What went right: it discusses the best practices adopted by developers, solutions, improvements, and project management decisions that have improved the efficiency of the team. All these aspects are critical elements to be used in planning future projects. • What went wrong: it discusses difficulties, pitfalls, and mistakes experienced by the development team in the project, in both technical and management aspects.
• inclusion by programmers of new interesting or attrac-
708
tive features during development process, mainly without a strong effort of requirements analysis;
“On one game I worked on, one of the team members complained to the producer that months of working 12-hours-perday, seven-days-a-week was getting to him. He said he’d like to have at least one weekend off. The producer replied, “This is the game industry - crunch time is a fact of life. Believing that crunch time is a fact of life is a result of following the “blind religion” that’s so pervasive in our industry”.
• addition of external code (a new component, for example) without planning, for the purpose of gaining time; however what happens many times is that an extremely hard work is necessary for the integration of this new component;
3.2.4
• developers decide to implement their own algorithms, despite counting on a set of a solid libraries. However, the games industry has a lot of examples of situations in which features discovered during the development phase have transformed the game into a success. Game development is not a linear process [10]. Thus, if an interesting function is discovered, it must be analyzed in terms of its risks and, if viable, be added to the project schedule.
3.2.2
Schedule problems
The electronic games specialized literature exhaustingly describes schedule problems in game projects. Although they usually initiate with reasonably structured schedules, bright ideas and enthusiasm, something always goes wrong. And the number of things that can go wrong is virtually endless [2]. These problems are many times associated with the multidisciplinarity needed in the elaboration of games, generating delays caused by waiting for the work of other team members, in addition to the lack of a realistic estimate on the initial plan of development, making the team not capable of finding a deadline for the projects [10]. The production of estimates of time needed to complete a task has a series of risks that imply in underestimates, causing cumulative schedule delays. According to Flynt and Salem [10], developers recurrently fail in their estimates due to lack of historical data that should assist the perception of time needed to carry through a task. Another common problem in deadline estimate is not taking into account the time consumed in complementary meetings and other activities, in addition to not planning the time needed to correct defects in the game. However, for Flynt and Salem [10], the key to the problems in the project’s schedule is the unbalance among quality, cost and opportunities. If a game is delivered two months after the release of a similar game from another company, the possibilities for success are diminished. On the other hand, a game that is launched before another one with an important number of defects, can have more problems than one that is delivered later with less bugs.
3.2.3
Technological problems
As a software device, all games are technology dependents. Moreover, the technologies used in games are so advanced that the leaders in the area of graphical computation are games companies. However, the technological component brings risks to the projects of games. Flynt and Salem [10] describe the risk of using new technologies, that many times cause a great effort and high investment of time in order to use them. According to Gershenfeld et al.[11], the technological risks are generally higher when the team is working in a new platform, that has not been completely delivered neither consolidated. The first one of these risks is that no developer has ever worked in it. The second risk is that in many times the platform hardware still contains problems, that are only found by development team of the launch title 1 .
3.3
Discussion of postmortems and research results
The 20 postmortems analyzed in these survey are listed in the table 1. Intentionally projects of several team size, budget and development time were analyzed. It is also important to point out that all postmortems analyzed were finished projects, that is, projects that had resulted in complete games and delivered to the market. Table 1: Analyzed postmortems Project Beam Runner Hyper Cross Dark Age of Camelot Black & Whire Rangers Lead the Way Wild 9 Trade Empires Rainbow Six The X-Files Draconus Cel Damage Comand and Conquer: Tiberian Sun Asheron’s Call Age of Empires II: The Age of Kings Diablo II Operation Flashpoint Hidden Evil Resident Evil 2 Vampire: The Masquerade Unreal Tournament Tropico
Crunch Time
In the games industry, the term crunch time is used for the periods of extreme work overload, typically occurring in the last weeks before validation phase and mainly in the weeks that precede the final deadline for the project delivery. In these periods, a work load of more than 12 hours a day is common, from 6 to 7 days per week, without rest intervals. Although this cyclical problem is also present in traditional software companies, Gershenfeld et al. [11] claims that the electronic games industry goes more frequency through crunch time periods. This assertion can be confirmed by Hamann [12], when describing his experience in a project:
Team 5 25 25 13 18 9 22 36 19 16 35 60 40 40 10 22 9 12 16 10
Time (Month)
4 18 37 20 NI 15 24 48 10 24 36 48 24 36 48 12 12 24 18 12
In order to organize the information about games problems, each postmortem was read and citations that were 1 Launch title is a game that is delivered at the same time that the new platform.
709
compare, into terms of its problems, the game development with the traditional software industry. Here, we compare the problems listed in the section 2 with problems in the sections 3.2 and 3.3. Initially, we will see that in fact all the main problems of the traditional software industry is also found in the electronic games industry. In both contexts, the unreal scope is highlighted, as well as the problems with the requirements analysis. Also Charette [7] points out that the unreal objectives and undefined requirements are among the main problems, strengthening the similarities. But of all the problems, what really is almost universal, as much as in information systems rather than in projects of games, is the unreal scope or the exaggerated optimism. With a percentage of 75% of the projects of analyzed games and clearly highlighted by Yourdon [17], Brooks [5] and DeMarco and Lister [8], our investigation shows clearly that optimism and naivety in the scope definition, as well as in the estimate of effort needed to do a tasks, are determinant factors in the occurrence of most problems in the two industries. Moreover, both industries suffer from the Machiavellian attitude of controlling seniors and game publishers. In short, we have evidences allowing affirm the traditional software industry and games industry do not have essentially technological problems, but mainly management problems. In terms of differences between the two industries, perhaps an important problem, specific to the game industry, is the communication among the teams [6]. In the traditional software engineering, the team is usually relatively homogeneous, basically formed by technicians with skills on computer science. However, in the electronic games industry, a multidisciplinary team includes people with distinct profiles, as plastic artists, musicians, scriptwriters and software engineers. This mixture, in spite of been positive in the sense of having the more creative work environment, seems to produce a true split on the team, dividing it in to “the artists” and “the programmers” [6]. This division, that basically does not exist in the traditional software industry, is the main source of important misunderstanding problems [10], since both teams believe to communicate clearly when using their specific vocabularies to express their ideas [6]. Another important difference is that the game requirements elaboration is much more complex, therefore subjective elements as the “fun” does not have on efficient techniques for its determination, the point of Callele et al. [6] to identify clearly that it is necessary to extend the traditional techniques of requirement engineering to support the creative process of the electronic game development.
Figure 1: Occurrence of problems found interesting were highlighted. Then, using the categories described in section 3.2, the problems were classified to be tabulated 2 . When we analyze this results more closely, we see that the most cited problems were the unreal or ambitious scope and features creep with 75% (15 of 20) of projects reporting this problems. After that, demonstrating the coherence of the postmortems, 70% of projects citing the cutting features during development process. The other most found were problems in the design phase and delay or optimistic schedule, with 65%. Also the technological problems, with 60% can be highlighted. Figure 1 presents the histogram of occurrence of problems in sequence decreasing, in which we can make a graphical comparison of these results. Perhaps the most surprising result was that problems said to be “universal” in the game, as crunch time and over budget, had an occurrence rate relatively low, being that for the crunch time, only 45% of projects showed this problem. The over budget and loss of professionals problems had the lower incidence among all the analyzed problems, only 25%.
3.4
4.
CONCLUSIONS AND FUTURE WORK
In this paper, we presented a survey of problems found in the development process of electronic games. The electronic games industry, for its competitiveness and corporative way of working, generally turns difficult to access internal data from projects. For this reason, we have decided to collect information mainly from game postmortems and specialized litterature on game development. Our intention is to allowing a discussion about these problems and a comparison with respect to well-known problems in the traditional software industry. Our work shows that indeed all the main problems of traditional software industry are also found in the
Comparative between the traditional and electronic games industry
In last instance, to develop games is to develop software [2, 11]. The statement also insinuates that it is possible to 2
The final result of this tabulation can be found in a previous version of this paper published as technical report at http: //www.inf.ufrgs.br/~mpimenta/Pubs/se132.pdf.
710
5.
games industry, and it is possible to affirm that they are very related. In both contexts, for example, the unreal scope was pointed out as critical, as the problems with requirements definition. This contrasts with the one of so called universal problems of games industry, like crunch time, which had a relatively low occurrence rate, with 45% for crunch time. The problems with over budget and the loss of professionals have had the lowest incidence among all the problems analyzed, with a ocurrence rate of only 25%. Budget problems have a considerable discrepancy, probably due to the greater emphasis in the technological and management aspects given by the stories authors, minimizing the financial aspects of the project. We can note similarities with respect to classification (same problems with same importance) and frequency of some problems, as schedule delays and requirement problems, having almost equal rates. Thus, one possible consequence of this similarity is obvious: solutions adopted successfully to traditional software industry to solve some problems can be also adopted by the games industry. The main limitations that were found are the selection of postmortems to analyze and the subjectivity of reading. All postmortems selected have been written after the conclusion of projects, with at least a game delivered. Cancelled projects have not been analyzed, neither projects that, despite the efforts, have not produced a complete game for the market. This can have led to optimistical results in comparison with what reality shows, because certainly incomplete projects include many other possible perspectives and problems to explore. Postmortems do not have a formal structure or organization, being composed basically by stories in daily language and then the analysis of problems depended significantly on the text interpretation made by someone. Thus, different criteria of judgment could be adopted by different people. Trying to minimize this problem, our team have defined collectively a set of criteria to guide the analysis and classification activities. Finally, the results obtained in this work, as well as its limitations, open some possibilities for future works:
REFERENCES
[1] J. Bach. The challenge of “good enough” software. American Programmer, october 1995. [2] E. Bethke. Game development and production. Wordware Publishing, Plano, 2003. [3] A. Birk, T. Dingsoyr, and T. Stalhane. Postmortem: never leave a project without it. IEEE Software, 19, may 2002. [4] J. Blow. Game development: Harder than you think. ACM Press Queue, 1(10), February 2004. [5] F. P. Brooks. The Mythical Man-Month - Essays on Software Engineering. Addison-Wesley Professional, United States of America, anniversary edition edition, 1995. [6] D. Callele, E. Neufeld, and K. Schneider. Requirements engineering and the creative process in the video game industry. In 13th IEEE International Conference on Requirements Engineering, August 2005. [7] R. N. Charette. Why software fails. IEEE Spectrum, September 2005. [8] T. DeMarco and T. Lister. Waltzing with bears managing risk on software projetcs. Dorset House Publishing, 2003. [9] K. Flood. Game unified process. GameDev.net, may 2003. [10] J. P. Flynt and O. Salem. Software Engineering for Game Developers. Software Engineering Series. Course Technology PTR, 1 ed. edition, November 2004. [11] A. Gershenfeld, M. Loparco, and C. Barajas. Game plan: the insider’s guide to breaking in and succeeding in the computer and video game business. St. Martin’ s Griffin Press, New York, 2003. [12] W. Hamann. Goodbye postmortems, hello critical stage analysis. Gamasutra - The Art & Business of Making Games, July 2003. [13] M. Myllyaho, O. Salo, J. Kriinen, J. Hyysalo, and J. Koskela. A review of small and large post-mortem analysis methods. In IEEE France, Paris, November 2004. 17th International Conference Software & Systems Engineering and their Applications. [14] R. S. Pressman. Engenharia de Software. McGraw-Hill, So Paulo, 6 edition, 2006. [15] I. Sommerville. Software Engineering. International computer science seres. Addison-Wesley, London, 6th edition, 2001. [16] Standish Group. Chaos. Technical report, 1995. [17] E. Yourdon. Death March. Prentice Hall PTR, Estados Unidos, 2 edition, december 2003.
• Since it focused on problems, it is possible to perform the same study aiming at the best software development practices which are applied in the games industry. Maybe we can surprise ourselves again, but now with the number of good practices adopted. • An interesting work could be to attend the demand made by Flynt and Salem [10], developing specific techniques to improve the inquiry and verification of requirements in electronic games projects. • Listening to the recommendations of Callele et al. [6], to consider a software engineering methodology specialized in the domain of the games industry, contemplating aspects as complexity, multidisciplinarity and creativity.
711
