Evaluation of Game-based Learning approaches

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

Computing Track – Full Papers

Evaluation of Game-based Learning approaches through Digital Serious Games in Computer Science Higher Education: A Systematic Mapping Aliane Loureiro Krassmann Leo Natan Paschoal* Andressa Falcade Roseclea Duarte Medina Santa Maria’s Federal University (UFSM) – Computer Science Post-Graduation Program – Brazil *Cruz Alta’s University (UNICRUZ) – Computer Science Graduation – Brazil Abstract This paper conducts a Systematic Mapping in order to outline a current overview of digital serious games applications’ evaluations for game-based learning purposes in computer science higher education. The search was performed on informatics’ area electronic databases, resulting in 2,174 publications initially found. After sorting through the application of inclusion and exclusion criteria, 59 remained. In the final phase, after thorough reading of the extracted publications, it was reached 10 papers. The results reveal different practical approaches of digital serious games’ applications evaluations that assess its different aspects' influence on students. Keywords: serious games, education, computer science, evaluation, assessment, systematic mapping. Authors’ contact: {alkrassmann,leonatanpaschoal, andressafalcade, roseclea.medina}@gmail.com

1. Introduction Education it's been increasingly related to technological innovations, due to the society's informatization. Prensky [2012] highlights important changes on digital natives’ cognitive styles, such as processing information at a higher speed; ease of performing many tasks in parallel; preference for graphic than texts; preference for random than sequential access; besides being highly connected, active and fluent in several and different technologies. In this context emerge the serious games, which have the main and initial objective to serve a serious purpose, in particular, to education [Buchinger and Hounsell 2013]. These games are aligned with the current generation’s needs and learning styles. Besides motivating by fun, they are versatile and can be adapted to different disciplines, information or skills to be learned [Prensky 2012]. According to McGonigal [2012], games offer rewards that reality cannot provide, teaching, inspiring,

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

involving and uniting its members in a manner in which society often fails to promote. The Game-Based Learning (GBL) is defended by Squire [2007] as a way to engage students in meaningful activities that allow to assume new identities, explore new worlds and learn. According to the author, there is an emerging paradigm of GBL related to cognition theories, which argue that "we learn better when detained in activities aimed at goals in which we are significantly involved and invited to assume the identity of experts". Several researches, such as Raman et al. [2014], Bordini et al. [2014] and Rodríguez-Cerez et al. [2014], support the use of serious games as tools to motivate and to enhance learning. However, these innovative approaches lack methods and techniques for assessing its performance, in order to prove its benefits among students and to open space to other initiatives. As Klopfer et al. [2009] affirm, where educational games are embedded in a classroom setting, teachers have to assess how well students play, and if they extracted enough from this experience. This evaluation usually does not make sense through traditional ratings models. Teachers should consider qualitative methods such as essays, presentations and portfolios, which allow students to demonstrate learning outcomes that would not appear in conventional tests. Therefore, it is necessary to develop approaches, methods and/or techniques to evaluate serious games’ performance in its various aspects, especially the pedagogic one, essential to determine the educational technologies quality and the effectiveness of the gamebased learning approach. The choice for the computer science higher education scope took place regarding a proposed development, application and evaluation of a serious game on this context in the authors’ research group. This area differs from others by having students with more familiarity with networks and technologies such as digital games, which can make them more critical and demanding in their use.

16

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

In order to identify studies with this theme, this research conducted a Systematic Mapping (SM) aiming to outline a current overview of serious games applications’ assessments, in the computer science formal higher educational field.

2. Related Work It was sought out researches that also conducted SM regarding serious games’ evaluations, to present the relevance and to justify this study. However it was not found publications related to computer science higher education scenario. The closest ones to this study are synthesized below. The SM conducted by Buchinger and Hounsell [2013] assumes that some research has evaluated the influence of competition or collaboration for learning in serious games, but there are few studies about the use of both simultaneously. So, they focused on games that incorporate both collaboration and competition in its gameplay. From a list of 742 articles analyzed, 17 resulted in the SM. Due to the low number of publications found, the authors Buchinger and Hounsell, [2013] find it difficult to predict related search trends, but highlighted the use of games developed in the universities themselves. The study concludes that serious games aimed at education require learning effectiveness evaluation. The SM from Vargas et al. [2011] sought to discover the current state of serious games' quality initiatives, identifying gaps for the future. It was selected 112 works from 1,236 initially found in 6 digital libraries. The results showed that the papers about serious games’ quality have experienced an important growth, from three papers in 2007 to 34 in 2012. Vargas et al. [2011] found out that researchers are concerned to demonstrate or to confirm the efficacy of serious games in addition to their capacity to provide pleasure and entertainment, but little research has been carried out with regarding the gameplay features, which, according to the authors, make the serious games more effective tools. Although 28.57% of studies have produced a result that can be applied to any serious game, only 2.68% (3 studies) were validated. The SM conducted by Connolly et al. [2012] examined the literature on computer and serious games regarding the potential positive impacts on users aged 14 or more, especially to learning, skills enhancement and engagement. From the total of 7,392 publications found by Connolly et al. [2012], only 129 remained after the application of inclusion and exclusion criteria. Of these, the most common issues identified were the acquisition of knowledge, content understanding and

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

Computing Track – Full Papers

emotional and motivational results. The range of indicators and measures used in the work is discussed, along with methodological limitations and recommendations for future work in this area. As it can be seen in the works above, it’s been looking heavily for validation when it comes to the educational field. The studies here presented treated with different factors involving serious games’ applications, like collaboration and competition, learning efficacy and engagement. However, as a result of SM related to this research, there are few studies who clearly evaluate their approaches in terms of game-based learning elapsed with the serious games’ applications, highlighting the importance of this topic. This research differs from the others for having a scope specific for computer science formal higher education, and for looking only for digital games, not considering the serious games occurring without technology support, like card or table games.

3. Systematic Mapping As Kitchenham [2004] affirms, in the scientific production process, the bibliographic research summarizes and provides an overview of a knowledge area in face of its maturation. To achieve this goal, the Systematic Mapping (SM) emerges as an in-depth research that produces specific and detailed results. A SM allows to identify, to evaluate and to interpret the relevant material collected in a survey, forming an organized and reliable bibliographic database [Kitchenham 2004]. According to Kitchenham and Charters [2007], SM studies provide a research reports and results’ structure of what have been published by categorizing them. It often gives a visual summary, the results’ map. It requires less effort while providing a large overview. It’s been recommended mostly for research areas where there is a lack of relevant, high-quality primary studies. In the following subsections the SM held this research is presented. 3.1 Systematic Mapping Planning This research’s systematic mapping follows the principles of Kitchenham and Charters [2007] and it aims to identify what methods and/or techniques are being used to evaluate serious games’ applications for teaching purposes, i.e., GBL, focusing on higher formal computer science education. One of the characteristics that differentiate a systematic from a conventional literature review,

17

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

Computing Track – Full Papers

according to Kitchenham [2004], is the fact that systematic reviews start by defining an assessment protocol, which specifies the research question being addressed and the methods that will be used to conduct the evaluation.

should be discarded. So, five criteria for inclusion and six criteria for exclusion were stipulated, as shown in Table 1.

Adapted from the works of Buchinger and Hounsell [2013], Vargas et al. [2011] and Connolly et al. [2012], this SM question is defined as follows:

IC-1: Presents digital serious games’ applications in formal computer science higher education, for game-based learning purposes;

EC-1: It does not directly address the application of digital serious games in formal education;

IC-2: Discusses practical aspects of a technique, approach or implementation of digital serious games on formal computer science higher education, for game-based learning purposes;

EC-2: Technical reports, documents that are available in the form of summaries or presentations, as well as secondary literature reviews;

IC-3: Clearly presents the method or the approach used to apply digital serious games on formal computer science higher education, for game-based learning purposes;

EC-3: Full version unavailable;

IC-4: It presents an evaluation method of digital serious games applied in formal computer science higher education, for game-based learning purposes;

EC-4: Addresses only theoretical and philosophical aspects (without definition of technique, method, etc.);

IC-5: It provides clear results and the instruments used in the evaluation of digital serious games on formal computer science higher education, for game-based learning purposes.

EC-5: Does not have direct relation to serious games applications on formal computer science higher education, with game-based learning purposes;

"What methods, techniques, and/or frameworks are being used for evaluation of serious games applications for game-based learning purposes in higher formal educational computer science context?" The focus with this question is to find publications of serious games with pedagogical purposes, i.e., games that envision increasing the learning level of players, which evaluate its influence on students. The search string was built along the lines of the advanced web search tool Google Scholar [2015], consisting of derived keywords from the main question. This search engine works with a limited number of words, which resulted in the restriction of terms used in the string composition. Thus, the entered terms / defined in the search engine included: With all these words: education learning evaluation; With the exact phrase: Serious Games; With at least one of the words: "Digital Game" OR "Game-based" OR "Higher Education" OR "Computer Science". Where in the article: anywhere in the article. Period: 2010 to 2014. Language: English. The choice for the English language took in consideration the language with main publications dissemination around the scope, and the default language of the electronic databases (sources). The period definition occurred in view of reducing the scope, excluding 2015 due to not having as parameter the entire year. Already the choice of research sources took in account the works of Buchinger and Hounsell [2013], Zem-Lopes et al. [2014], and Borges et al. [2013] which also carried out systematic mappings in computer science field. Thus, the electronic bases target in this study were IEEE Xplore, Science Direct (Elsevier), Scopus, Slactions, SciELO, ACM Digital Library and SBGames. All of them are international bases except SBGames which is a Brazilian base, considered a reference in the country in this research’s context. The screening of publications occurred through the inclusion and exclusion criteria, formulated from the research question, aiming to stipulate parameters that would help on defining what is interesting and what

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

Table 1: Inclusion and exclusion criteria Inclusion Criteria (IC) Exclusion Criteria ( EC)

EC-6: Not clearly addresses the evaluation of digital serious games on formal computer science higher education, for game-based learning purposes.

3.2 Systematic Mapping Conduction The SM took place from January to April 2015 and was divided into seven stages, as describe as follows (Table 2). For this study it was used StArt [2015] and Mendeley [2015] tools, which facilitated the bibliographic importing process, selection and conduct of readings. Table 2: Systematic mapping conduction stages

Stage

Description

6

Sources definition, formulation of the research question and elaboration of the inclusion and exclusion criteria. Search string creation and application in Google Scholar web tool. Titles, abstract and keywords of publications found reading (selection phase). Full reading of selected publications in the previous step (extraction phase). Complete reinterpretation of publications extracted in the previous step (results). Systematization of data found.

7

Analysis, discussion and presentation of results.

1

2 3 4 5

18

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

Computing Track – Full Papers

The first stage consisted basically on the protocol composition in the planning phase, where the sources, the research question and the exclusion and inclusion criteria were defined. On the second stage the application of the search string was done in Google Scholar [2015], collecting the publications found. However, Scopus and Slactions directories returned zero (0) results, and therefore were disregarded. On the third stage, the publications from the electronic databases IEEE Xplore, ACM Digital Library, Science Direct (Elsevier) SciELO and SBGames were moved to the selection phase, where the title, abstract and keywords were read. The publications have been accepted or rejected accordingly to the inclusion and exclusion criteria defined in the SM planning. The resulting publications were included in the fourth stage, called extraction phase. On the fourth stage, the articles were fully read, confronting them again with the inclusion and exclusion criteria, conducting a new screening of publications acceptance or rejection.

through the application of inclusion and exclusion criteria, 59 remained. In the final phase, after thorough reading of the extracted publications, 10 papers were kept. The results reveal practical approaches of digital serious games applications that assesses its influence on students, on formal computer science higher education. The analysis will be held in the section 4. 3.3 Systematic mapping results In this subsection is carried out the result systematization of data obtained, in order to provide an overview of the SM. First, focusing on the selection phase, followed by the extraction phase. Figure 2 shows that most of the publications found in the initial search are from the SBGames base, 37% (807 publications). This can be attributed to the fact that this directory deal exclusively with the digital games topic. Second is the IEEE Xplore base, with 664 publications, composing 31% of the total. The SciELO electronic base returned the less matches, only 4, representing a slice lowest than 0% of total papers. The electronic bases Science Direct and ACM Digital Library had an 18% and 14% return, respectively.

On the fifth stage, the papers were finally extracted, that is, those who have attained the objectives of the SM and responded to the research question were accepted as a result. They were carefully read again, aiming to analyze and to discuss the final details of this research. On the sixth stage the results were systematized and organized to be presented in this study. The quantity of publications found per database at each stage of the SM can be seen in Table 3, which also shows other data as total number of articles in the initial search and duplicated ones. Finally, on the seventh stage a complete reinterpretation of publications resulted from the SM was made, aiming analysis, discussion and presentation of the results. Table 3: Systematic Mapping conduction summary Selection phase Extraction phase EB IR D I E I E IEEE ACM Elsevier SciELO SBGames Total

Table Legend:

664 299 400 4 807 2.174 EB IR D I E

17 9 4 0 0 30 – – – – –

39 6 13 0 1 59

625 293 387 4 806 2.115

7 0 3 0 0 10

32 6 10 0 1 49

Figure 1: Publications distribution per electronic base from the initial search

On the extraction phase, Figure 2 shows that the predominant number of publications belongs to the IEEE Xplore, with 39 of the total 59. Science Direct gets highlighted at this phase, receiving the second place with 13 publications. The ACM Digital Library and SBGames bases got 6 and 1 publications, respectively. SciELO database had no publications approved for the extraction phase.

Eletronic Base Initial Result Duplicated Included Excluded

The search for publications resulted in a total of 2,174 publications initially found. After sorting

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

Figure 2: Number of publications accepted to the extraction phase per electronic database

19

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

Computing Track – Full Papers

3

Still in the extraction phase, it was found that the incidence of publications, having as parameter the period between 2010 and 2014, is higher in 2012, followed by 2014 and 2013. This fact denotes a decline in the number of publications on the subject in 2013, but resumed in 2014, demonstrating the potential for research on evaluation of serious games in education and game-based learning, increasingly widespread and perfected. Much of the rejected publications, namely, that do not met the criteria set out in the SM, are related to the issues shown in Table 4. Table 4: Main themes of excluded publications by the SM Order Description (a)

(b)

(c)

(d)

(e)

(f)

Generic serious games solutions, which propose methodologies that can be used for various disciplines, without specifying the area used for the approach validation test; Models containing only hypotheses, prototypes and proposals that do not evaluate or make it in a subjective or superficial way, without explaining the method; Researches that point out only the design of the educational serious game, without developing it or applying it; Studies involving learn derived from developing a game, and not exploiting its gameplay in practice (game application); Approaches that address aspects as motivation and collaboration between students, and not the pedagogical aspect of serious games (influence on learning). Manuscripts describing the gaming application in other areas of knowledge such as health (dental, disease prevention) and some aimed at basic education content for the classroom.

The exclusion of many publications in the qualifying stages (selection and extraction) also occurred due to incomplete and / or theoretical or philosophical versions, meeting the EC-2 and EC-3, from Table 1.

4. Results Analysis This section provides a synopsis of the 10 articles resulting from the systematic mapping after the analytical reading thereof, summarizing its main aspects, such as objectives, methods and results. To clarify the understanding, the results were divided into 5 groups according to the type (focus) of evaluation undertaken on the serious games’ applications on computer science higher education context, as summarized in Table 5. Table 5: Main themes of publications resulted from the MS Order Evaluation focus Incidences 1

2

Comparison between traditional approach x GBL: Cheng and Su [2011]. Comparison between different serious games using the same methodology: Rais et al. [2011].

1

1

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

4

5

Analysis of cognitive improvements (before and after using the game): Karapinar et al. [2012], Zhang et al. [2013], Raman and Achutan [2014] and Potter et al. [2014]. Comparison of the same GBL approach at different educational levels: Hainey et al. [2011]. Analysis of GBL approach acceptability / applicability: Rodríguez-Cerezo [2014], AdamoVillani et al. [2013] and Terzidou et al. [2012].

Total of publications analyzed

4

1

3

10

4.1 Comparison between traditional X Gamebased Learning (GBL) approaches For this comparison it was considered the traditional approach as the adoption of expository educational practices/lectures. One publication was found that focus on the comparison between traditional and innovative approaches, through GBL, using serious games as a pedagogical tool. The study by Cheng and Su [2011] discusses the use of games relevance in learning from a comparative study between traditional teaching methods and the ones based on games. Therefore, the authors carried out the experiment in a systems analysis course with two groups, the first group working through games and the second one through the traditional method. The authors evaluated the efficacy of GBL through the ARCS (Attention, Relevance, Confidence, and Satisfaction) model, from Keller [1983], that it is a problem-solving approach that designs the motivational aspects of learning environments, measuring the change in student motivation. The results by Cheng and Su [2011] show that when there is the use of games, the students’ motivation have a significant impact on learning, allowing assert that GBL can achieve the learning goal effectively. This study evaluated the impact of a set of learning through a quasi-experimental approach (using control group), focusing on the motivational effectiveness of games. As it can see the comparison realized between traditional method and GBL emphasized the motivational factor, not clearly the cognitive influence. 4.2 Comparison between different serious games using the same method The paper of Rais et al. [2011] has among its objectives to improve Object-Oriented programming concepts (OO), using educational games with students from computer science, and identifying challenges in OO learning using games. From this perspective, the

20

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

study used two methods and two different serious games: 1. Method 1 - Traditional: students studied via books or laboratories; 2. Method 2 GBL - Game 1: GAPS 1.0 (GameBased Approach to Support Programming Skills): 2D game that trains basic programming skills, as 'object' and 'class'; 3. Method 2 GBL - Game 2: Alice 2.0: 3D game that introduces to programming, allowing students to drag and drop blocks to create a program. For the approach evaluation, Rais et al. [2011] applied a questionnaire based on Goal Question Metric (GQM), to collect the students’ opinions. It was concluded that 40% of students prefer to study OO via online tutorials, 30% prefer to study OO via games, and 30% in the traditional way. Thus, the authors conclude that joining games with online tutorials can be a good idea. Already the results of usability reveal that Alice 2.0 game performed better than the GAPS 1.0 game and the traditional method. The assessment on learning showed the weakest topics of the subject: inheritance, polymorphism and abstraction. Once more here it’s evaluated only the students' acceptability aspect, but comparing two different digital serious games, using the same method to display this approach. 4.3 Analysis of cognitive advances (comparison before and after games) Four publications focus on an analytical assessment of cognitive improvements observed by applying examinations before and after use of serious games. The research of Karapinar et al. [2012] develops a game proposal related to the programming discipline content "pointers", in order to test the students’ knowledge and strengthen explanations classes. The case study was conducted within a group of 10 students. After finalizing the content in class, they were invited to play the "Path Finder" game and then, to realize a classic examination relating to the content. It was observed that the students who lost in the game, also failed the exam related to pointers. As for the students who won in the game were those who passed the exam. Thus, the results of the game matched the examination. This result shows that further testing mechanisms such as games, work well and can therefore be used. Besides proving the cognitive advance after gameplaying, the study of Karapinar et al. [2012] found that games can be used as a substitute or complement for classic evaluative tests. The work of Zhang et al. [2013] aims to help students on understanding concepts of inheritance in Java language. The approach applied in the

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

Computing Track – Full Papers

programming course, was the before and after the game use test (pre and post-game). The results show that from the 14 students who participated in the tests, only 50% obtained notes above 60 after performing the first test. After the method application and the posttest, there was an increase of students with over 60 notes, 71% of them. Students also responded to nine multiple choice questions and to an open question. The overall students feedback was positive with most of the questions, highlighting the items "nice to play," "provides relevant feedback", "would like to see more concepts like this" and "would recommend to others." The study of Zhang et al. [2013] proves that the number of students with good grades had a significant increase after playing games, around 21% of the class. Moreover, students reported the nice experience obtained after the serious games’ application. Raman and Achuthan [2014] used the CyberCIEGE game scenarios, to analyze how effective it is the approach for teaching subjects and applications related to cybersecurity. Five game scenarios were selected based on parameters such as complexity, level of cybersecurity knowledge, and time to understand the scenario. For each scenario, 15 questions were created to address technical depth and difficulty levels. Students were divided into 2 groups of 10 members. While the first group began by trying to answer the questions, the other started playing the game scenarios, and then focusing on the questionnaire. These students, at baseline, had the same level of exposure to cybersecurity-related content, provided by the course. As a result the authors realized that there was a significant difference between the performances of the two groups, implying that the one which played the CyberCIEGE game got better learning performance of concepts, hitting questions in unanimity regarding the first group, in which just a few students hit the same questions. The work of Raman and Achutan [2014] clearly show the effectiveness of serious games as a learning tool. But the parameter used can be discussed, as the game group, in theory, had more time with the concepts (by game-playing). The research of Potter et al. [2014] aims to train and to teach concepts about the inspection process and software quality through the game Inspector X, investigating possibilities to transform the inspection learning in a more attractive and efficient task. The experiment involved the participation of 39 undergraduate students of computer science. The evaluation was divided into five main parts: a) an initial diagnosis; b) pre-test of knowledge; c) use of the game; d) post-test of knowledge; e) filling in a questionnaire about the game. The results show that there was a significant improvement of test results after the experiment, increasing in 1.37 to 4.88 (average). The survey also showed that the students’ impression

21

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

was very positive in relation to the learning capacity and immersion situation provided by the game. In the study of Potter et al. [2014] it becomes evident the cognitive influence of GBL approach on students, as well as its positive impression among them. It can be seen that, although they had different objectives, used different research designs, and evaluate different elements, the studies in this subsection showed positive results regarding the cognitive advance of students using the GBL approach, improving the learning rates. 4.4 Same GBL approach compared in different educational levels In the research of Hainey et al. [2011] is developed a game-based approach to teach collection and analysis of requirements in software engineering. The participants were 55 students from Higher Education (HE) and 37 students from Further Education (FE). The students were divided into two groups, one experimental group and one control group, both similar in quantity. The game is compared to the traditional methods of software engineering education using a pretest and post-test. The results show that the two groups (experimental and control) had a significant knowledge increase in post-test. More specifically, the study of Hainey et al. [2011] showed that HE students who participated in the experimental group had higher level of knowledge in the post-test compared to FE students, indicating that GBL approach was more effective in HE level than FE level. There was no significant difference in the game aspects’ ratings between the groups, indicating that it remained consistent and did not affect the results. Although the FE cannot really be considered a way of formal higher education, it was considered interesting to accept and analyze a study comparing the game application in different educational levels from computer science, showing that the GBL approach can be more efficient in the formal HE level. 4.5 Acceptability / Applicability analysis of the GBL approach

Computing Track – Full Papers

game, while the last two years included). Thus, the evaluation was carried out according to three different dimensions: students’ perspective, teachers’ perspective, and educational effectiveness’ perspective. The results indicate that the targeted to the exercise approach of "Evaluators" is likely cost-benefit extrapolation to other areas of computer science higher education. It showed a positive attitude from students towards serious games; and a positive effect of the system and its pedagogical strategy on students’ performance in the long run. Although the work of Rodríguez-Cerezo et al. [2014] refers to a system to develop serious games, and evaluates it through three different perspectives, the most salient assessment aspect is the positive acceptability about games from students. Adamo-Villani et al. [2013] presented a paper in order to teach information assurance concepts, especially operator precedence, by GBL approach, with the participation of 14 computer science and computer graphics technology students. It included two forms of evaluation: formative (game design) and summative (efficiency test), aiming to answer the following questions: (1) The levels of play are useful, interesting and visually pleasing? (2) Students consider the game useful for learning / review the concept of operator precedence? (3) How the game levels can be improved? It was collected quantitative data, such as time spent on gameplay activity, number of gameplay bugs, number of correct and incorrect answers, and completion and non-completion of the play levels; as well as qualitative data, such as student responses to a survey of scores for usability and visual quality of game's levels. The results from Adamo-Villani et al. [2013] show that the game is feasible, interesting and useful for learning and understanding the operator precedence concept. The participants’ reaction about the graphics quality was positive. Five students complained about the darkness of the environment and the difficulty to read the texts. Ten students responded that they would play the game level again. Three students mentioned that the game is an interesting way to learn rules of operators’ precedence.

Three publications aimed at analyzing the impact of the game approach among its users.

It can be seen in the study of Adamo-Villani et al. [2013] the nice reaction of students receiving an information assurance game, that had a good performance among them.

The approach by Rodríguez-Cerezo et al. [2014] seeks to assess "Evaluators", a system for the development of educational serious games targeted for introductory courses of computer language implementation. The audience was 75 students from Compilers discipline. It analyzed the educational effectiveness of "Evaluators" in terms of the grades obtained by students in the last five academic years (the first three years did not include the use of the

The study of Terzidou et al. [2012] aimed at investigating the effectiveness of 3D educational games in higher education, using the virtual environment Second Life and the language LSL (Linden Scripting Language), next to Multimedia course students. It was invited 20 students, divided into four groups to participate in the evaluation that included pre-test, playing game time and post-test. From the results it can be observed the satisfaction of

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

22

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

students in carrying out the activity. The majority of participants, 90%, reported that virtual worlds and games can be used for learning, with 45% of them saying the feeling obtained in the virtual world was pleasant. From the students approved 65% agreed with activity, 25% were neutral while the rest disagreed. Finally, 80% of students agreed with the inclusion of such activities because they felt satisfied and entertained, and that it facilitated on learning. The research of Terzidou et al. [2012] shows the positive receptivity of a virtual environment and game from students, what can be applied to other classes or disciplines. Although they had applied a pre and posttest, the results were no significant to frame it as cognitive advances category. 4.6 Results discussion This section discusses the results from the SM, pointing out the main directions from the knowledge produced by this research. Observing Figure 4 it can be inferred that the most studies’ focus identified by the SM are the ones which analyze the cognitive students’ progress after the GBL approach. On the second place appear assessments that compares between traditional approaches and approaches using GBL. Thirdly, the ones which verify the acceptability or applicability of GBL approach in its context or among its users. Tied with the lowest incidences are evaluations that make comparisons between different serious games or in between different levels of education, using the same method.

Computing Track – Full Papers

Networks and Computer Graphics, with 1 incidence each of them.

Figure 4: Computer Science disciplines covered in the serious games’ applications evaluations

This finding can clarify the main disciplines where applications of serious games in formal computer science higher education are being evaluated. This can be useful when deciding to use a paradigm of gamebased learning, for example, taking into account that programming disciplines are more debated and worked in this area, and so the approaches and methods for evaluation are more disseminated. Answering the SM question were found researches that used simplified methods and techniques, mostly empirical, for evaluation of serious games applications in computer science higher formal educational context. The resulting articles enable to affirm that this topic is diverse and immature, with restrict types of techniques, that assess restrict aspects of GBL approaches. No frameworks being used for this end were found, once more emphasizing the importance of this research line.

5. Conclusion

Figure 3: Evaluation focus of publications resulted by SM

This denotes the emphasis on demonstrating the educational potential of serious games, either through evidencing the cognitive students' advance after playing games or by comparing traditional approaches and game-based innovative ones. Another feature evidenced by the results is the predominance of articles that apply serious games in the disciplines of: Programming, with 5 incidences; Software Engineering, with 3 incidences; Computer

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

The emergence of serious games in education can contribute to increased motivation, socialization and learning of students around the world, but still lack up strategies to assess reliably the actions involving serious games. Although it have been found 2,174 publications about evaluations of game-based learning through digital serious games’ applications in formal computer science higher education, only 10 actually contained elements that propose in a clear and objective manner, an assessment of its approach to validate the experiment carried out. What can be concluded with the completion of this SM is that the research on serious games is growing; however, it needs to deepen more to be clear about the benefits of its aspects (assessment and approaches’ validation). It is evident the lack of clear methods for the different aspects’ performance measurement of educational serious games. It is observed that lack of methodological tools and goals for evaluation and validation the pedagogical aspect of serious games, i.e., demonstrating cognitive performance in students by checking possible advances derived from interaction with digital games, which can

23

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

be considered the main characteristic to motivate teachers to adopt this practice. As identified in this study, most ratings are subjective, empirical, based on user opinion. Although they were applied in computer science area, in none of the selected articles it seems that the mechanism for evaluating the approach was specific for computer science, i.e., none of them specifically presented the approach to evaluate de game among this specific public of students. Also, it was found a significant fluctuation in the number of publications, but a considerable increase in the last years, which can prove the growing trend of publications in the coming years on the subject. This may demonstrate a topic with great potential for academic research. To conclude it is clear the needing for metrics, processes or strategies for conducting an assessment and validation of the most significant approaches that effectively provide evidence of cognitive impairment from the use of serious games and the game-based learning. The results reveal different practical approaches of digital serious games’ applications that assess their influence in computer science higher formal education students, revealing a contribution to this scientific filed in terms of validation, in view of that technology itself and by itself cannot do a significant difference.

References PRENSKY, M., 2012. Aprendizagem baseada em jogos digitais. São Paulo: Editora Senac, São Paulo, Brasil. BUCHINGER, D.; HOUNSELL, M. S., 2013. Jogos Sérios Competitivo-Colaborativos: Um Mapeamento Sistemático da Literatura. II Congresso Brasileiro de Informática na Educação – CBIE. XXIV Simpósio Brasileiro de Informática na Educação – SBIE. DOI: 10.5753/CBIE.SBIE.2013.275 275.

Computing Track – Full Papers

RODRÍGUEZ-CEREZO, D.; SARASA-CABEZUELO, A.; GÓMEZALBARRÁN, J. L. S., 2014. Serious games in tertiary education: A case study concerning the comprehension of basic concepts in computer language implementation courses. Computers in Human Behavior, 3, p. 558–570. KLOPFER, E.; OSTERWEIL, S.; SALEN, K.. 2009. Moving Learning Games Forward: Obstacles, Opportunities and Opennes. The Education Arcade, Massachusetts of Technology. VARGAS, J. A.; GARCÍA-MUNDO, L.; GENERO, M.; PIATTINI, M., 2011. A Systematic Mapping Study on Serious Game Quality. Springer-Verlag Berlin Heidelberg. CONNOLLY, T. M., BOYLE, E. A., MACARTHUR, E., HAINEY, T., BOYLE, J. M. 2012. A systematic literature review of empirical evidence on computer games and serious games. In Computers & Education, 59, p. 661-686. KITCHENHAM, B., 2004. Procedures for performing systematic reviews. Technical Report Technical Report TR/SE-0401, Keele University and NICTA. KITCHENHAM, B.; CHARTERS, S., 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01. Software Engineering Group of Keele University Durham UK. GOOGLE SCHOLAR, 2015. Disponível em: https:// scholar.google.com.br/. Último acesso em 23 de junho de 2015. ZEM-LOPES, A. M.; PEDRO, L. Z.; ISOTANI, S., 2014. Qualidade de Softwares Educacionais Baseados na Web (Semântica): Um Mapeamento Sistemático. XXIII Ciclo de Palestras sobre Novas Tecnologias na Educação. BORGES, S. S.; REIS, H. M.; DURELLI, V. H. S.; BITTENCOURT, I. I.; JAQUES, P. A.; ISOTANI, S., 2013. Gamificação Aplicada à Educação: Um Mapeamento Sistemático. II Congresso Brasileiro de Informática na Educação (CBIE). XXIV Simpósio Brasileiro de Informática na Educação (SBIE).

MCGONIGAL, J., 2012. A Realidade Em Jogo - Por Que Os Games Nos Tornam Melhores e Como Eles Podem Mudar o Mundo. BestSeller.

START., 2015. State of the Art Through Systematic Review. LaPES – Laboratório de Pesquisa em Engenharia de Software. Universidade Federal de São Carlos – UFSCAR. Disponível em: http://lapes.dc.ufscar.br/tools/start_tool. Último acesso em 06 de abril de 2015.

SQUIRE, K. D. 2007. Games, Learning and Society: Building a Field. Educational Technology, p. 51-54, sep/oct.. p. 52-53.

MENDELEY., 2015. Free Reference Manager and PDF Organizer. Disponível em: https://www.mendeley.com/. Último acesso em 06 de abril de 2015.

RAMAN, R.; LAL, A.; ACHUTHAN, K., 2014. Serious Games based approach to cyber security concept learning: Indian context. Green Computing Communication and Electrical Engineering (ICGCCEE).

CHENG, C.; SU, C., 2011. A Game-based learning system for improving student’s learning effectiveness in system analysis course. Elsevier Ltd. Selection and/or peerreview under responsibility of Prof. Hüseyin Uzunboylu. DOI: 10.1016/j.sbspro.2011.12.122. World Conference on Learning, Teaching & Administration, WLCTA.

BORDINI, R. A.; FREITAS, P. A. G.; OTSUKA, J. L.; NUNES, A. P. A.; BEDER, D. M.; SANTIAGO, D. L.; FONSECA, L. F.; SANTIAGO, G. L. A.; OLIVEIRA, M. R. G., 2014. Avaliação do Protótipo de um game educacional de Música. Nuevas Ideas en Informática Educativa TISE, FortalezaCE, Brasil.

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

RAIS, A. E.; SULAMAIN, S.; SYED-MOHAMAD, S. M., 2011. Game-based Approach and its Feasibility to Support the Learning of Object-Oriented Concepts and Programming. 5th Malaysian Conference in Software Engineering (MySEC).

24

SBC – Proceedings of SBGames 2015 | ISSN: 2179-2259

Computing Track – Full Papers

KARAPINAR, ZEHRA; SENTURK, ARAFAT; ZAVRAK, SULTAN; KARA, RESUL; ERDOGMUS, PAKIZE., 2012. A Game to Test Pointers: Path Finder. International Conference on Information Technology Based Higher Education and Training (ITHET). ZHANG, J.; CALDWELL, E. R.; SMITH, E., 2013. Learning the Concept of Java Inheritance in a Game. The 18th International Conference on Computer Games – CGAMES. POTTER, H.; SCHOTS, M. S.; DUBOC, L.; WERNECK, V., 2014. Inspector X: A Game for Software Inspection Training and Learning. CSEE&T. HAINEY, T.; CONNOLLY, T. M.; STANSFIELD, M.; BOYLE, E. A., 2011. Evaluation of a game to teach requirements collection and analysis in software engineering at tertiary education level. In Computers & Education, 56, p. 21– 35. ADAMO-VILLANI, N.; HALEY-HERMIZ, T.; CUTLER, R., 2013. Using a Serious Game Approach to Teach ‘Operator Precedence’ to Introductory Programming Students. 17th International Conference on Information Visualisation. TERZIDOU, T.; TSIATSOS, T.; DAE, A. D.; SAMARAS, O., CHASANIDOU, A., 2012. Utilizing virtual worlds for game based learning: Grafica, a 3D educational game. 12th IEEE International Conference on Advanced Learning Technologies. KELLER, J. M. 1983. Motivational design of instruction. In C. M. REIGELUTH (Eds.), Instructional-design theories and models: An overview of their current status (pp. 386-434). Hillsdale, NJ: Lawrence Erlbaum Associates.

XIV SBGames – Teresina – PI – Brazil, November 11th - 13th, 2015

25