Heuristics Evaluation Strategy for Mobile Game-Based Learning

The 6th IEEE International Conference on Wireless, Mobile, and Ubiquitous Technologies in Education

Heuristics Evaluation Strategy for Mobile Game-Based Learning Syamsul Bahrin Zaibon

Norshuhada Shiratuddin

College of Arts & Sciences Universiti Utara Malaysia Sintok, Kedah, Malaysia [email protected]

College of Arts & Sciences Universiti Utara Malaysia Sintok, Kedah, Malaysia [email protected] reach the application objectives. There are a lot of usability evaluations which are those initially developed by [4], [5], and currently the list of usability heuristics as described in [6]. Some modifications and additions have also been made to the evaluation which also considered being useful [7]. These heuristics however more focused on the general applications not specific to game. Games, however differ from general applications where games are most enjoyable and fun when they provide sufficient challenge for a player [8]. Therefore, the evaluation should be focused on the playability aspect. Besides, to be specific in mobile game, another concerning aspect needs to be considered is the mobility issue. A set of heuristics have been proposed by Korhonen and Koivisto [8], Koivisto and Korhonen [9] who have introduced playability heuristics for mobile games. These heuristics can be used for evaluating any mobile game which consists of three modules: Game Usability, Mobility, and Gameplay. In addition, in specific to games for learning, Malone has developed the first heuristics for evaluating educational games [10] that based on three categories: challenge, fantasy, and curiosity. The main purpose of the heuristics is to serve as a checklist for designing enjoyable user interfaces. It is considered to use these heuristics as described in [6], [8], [9], [10] for mGBL evaluations, but not really feasible because they were not described in specific to mGBL. Therefore, this study proposes a strategy of heuristics evaluation of mGBL. The heuristics are adapted from Korhonen and Koivisto [8]; Koivisto and Korhonen [9] by adding a new component which is a learning content that would overcome the learning content in mGBL.

Abstract— Evaluation of learning media is necessary for determining the effectiveness of the produced media. When evaluating mobile game-based learning (mGBL), conventional usability heuristics evaluations lack comprehension and difficult to be directly implemented. Therefore, heuristics evaluation strategy is proposed to evaluate specifically for mGBL. The strategy consists of four components: Game Usability, Mobility, Game Play, and Learning Content. Each of the components represents the issues to be considered and evaluated for mGBL. Additionally, in this study, a prototype of mGBL was developed and evaluated by utilizing the proposed strategy. The results indicate that the strategy is useful and potential to be implemented for similar mGBL applications. Keywords-mGBL; heuristics evaluation; mobile game-based learning; mobile game

I.

INTRODUCTION

mGBL can be defined as a game for learning purpose that utilizes mobile technologies for playing platform. Such technologies are mobile phones, smart phones, personal digital assistant (PDA) and other handheld devices. The main concerning issues of mGBL are mobility and restrictions on mobile technologies. mGBL concepts are grounded in pedagogical theory and adjusted to the technical capabilities of current standard mobile phones [1]. A successful example project is proposed by Mitchel et al. [2] who have proposed the three year pan-European funded project, which prototyped mGBL in three sectors: i) e-health, ii) e-commerce, and iii) career guidance. The project was based on research findings that have been conducted by Mitchell [2]; Mitchell and Savill-Smith [3]. Their findings show that mGBL is considerable potential for promoting and encouraging learning in mobile learning environment. mGBL applications developed for learning need to be evaluated in terms of a few aspects such as usability, mobility, game play, and learning content. These aspects should be further considered for heuristics evaluation strategy. However, current literatures still lack of heuristics evaluation for mGBL applications. II.

A. Components of Heuristics Evaluation Strategy The heuristics evaluation strategy can be implemented for evaluating any mGBL which consists of four components: Game Usability (GU), Mobility (MO), Game Play (GP), and Learning Content (LC). The ten GU components (Table I) describe the interface and game controls which the player interacts with the game. Game interface allows player to play smoothly and reacts based on user actions. In general, good game usability ensures that the player have interest to play the game until the end. Next, in Table II, the MO consists of three components which concern about the issues that affect mobility of the game. Mobility can be defined as the easiness of a player to enter to the game world and the accessibility of the game at anywhere and anytime.

HEURISTICS EVALUATION STRATEGY FOR MGBL

Heuristics evaluations are developed for evaluating the effectiveness of an application which commonly in usability aspect. The usability evaluation is conducted to users in order to find out how the users easily and efficiently can

978-0-7695-3992-8/10 $26.00 © 2010 IEEE DOI 10.1109/WMUTE.2010.27

127

TABLE I. No. GU1 GU2 GU3 GU4 GU5 GU6 GU7 GU8 GU9 GU10

In the process of understanding the user preferences, a preliminary study was conducted. The analyses of the study further support the development of the mGBL prototype. The main objectives of this study are to find out the specific target audiences for mGBL and their preferences in learning. Basic statistical method was used to assess the student responses which were based on descriptive technique. Two months (between August to September 2008) were allocated for the data collection period. The targeted samples were among students at Malaysian secondary schools. They were randomly selected in different background of school types which range from urban to rural areas and public to boarding schools. As illustrated in Table V, about 61.9% of the respondents are female and the remainder male. As for race composition, the majority of the respondents were Malay (81.4%), while the rest were Chinese (9.1%), Indian (8.1%) and other races (1.3%) that include Aborigines and Siamese. Some of the results have been discussed further in [11]. The results found that majority of the respondents with 73.9% have access to mobile phone. However, it was noted that fewer respondents (aged 13) did not have access to mobile phone as compared to other group of ages.

MOBILITY COMPONENTS

Mobility Components The game and play sessions can be started quickly The game accommodates with the surroundings Interruptions are handled reasonably

The ten GP components (Table III) specifically describe how the game is playable, run smoothly and consistent, meaningful, and not bored to player. The GP is important because it is dynamic and occurs when the player interacts with the game mechanics and rules. Lastly, the LC components as listed in Table IV are specifically concentrated on the learning content. The LC components should provide informative, useful, and understandable content to the users when playing the mGBL. TABLE III. No. GP1 GP2 GP3 GP4 GP5 GP6 GP7 GP8 GP9 GP10

No. LC1 LC2 LC3 LC4

TABLE V.

Ages

GAME PLAY COMPONENTS

Game Play Components The game provides clear goals or supports player created Thel player sees the progress in the game and can compare h players l The are rewarded and rewards are meaningful The player is in control Challenge, strategy, and pace are in balance The first-time experience is encouraging The game story supports the gameplay and is meaningful There are no repetitive or boring tasks The game does not stagnate The game is consistent TABLE IV.

STUDY ON MGBL PREFERENCES

III.

Game Usability Components Audio-visual representation supports the game Screen layout is efficient and visually pleasing Device UI and game UI are used for their own purposes Navigation is consistent, logical, and minimalist Control keys are consistent and follow standard conventions Game controls are convenient and flexible The game gives feedback on the player’s actions The player cannot make irreversible errors The player does not have to memorize things unnecessarily The game contains help TABLE II.

No. MO1 MO2 MO3

GAME USABILITY COMPONENTS

DEMOGRAPHICS PROFILES OF RESPONDENTS

- 13 (Form 1) - 14 (Form 2) - 15 (Form 3) - 16 (Form 4) - 17 (Form 5) Total

Gender Male Female 71 95 12 94 30 62 65 65 47 50 225 366

Total 166 106 92 130 97 591

Next question in the instrument, regarding whether they play mobile games, 437 students answered (see Fig. 1). Most of them (69.8%) reported that they play mobile games (n = 305); of these, 40% players were males, and 60% were females. In addition, it can be seen that girls (68%) like to play mobile games as much as boys (72%). Here, it can be concluded that female students like playing mobile games.

LEARNING CONTENT COMPONENTS

Learning Content Components The content can be learned easily The game provides learning content The learning objective from the game is achieved The content is understandable

85

No Yes 47

183

In implementing the strategy, a series of study has been conducted which comprises a user preference survey and mGBL prototype development. The prototype was evaluated using the proposed heuristics evaluation strategy, and the results were then further discussed.

122

Male 1

Female 2

Figure 1. Play mobile games

128

Next, when the students were asked about the time spent for playing mobile games, as seen in Table VI, most of the students stated that they play mobile games between 6 to 15 minutes (42%); and less than 5 minutes (21.5%). The results show that mobile games are usually played in a tiny time in a casual way. This is noted that a successful mobile game should be designed in a simple game play, easy game rules, and uncomplicated learning content. TABLE VI.

(PECIPTA 2009) in Kuala Lumpur, Malaysia. Guests who visited to our booth were randomly invited to play the 1M’sia mGBL and at the end of the three-day expo, 80 guests tried the mGBL. All guests played the 1M’sia mGBL using the mobile phones that were provided to them. Some of them played the game by giving guidance from the researcher. The questions (based on 5 Likert scales) were asked spontaneously to the users which were based from the four components of the heuristics evaluation strategy - game usability, game mobility, game play, and learning content (as depicted in Table I to IV).

TIME SPENT TO PLAY MOBILE GAMES

Time Spent Less than 5 mins Between 6 to 15 mins Between 16 to 30 mins Between 31 mins to 1 hour Between 1 to 2 hours More than 2 hours Total

Percentages 21.5 42.0 19.5 8.5 4.2 4.2 100.0

V.

Generally, the results in the study are consistent with previous studies (for example in [12]; [13]; [14]; and [15]). The results reveal that a majority of the surveyed students have access to mobile phone. Most of them played mobile games, and female students played more than males. The finding also disclosed that, in order to make the mGBL successful in learning environment, it should be easily played in a short time. Consequently, this study provides evidence that there is a huge potential in implementing mobile game for educational purpose, and indication that mGBL is preferred to be useful in learning general knowledge. Therefore, a mGBL prototype is developed based on the findings of the study. A mGBL prototype about local learning content that could foster the concept of 1Malaysia (http://www.1malaysia.com.my) is produced. The game is named 1M’sia which is abbreviated from one Malaysia. In general, the game is generated into two game plays which are simple quiz and mix-and-match. The 1M’sia mGBL is aimed at demonstrating the values and challenging knowledge capabilities of the players of 1Malaysia values. The players’ skills and knowledge will determine how well they are able to do the right things, and the values will either be mastered or not. At the end of the game, the player is shown to the score of the level of their 1Malaysia concept comprehension. IV.

FINDINGS AND DISCUSSION

This section presents the findings of the heuristics evaluation strategy and analysis of the main results. The discussion is intended to highlight the key issues that arise from the responses obtained. First, the demographic profile of the users who participated in the study as illustrated in Table VII. Out of 80 users, 45% of the respondents are male and the remainder female. In age range, the majority of the respondents were 13 and above (71.25%), and most of them from age 16 (25%). Fig. 2(a), 2(b) and 3 show the user evaluation study taking place at the expo. TABLE VII.

DEMOGRAPHICS PROFILES OF USERS

9 10 11 12 13 14 15 16 17

Ages

Total

Gender Male Female 3 2 4 2 2 3 2 5 4 8 6 4 2 3 7 13 6 4 36 44

Total 5 6 5 7 12 10 5 20 10 80

EVALUATION SESSIONS AND METHODS

To illustrate the implementation of the heuristics evaluation strategy, mGBL prototype was utilized. The methods as implemented in [16] were adapted which gave users to evaluate the 1M’sia mGBL. The methods were chosen because being prepared in a natural setting while users playing the mGBL. In addition, these methods have a better sense of users without having any formal circumstances. The evaluation sessions have been conducted in October 2009 during the International Exposition of Research and Inventions of Institutions of Higher Learning 2009

(a)

(b)

Figure 2. Users Playing and Evaluating 1’Msia mGBL

129

5 4 3 2 1 0 GP1 GP2 GP3 GP4 GP5 GP6 GP7 GP8 GP9 GP10 Figure 3. Researcher Conducting the Evaluation

Figure 6. Game Play

The first component of the evaluation is game usability. As seen in Fig. 4, overall mean = 3.7 and the highest score is GU6 (Game controls are convenient and flexible). In general, the results showed that the 1M’sia mGBL is at average in term of its game usability aspects.

Lastly, the component that is important for learning objective is the learning content component (Fig. 7). It is obvious that the overall mean= 4.0 which indicates that the learning content is quite informative, understandable and easy to learn. The highest score is LC4 (The content is understandable).

5 4

5

3

4 3

2

2

1

1

0

0

GU1 GU2 GU3 GU4 GU5 GU6 GU7 GU8 GU9 GU10

LC1

LC2

LC3

LC4

Figure 4. Game Usability Figure 7. Learning Content

Second component is the mobility issue. Fig. 5 shows that overall mean = 3.3 and this can be expected that the 1M’sia mGBL suits the mobility components. This is because the game was designed to be in mobile platforms.

VI.

In general, heuristics evaluations were proposed for evaluating application. Commonly, the evaluation is conducted to users in order to find out how usable the application to the users. However, from the literatures, current heuristics evaluations are not really feasible to be implemented for mGBL because of the mobility aspects, game features, and learning content issues. Therefore, in this study, a heuristics evaluation strategy is developed for evaluating the effectiveness of mGBL application. The strategy is intended to evaluate mGBL to users in the aspects of game usability, game mobility, game play, and learning content (as listed in Table I to IV). In order to analyze these aspects, 1M’sia mGBL was developed based on user’s preferences study, run the engine for its execution in mobile phone and then conducted the evaluation sessions using the proposed strategy. Overall, the evaluation sessions were successfully conducted and employed to the 1M’sia mGBL. It was simple to conduct, and provided an easy heuristics evaluation of similar concept of mGBL applications. Overall, the 1M’sia mGBL is at average in term of its game usability aspects and suits the mobility components. The game also was expected by users to be more adventurous rather than simpler version of the

5 4 3 2 1 0 MO1

MO2

CONCLUSION AND FUTURE STUDY

MO3

Figure 5. Game Mobility

Next component is the game play component. Fig. 6 depicts that the overall mean = 3.6. The highest score is GP4 (The player is in control) and the lowest is GP6 (The firsttime experience is encouraging). Although the scores were not consistent, the game was expected by users to be more adventurous rather than simpler version of the mGBL.

130

mGBL, and the learning content is quite informative, understandable and easy to learn. Prospective works can be suggested for this study, for example the heuristics evaluation strategy should be extended to include further data analysis such as hypothesis testing and correlation analysis. The analyses can perhaps provide more accurate findings and discussions. In addition, the evaluation strategy can be conducted to other mGBL applications.

[6] [7]

ACKNOWLEDGMENT

[10]

[8]

[9]

This project was awarded two gold medals in two different expositions, at the PECIPTA 2009 expo in Kuala Lumpur and at the Seoul International Invention Fair (SIIF) 2009 in Korea respectively. The authors also are thankful to Universiti Utara Malaysia for providing support as grant-inaid for this study.

[11]

[12] [13]

REFERENCES [1]

[2]

[3] [4] [5]

N. Mohamudally, “A massive multiplayer game framework for mobile learning”, Proc. IEEE Intl. Workshop on Wireless, Mobile and Ubiquitous Technology in Education (WMUTE '06), IEEE Computer Society, Nov. 2006, pp. 23-25. A. Mitchell, “Exploring the potential of a games-oriented implementation for m-Portal”, Proc. MLEARN 2003 Learning with mobile devices, May. 2003, London, UK, pp. 105–115. A. Mitchell, and C. Savill-Smith, “The use of computer and video games for learning: A review of the literature”, LSDA, London, 2004. J. Nielsen, and R. Molich, “Heuristic evaluation of user iInterfaces”, Proc. Computer Human Interaction, 1990, pp. 249-256. J. Nielsen, “Enhancing the explanatory power of usability heuristics”. Proc. Computer Human Interaction, 1994, pp. 152-158.

[14]

[15]

[16]

131

J. Nielsen, “Usability engineering”, Academic Press, London, 1993. M. J. Muller, A. McClard, B. Bell, S. Dooley, L. Meiskey, J. A. Meskill, R. Sparks and D. Tellam, “Validating and extension to participatory heuristic evaluation: Quality of work and quality of work life”, Proc. Computer Human Interaction. 1995, pp. 115-116. H. Korhonen and E. M. I Koivisto, “Mobile entertainment: Playability heuristics for mobile games”, Proc. of MobileHCI, ACM Press, 2006, pp. 9-16. E. M. I. Koivisto and H. Korhonen, “Mobile game playability heuristics’’, Nokia Corporation, 2006. [Online]. Available: www.forum.nokia.com [Accessed: Aug. 20, 2008]. T. W. Malone, “Heuristics for designing enjoyable user interfaces: Lessons from computer games”, Proc. ACM SIGSMALL Symposium and the first SIGPC Symposium, 1980, pp. 162-169. N. Shiratuddin and S. B. Zaibon, “Local content game: the preferred choice for mobile learning space”, Proc. of the ASCILITE: In Same places, different spaces, Auckland, 2009, pp. 684-694. K. S. Lee, “More youngsters using handphones”, New Straits Times, January, 25, 2006, pp. 6. M. S. Norbayah, and M. S. Norazah, “Mobile phone usage for mlearning: comparing heavy and light mobile phone users”, CampusWide Information Systems, vol. 24, no. 5, 2007, pp. 355-365. Business Wire, “When it comes to mobile games, women play and men pay”, Yankee Group, Jul. 1, 2004. [Online]. Available: http://findarticles.com/p/articles/mi_m0EIN/is_2004_July_1/ai_n609 1852 [Accessed: Aug. 20, 2008]. A. H. Hafizullah, H. A. Siti and A. W. Rozali, “A survey on mobile games usage among the institute of higher learning (IHL) students in Malaysia”, Proc. IEEE Intl. Symp. on Information Technologies and Applications in Education, 2007, Kunming, China, pp. 40-44. N. Shiratuddin and S. B. Zaibon, “Mobile games based learning (mGBL) with local content and appealing characters”, Int. Journal of Mobile Learning and Organization, vol. 4 no.1, Inderscience, USA, 2010, pp. 55-82.