Usability Testing for Educational Computer Game Using Observation Method Norizan Mat Diah 1, Marina Ismail2, Suzana Ahmad3 and Mohd Khairulnizam Md Dahari4 Faculty of Computer and Mathematical Sciences Universiti Teknologi MARA Shah 40450 Alam Selangor, Malaysia
[email protected],
[email protected],
[email protected],
[email protected] Abstract— For most children the first contact with the computer is through some sort of educational game. Therefore, it is important that computer games for children are well-designed and usable for the children. In order to evaluate the usability of the computer game, children must be observed while exploring and playing the game and given the chance to express their opinions. Therefore, to involve children effectively, appropriate evaluation techniques are needed. This paper will discuss an observation method that has been adopted in a usability testing with the children. In this study, a computer game called Jelajah is chosen as the game to be tested for the usability evaluation with children. Jelajah is suitable for the experiment because it is an adventure game which is a very common game genre for children. Five preschool children aged between five and six years old are selected for this study. During the usability testing session, facilitator observes the child and fill in the observation checklist form in order to track the effectiveness and efficiency of the game. After the test session completed, the child is required to answer a simple interview-based questionnaire to know their level of satisfaction after playing the game. Data collected are both qualitative and quantitative. The collected data are analyzed to determine the levels of effectiveness, efficiency and satisfaction. The levels of effectiveness, efficiency and satisfaction are being measured by averaging them to a single metric that can represent the usability of Jelajah. The result obtained showed that the approach adopted to conduct a usability testing of a computer game with children using Jelajah can also be adopted for evaluating any educational computer game with children. Keywords— Usability; Observation method
Educational
computer
game;
I. INTRODUCTION Nowadays, computers are not just being used by domain experts such as engineers and programmers. They are now used by almost everyone including children who want to get exposed to computers and information technology at an early age. For most children the first contact with the computer is through some sort of educational game [1]. Therefore, it is important that computer games for children are well-designed and should have a good usability. The term usability is used to denote that a design is good from Human-Computer Interaction (HCI) point of view and most efforts by HCI researchers and practitioners are ultimately aimed at achieving more usable computer systems [2]. In general, usability is the extent to which a product or a system enables users to achieve specified goals. Usability is also the ability of a product or system to effectively and efficiently fulfill the needs and specifications of users.
usability lifecycle in their software development lifecycle. Various usability models and methods have been proposed to help in comparing the usability levels attained by different organizations [3]. II. USABILITY TESTING Usability is one of the most important quality characteristics of a product or system. Usable systems are often easy to learn, efficient, not error-prone and satisfactory in use [4]. Usability brings many benefits, which include increased productivity, enhanced quality of work, reductions in support and training cost, and improved user satisfaction [5]. Usability has not been defined consistently, and various definitions exist. There is not one agreed upon definition of usability, and usability certainly cannot be expressed in one objective measure. Several researchers have proposed definitions and categorizations of usability but there is still some consensus on the concept and they mostly differ on more detailed levels [6]. One of the well-known usability definitions is by Jacob Nielsen which stated “usability is a quality attribute that assesses how easy user interfaces are to use, making it possible to the customers to develop tasks in a clear, transparent, agile and useful way” [7]. Nielsen considers that the usability of a system can have five quality components: 1) Learnability: How easy is it for the users to accomplish basic tasks the first time they encounter the design? 2) Efficiency: Once users have learned the design, how quickly can they perform tasks? 3) Memorability: When users return to the design after a period not using it, how easily can they reestablish proficiency? 4) Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors? 5) Satisfaction: How pleasant is it to use the design? However the definition from ISO 9241-11 which stated usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [8]” becoming the main reference in usability. This definition represents a wide scope of usability, emphasizing how the usability of a product relates to its context of use [9].
Today, many organizations have now started incorporating
978-1-4244-5651-2/10/$26.00 ©2010 IEEE
157
TABLE 1 USABILITY AS ISO 9241-11 AND JACOB NIELSEN ISO 9241-11 Efficiency Effectiveness Satisfaction
the usability test, debriefing the participant and analyzing the data of the usability test (Figure 1).
Jacob Nielsen Efficiency Learnability Memorability Errors / Safety Satisfaction
Table 1 shows the definitions mapping between ISO 9241-11 and Nielsen’s as pointed out by [9]. From the table, it was proved that Nielsen and the ISO standard give a concise outline of the term usability. There are two major considerations when conducting usability testing. The first is to ensure that the best possible method for testing is used. Generally, the best method is to conduct a test where representative participants interact with representative scenarios. The facilitator collects data on the participant’s success (effectiveness), speed of performance (efficiency), and satisfaction. The findings, including both quantitative data and qualitative observations information, are provided to designers in a test report. Using ‘inspection evaluations’ in a well-controlled environment, usability tests must be done with caution. The second major consideration is to ensure that an iterative approach is used. After the first test results are provided to the designers, they should make changes and then have the product tested again. Generally, the more iteration on usability testing, the better the product [10]. III. OBSERVATION METHOD Observation method is the evaluation method that collects data by observing users’ experiences with a product [1]. Often, observational usability evaluations are performed to determine quantitative measures like efficiency, effectiveness, and satisfaction [8]. This method has been proven by many researches as one of the best technique that could be used in usability testing. The approach is to have a session with end users, and observe them while performing a task. During observation, a representative user is asked to perform several authentic tasks using the prototype or instructional resource. A video camera captures what is happening on the users screen, their interaction and navigation of the product, and also the comments they made when interacting with the product. Additional video cameras can be used to record the users’ facial expressions, body language, keyboard and mouse, or any other important part of the user environment. Upon completion of each test, users will need to answer post interview like questionnaire about other subjective inputs regarding the product such as ease of use and levels of satisfaction. IV. METHODOLOGY OF THE RESEARCH There are nine activities involved in the methodology used in this research. Those are planning a usability test, identify usability model, preparing the test, selecting a representative sample, ethical issues in recruiting participants, conducting
Fig 1. Methodology of usability testing on Jelajah A. Planning a Usability Test Planning process is an essential part of a usability testing. Careful planning will ensure fully understanding and smooth process progress during usability testing. In this step objective, goals and problem statement will be indentified. B. Identify Usability Model In order to identify usability model, we need to set dimensions of usability testing, identify the courseware elements and identify methods of usability testing to be used. 1) Dimensions of usability testing This study will be based on usability definitions and characteristics done by ISO 9241-11 which will focus on measuring the effectiveness, efficiency and level of satisfaction from children. 2) Identify the suitable method Observational evaluation has been chosen as the method to be used in the usability testing. The usability testing will be conducted in a kindergarten. Each session should not be more than 20 minutes. The sessions will be facilitated and observed by only one facilitator. 3) Children need to be encouraged to participate and facilitator should stress the value of the child’s input and show appreciation and gratitude. It is extremely important to make the child feel at ease[11]. For this
158
reason, testing will be done in the kindergarten classroom (familiar environment) with the aid of the teacher. When testing with young children facilitator notes are crucial, rather than relying on verbal comments, expressions such as smiling or sighing often are used to judge the quality of interaction. C. Preparing the Test Materials The following is a list of materials that need to be prepared when running the usability test in this research: 1) Select test facilitator Only one facilitator will handle the usability test with children in this study. The facilitator will sit next to the child during the session and his/her role is to fill in the observation form while interacting with the child to make them feel at ease. 2) Screen the questionnaire There are two questionnaires that need to be assessed which are demographic questionnaire and post task questionnaire. The demographic questionnaire shall be completed by parents in order to know the child’s backgrounds, family environment, experiences with computer and what are their ordinary tasks using computer. Post task questionnaire which is administered as an interview will be conducted right after each test session with the help from the facilitator. The post task questionnaires contain questions that are supposed to capture subjective input from the child that cannot be captured using the task-based questions. This includes; the child’s feeling, fun, ease of use and their satisfaction level towards the game. 3) Set the test environment The usability testing session will be done in a kindergarten classroom. To prepare for the test environment, a computer, a camera and the screen recording software is required. This is the minimal setups needed to run the usability test with users. Below are summary of items needed for the test environment: • Computer: The Jelajah game will run on the computer and children will need to control the game using the keyboard hooked to this computer. • Sound devices: Sound devices are needed in order to play the sound and background music of the game. This is believed could add more attraction for children to play the game. • Video camera: Video camera is needed to record the behaviors and expression of the child while playing the game. • Screen recording software: A software called CamStudio will be used to record screen activities during the test. • Observation Form: It is crucial to have an observation form during usability test session. The document shall contain several important
documents such as parent consent form, demographic questionnaire, observation form and checklist, and post test questionnaire. 4) Identify task scenarios Task scenarios for this usability test will be based on the game’s instructional task and not predefined task that children need to accomplish since this is irrelevant to evaluate games. According to [12], games are entertainment and have to be fun. This has nothing to do with accomplishing tasks as quickly or as efficiently possible. Hence, our test scenarios are actually the game instructions which help children to understand on how the game should work and what they should do. The task scenarios have been divided according to several screens in Jelajah which are; starting screen, character selection screen, exploration game screen and word puzzle game screen. 5) Observation checklist Observation method needs facilitator to record all children action, behaviours and facial expressions while observing children playing the game. In order to guide facilitator, an observation checklist is needed. The checklist should be created based on functionalities of the game and what suppose to be done by players in ideal ways. Every checklist item should be constructed using positive phrasing. D. Selecting a Representative Sample Effective representative sampling is crucial to collect reliable data during usability test. Not only should test participants reflect the characteristics of the targeted users of the product or service, but they also must be likely to use it. Only then will their experiences and opinions yield reliable data for identifying meaningful improvements [13]. The first step in recruiting young participants is sending information about our study to the correct places. Unlike adults, recruiting children for a research study is complicated because we cannot seek permission directly from the children. Parents must first trust our organization and the legitimacy of our research before they allow their child to participate in a study. Usability studies can become expensive when it comes to designing and selecting users. [14] said, “ The best users come from testing no more than five users and running as many small tests as you can afford” Nielsen’s study showed that a group of five users were able to find about 80% of the findings in a system and as more users were added, less additional information was found, but more and more money was spent to run tests and to compensate additional users. The idea behind the assumption is that you can learn more from a group of five completing multiple tests than you would on 15 participants completing one test.
159
Conforming to Nielsen’s study; the usability testing is conducted with five children with different background, mixed gender and aged five to six years old. E. Ethical Issues in Recruiting Participants [15] suggests that, before conducting any usability test we must do the following: •
• •
• •
Receive signed consent from a parent or guardian before any child under the age of 18 may participate in a usability test – This ensures that the study meets ethical guidelines and standards. Carefully explain the purpose of the research to younger participants to assure that they are not confused and do not feel manipulated in any way. Ensure that child participants are protected at all times during the usability test – Parents are allowed to wait for their child outside the testing room if they so choose. Schedule a break halfway through the test session and remind participants that they can stop at anytime. Having videotape constantly recording during the session offers protection for everyone involved.
For this study, parents’ permission is obtained prior to conducting the usability test with the children. Parents were asked to complete a demographic survey and a consent form for their child’s participation. Parents were explained that the purpose of the test is to test the software and not the children. Parents were also ensured that this test will endangered their children in any situation. Building a good relationship with participant is also another important issue. A successful test session with children or adults are usually ones in which we have been able to establish a good relationship [15] . [11] conducted a series of ‘friendly sessions with her testers prior conducting the usability test with them. The ‘friendly session’ consists of activities that are enjoyable to children such as sing-a-long and storytelling. Having a good rapport with test participants puts them at ease, and helps them to feel more comfortable sharing their thoughts and insights than if they do not trust the test facilitator. Prior to the test, the facilitator visited to the kindergarten to meet the children and build a friendship with them. This process assisted facilitator while conducting the test as the children already know him. F. Conducting the Usability Test The usability test is conducted in the children’s own familiar environment that is the kindergarten. It is conducted in a closed-session situation where only the students as the tester and facilitator were in the room to test the Jelajah game. During the testing period, the child was allowed to play the game with minimal intervention or assistance from the facilitator. Two cameras were used to capture the whole session while a software is used to record the cursor
movement on the screen. Children are required to complete task scenarios that is the game. Data that are collected are both quantitative and qualitative data. Data are collected while observing the child performing the task scenarios. In observing the child performing the task scenarios, the facilitator must be aware on every aspect that the child might show such as their body language, facial expressions and comments. The facilitator needs to record all verbal comments and complete the observation checklist. G. Debriefing the Participant After completing the usability test session with a child, facilitator needs to ask the child to answer post task questionnaire. Facilitator will need to take notes on feedback or comments from the child on how they feel about the game. The children’s answers are later translated into a 3 points Likert scale format according to the Coding Scheme (PeKep) as introduced by [11]. This technique is important to measure the satisfaction levels of a product. H. Analyzing the Data of the Usability Test Both qualitative and quantitative data collected from the test sessions need to be analyzed further in order to have a better understanding about the usability of a product. Based on the ISO9241-11 definition that defines usability as consisting of three components: 1) Effectiveness: The accuracy and completeness with which customers achieve specified goals. 2) Efficiency: The accuracy and completeness of goals achieved in relation to resources. 3) Satisfaction: Freedom from discomfort and positive attitudes toward the use of the system. It is obvious that effectiveness and efficiency have objective characteristic, while satisfaction has the subjective characteristic to be measured. Based on this understanding, the effectiveness and efficiency metrics are measured using a simple usability measure introduced by [16] by analyzing user’s success rate. [14] defined success rate as the percentage of tasks that users complete correctly. This is a coarse metric. It does not tell about why users fail or how well they perform the tasks they completed. Success rates are easy to collect and measure. If users cannot accomplish their target task, all else is irrelevant. In this study, user success can be derived from the observation checklist which is used by the facilitator while performing usability test and the measurement of satisfaction are taken from post questionnaire with the children. However, the success rate calculation is not discussed here. V. CONCLUSION
Conducting a usability test requires a careful planning. Testing with children as the participant requires extra handling in terms of human and social interaction because the children cannot speak for themselves. This paper has laid down a comprehensive approach to conduct a usability testing with
160
children as the test participant for an educational game. Testing an educational game requires evaluation of the instructional effectiveness, efficiency and satisfaction. The study conducted with the children concluded that this approach can be adopted to test a usability of an instructional application.
[1]
[2] [3]
[4] [5] [6]
[7] [8] [9] [10] [11] [12] [13] [14] [15] [16]
REFERENCES W. Barendregt, M. M. Bekker, and M. Speerstra, “Empirical evaluation of usability and fun in computer games for children.”, in Proceedings of Human-Computer Interaction INTERACT-03', IOS Press, Zürich, Switzerland, 2003. pp. 705-708 J. Löwgren (1995), Perspectives on Usability. Department of Computer and Information Science, Linköping University. Sweden. [Online] http://webzone.k3.mah.se/k3jolo/Download/r-95-23.pdf, W. Barendregt, M.M. Bekker, D.G. Bouwhuis, and E. Baauw, (2006). “Identifying usability and fun problems in a computer game during first use and after some practice.”, in International Journal of HumanComputer Studies, 2006, pp 830-846. vol 64(9), J. Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993 ISO 9126-1, Software engineering-Product quality- Part 1: Quality Model. 2000 A. Seffah, J. Gulliksen and M. C. Desmarais. Human-Centered Software Engineering - Integrating Usability in the Development Process, Human-Computer Interaction . New York: Springer-Verlag, 2005 J. Nielsen (2003, August 25). Usability 101: Introduction to Usability. Alertbox, [Online] http://www.useit.com/alertbox/20030825.html ISO 9241-11 , Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability.1998. M. V. Welie , G.C. van der Veer, and A. Eliëns “Breaking down Usability.”, In Proceedings of Interact 99, 1999. pp. 613-620 S.J. Koyani, R.W. Bailey RW, and J.R. Nall (2003). Research-based Web design and usability guidelines.[Online] www.usability.gov/pdfs/guidelines.html. Marina Ismail. 2009. A multimedia courseware (BACA) for learning Bahasa Melayu for preschool using the Vygotsky’s approach. PhD Thesis. Universiti Kebangsaan Malaysia. B. Zaman, “Evaluating games with children.”in Proceedings of Interact 2005 Workshop on Child computer Interaction: Methodological Research. 2005. D. Hinderer , “Challenges in participant recruiting for usability testing,” in Proceedings Professional Communication Conference, 1998. IPCC 98. 1998, pp 23-25 J. Nielsen (2000, March 19). Why You Only Need to Test with 5 Users. Alertbox [Online] http://www.useit.com/alertbox/20000319.html. M. Patel and C. Paulsen, “Strategies for Recruiting Children for Usability Tests” . in Proceedings Usability Professionals Association, 11th Annual Conference.2002. J. Nielsen (2001, August 5). First Rule of Usability? Don't Listen to Users, Alertbox, [Online] http://www.useit.com/alertbox/20010805.html,
161
