Adapting usability testing for oral, rural users - ACM Digital Library

CHI 2011 • Session: User Studies/Ethnography in Developing Regions

May 7–12, 2011 • Vancouver, BC, Canada

Adapting Usability Testing for Oral, Rural Users Trina Gorman1, Emma Rose2, Judith Yaaqoubi2, Andrew Bayor1, Beth Kolko2 1 2 Literacy Bridge Department of Human Centered Design & Engineering 1904 Third Avenue, Suite 733 Box 352315, University of Washington Seattle, WA 98101, USA Seattle, WA 98195, USA [email protected], [email protected], [email protected], [email protected], [email protected] ABSTRACT

users and how it can be improved. While usability best practices and standards are well established [2, 3], what is lacking from the literature is a thorough understanding of how these approaches translate into diverse contexts. Furthermore, the little research that has been done suggests that many traditional approaches either do not transfer well within these settings [4, 5] or have to be modified [5, 6].

Traditional usability methods are of limited use when evaluating systems designed for distant, diverse populations. In this paper, we describe a study conducted in two Ghanaian villages that evaluated an audio computer designed for people living in oral cultures. Informed by ICTD and orality-grounded HCID, we modified existing usability testing practices and we reflect on the utility of these adaptations. We found that conducting a culturally appropriate study often meant forgoing more traditional approaches in favor of flexible, opportunistic methods. We acknowledge the challenges of adapting traditional usability methods for oral, rural users. However, we found that by implementing strategic modifications led by local staff, our study produced valuable, actionable results.

What follows is a summary of a study we conducted with oral, rural people. This work is informed by Information and Communication for Development (ICTD) literature including orality-grounded HCID as proposed by Sherwani, et al [5]. Sherwani is among the small group of researchers who stress the challenges of designing for oral users [7-9]. In particular, Sherwani says it is arguable whether traditional usability methods are of much analytical value.

Author Keywords

Below we discuss traditional usability approaches and how our study differed. We then elaborate on where the adaptations were successful (or ineffective) and what we learned as a result. Our intention is that lessons learned will be instructive to others conducting usability research with oral users, rural communities, and when devices are shared among users. We use the term ‘oral’ instead of ‘illiterate’ because it highlights a person’s ability. Or, as Sherwani says, because “terms like ‘illiterate’ devalue the identity and knowledge of oral cultures by implicitly suggesting that lack of literacy is equivalent to ‘backwardness.’”

user-centered design, usability testing, standards, illiteracy, oral cultures, methodologies, audio technology, ICTD. ACM Classification Keywords

H.5.1. Information interfaces and presentation: Multimedia Information Systems. General Terms

Design, Experimental, Human Factors, Languages INTRODUCTION

There is a growing acknowledgement that technology can assist with poverty alleviation efforts. However, as products are created by an eclectic group of people—from nonprofit organizations and for-profit companies to academics— designers face new obstacles in being able to iterate and evaluate for distant populations [1]. Distance, in this regard, can be geographic, cultural, or cognitive, and operative gaps can include issues of language, literacy, disability, and others. Techniques like usability testing become especially challenging when working with people who are distant in one or more of these ways. Usability testing is an accepted method in Human Computer Interaction (HCI) research used to determine how well a product works for its intended

USABILITY STUDY: THE TALKING BOOK

To share locally relevant information with and among oral people who live without electricity, Literacy Bridge created the Talking Book (Figure 1). The Talking Book is a handheld, durable, battery-powered device that enables users to create and listen to recordings and copy recordings between devices. To access recordings, users are guided by audio prompts, to which they respond with key presses. For Figure 1: instance, right and left arrows The Talking Book navigate through categories (e.g. health, agriculture, stories) and once in a category, the up and down arrows rotate through individual messages.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00.

k

Our study took place in two Ghanaian villages (Charia and Ving Ving), which are representative of Literacy Bridge’s

1437



Location and Test Equipment

target population. Each village has between 1000 and 1500 residents, almost all residents are farmers, and literacy levels are low (~10%). In terms of infrastructure, Charia has intermittent electricity and mobile network connectivity; Ving Ving has no electricity and weak mobile connectivity in a few areas.

Usability studies are often conducted in a lab that mimics a familiar location, such as an office or a home. Labs enable researchers to repeat conditions consistently and to use equipment (such as one way mirrors) to collect data. For our study, a “familiar” location meant outside in a village, typically under a tree or in a common area (Figure 2).

Goals

Our goal was to improve the Talking Book to make it easier for oral people to use. Specific research questions included: 1) Are the audio system instructions unclear or hard to follow? 2) Which button icons are more difficult to decipher and navigate with? 3) What tasks or parts of the information architecture are more challenging for users? Challenges

Conducting usability studies for the Talking Book presented a variety of challenges. First, the target users for the device are oral and therefore think, learn, and process information differently compared to those who have been taught to read and write [5]. This meant that while conducting the study and considering solutions, we had to try to understand the thought processes of oral people, which can be radically different from our own. Second, the target users also have limited exposure to technology. As a result, most users needed a detailed introduction as well as training prior to the study—making it difficult to determine if a usability problem was the result of the device or the training method. Third, this device was designed to fill the global need of getting verbal information to people in developing regions. Therefore, we could only consider changes that would benefit the entire targeted demographic. Lastly, we had a diverse team including members with local expertise and members with usability expertise, but no members with both skillsets.

Figure 2: Typical setting for a usability study

This setting made it more difficult to ensure privacy. Passersby would sometimes stop to see what we were doing, which was at times distracting to both facilitators and participants. However, the location also created unique learning opportunities. For example, one day we had too many participants who wanted to be involved and it was inappropriate to ask them to leave. After brainstorming as a team, we split up and were able to explore more scenarios with a more diverse group compared to our initial plan. This setting also meant that our data collection methods (video, note taking) were visible to users—a practice that is commonly avoided. We stopped using a still camera almost immediately as it was disruptive. However, using a small, inconspicuous video camera was less noticeable and enabled us to view the study after the fact. In the future we plan to use the Talking Book to capture keystrokes as well.

To overcome these challenges, Literacy Bridge’s staff from the United States worked with usability researchers to create an initial plan that considered traditional approaches. We then altered those approaches using existing literature and our field experiences [1, 5, 12]. Next, we worked with local staff who live and work in the region to adapt our plan to be culturally appropriate. ADAPTED METHODOLOGIES FOR USABILITY TESTING

So while there were drawbacks to this uncontrolled setting, we found that being creative with existing resources ensured our team had an effective work setting and our participants were in a familiar environment.

Study team

Participants and Testing in Groups

In traditional usability methods, trained facilitators ensure sessions are consistent and that data is collected in a systematic way. In our study, we asked local staff who had no previous usability experience to facilitate sessions [5]. We trained them on basic usability techniques, the goals of our study, and on orality-grounded HCID. Local staff then took ownership of the study while we played more of an advising role. We chose to favor cultural appropriateness over methodological expertise. This helped us ensure the findings were based on problems with the device and not on the misinterpretation of cultural norms.

Typically, in more developed regions, users are given a single product to use and sessions are conducted 1:1 so the facilitator can view the user’s progress without other influences. However, in Literacy Bridge’s programs, users often share Talking Books and teach one another how to use them. Therefore, we conducted our sessions in groups to simulate how users experience the device in practice. Our groups included men and women between the ages of 20 and 35. None of the adults we worked with were literate; all were farmers and had little to no access to technology. We also worked with smaller numbers of students and the elderly. To recruit these groups, we approached the village

1438


chief to discuss our plans and gain his support. Not only was this necessary to show respect, but we also learned of existing rivalries in the village; his involvement meant we did not exacerbate problems by favoring one group over another. Working with the chief also helped us assemble participants who knew each other, which proved to have pros and cons. For example, having users from the same social group (family, close friends) helped ensure they were comfortable—participants interacted socially, joking and laughing at times. However, one woman was teased by her husband because he thought she should be learning more quickly. She became self-conscious and lost confidence in her ability. When forming groups, researchers should anticipate negative consequences like this, particularly for those who tend to be marginalized, such as women.


users trained each other privately; users took the devices home after our session to continue practicing and learning from each other. While each approach had drawbacks, we had success with both facilitator and peer training methods. Peer training away from the facilitators proved to be particularly successful. This could be because participants felt more comfortable in their homes or away from the team. When we led the training, it was necessary to continually adapt. For example, when requesting that a participant perform a task, it was important to alter the task so users did not just memorize button presses. The method that was least effective was having new users train each other in front of the group—perhaps because of performance anxiety or because the selected users were simply not strong teachers. In the future, we will try ‘training the trainer’ to see if brief guidance would improve their training methods. Peer training and facilitation need more investigation, but are promising techniques for others to consider.

Introduction and Training

It is common practice for facilitators to give users only a brief introduction to a product and limit training. Since our users were unfamiliar with technology, including the Talking Book, we needed to thoroughly introduce the device and train users on how to use it.

Facilitation

Typically, to elicit naturalistic behavior, facilitators avoid intervening when a user struggles with a task. During our study, we often did the opposite and encouraged users to help each other, which aligns with how devices are used in practice. While this approach made it difficult to track the learning pace of users, it had various benefits. For example, it allowed the facilitator to learn a better way to explain something. Peer assistance also revealed when participants did not fully understand the device when he/she offered incorrect assistance. When it was necessary for the facilitator to intervene, he/she did not just offer minimal prompts as is typical, but rather thoroughly taught the task as before—explaining it in a slightly different way or adding a helpful tip. Since redundancy is an important part of oral communication [5], it is likely that the repetition of the training and facilitation method helped users to learn.

As Sherwani et al., point out, oral people do not internalize new information the same way as literate people. Based on our experience conducting studies in other developing contexts, we have also seen firsthand participants struggle with abstract introductions and scenarios. Therefore, the facilitator explained the device as a concrete solution to a local problem [5]. For example, the facilitator spoke of how complicated farming can be; participants agreed that each crop variety required detailed instructions. The facilitator then asked if they were satisfied with extension visits; participants said no, officers did not come often enough. The facilitator then explained how they can get relevant information from the officer through the Talking Book—at which point they saw value in the device and were interested in learning more.

The think aloud protocol (TAP) is a common technique, but one that was not feasible in our study for a variety of reasons. We were concerned that thinking out loud would make participants uncomfortable since they would be expected to articulate confusion in front of peers. This would be the case for any device that is shared among users. In addition, Talking Book users must listen to the instructions to operate it. Having users talking while the device was talking would have been chaotic—an important realization for orally-delivered information technology.

In addition, we found that a compelling introduction was important because it affected a person’s desire to use the device and therefore how quickly they learned. For example, one elderly woman appeared unmotivated and did not even seem to try. Following the session, she said she did not need to learn because her grandson could relay the information to her. Therefore, specifically tailoring and/or personalizing introductions was important—particularly for those who were intimidated by technology. After introducing the device, we tried two training methods—both of which were centered around repeated practice and apprenticeship, which is how oral people learn [5]. First, the facilitator taught interactively through demonstration. After performing the task repeatedly and explaining each step, he/she asked a participant to perform the task as the others watched. The facilitator repeated until all tasks had been covered and all users had used the device. Second, we tried ways of having participants adopt the role of trainer: users trained each other in front of the group;

Data Collection

In this study, collecting data was more difficult than in traditional studies not only because of the location, as previously mentioned, but also due to the varying skillsets of our team: local staff did not have usability experience; staff from the United States did not speak the local dialect. To work around these restrictions, local staff took notes in English focusing primarily on qualitative measures (underlying reasons for problems, meaningful social

1439


dynamics). English speaking staff found it particularly helpful when note takers wrote down verbatim quotes from users. They also found that success indicators were often unnecessary because they could typically follow a user’s progress through facial expressions and tone of speech. Another technique that helped bridge the language divide was when facilitators made brief comments in English during the session or when English speaking staff asked questions to clarify something that had occurred. This helped ensure all staff were involved in learning and could help alter plans to take advantage of learning opportunities.


effective to the extent they are driven primarily by local staff, based on a thorough understanding of orally-grounded HCID, and when researchers implement flexible, opportunistic approaches to best adapt to the local setting. REFERENCES

1. Putnam, C., et al., "Adapting User-Centered Design Methods to Design for Diverse Populations," Information Technologies & International Development, vol. 5, 2009. 2. Dumas, J. and Redish, G., A Practical Guide to Usability Testing. Exeter, UK: Intellect, Ltd. , 1999.

CONCLUSION

3. Barnum, C., Usability Testing and Research. New York: Longman, 2002.

What worked well in our study was working with local staff to alter methods in two primary ways. First, we mimicked how the device is used in practice by incorporating familiar surroundings, group interaction, and peer training. Second, we prioritized cultural appropriateness over avoiding bias in how we facilitated sessions and collected data.

4. Winschiers, H., Fendler, J., "Assumptions Considered Harmful," in Usability and Internationalization, ed Berlin: Springer, 2007, pp. 452-261. 5. Sherwani, J., et al., "Orality-Grounded HCID: Understanding the Oral User," Information Technologies and International Development, vol. 5, pp. 37-49, Dec 16 2009.

Many of the adaptations we chose meant our study was less controlled than a traditional study. Although the adaptations had drawbacks, users consistently struggled with the same issues, which over time revealed the ways the device could be improved. Through this work, we made the system instructions slower in pace and more conversational where the tone was too computerized. We also simplified the information architecture to be more consistent and less hierarchical. As we anticipated, users had difficulty locating the button icons for which there was not a word in their dialect. However, agreeing on new icons among users was more challenging than we envisioned. More research is needed to be sure any changes are indeed an improvement compared to the existing design.

6. Medhi, I., Sagar, A., and Toyama, K., "Text-free user interfaces for illiterate and semiliterate users," Information Technologies and International Development, vol. 4, pp. 37-50, 2007. 7. Sambasivan, N., et al., "Intermediated technology use in developing communities," Proceedings of the 28th international conference on Human factors in computing systems, pp. 2583-2592, 2010. 8. Medhi, I., Prasad, A., and Toyama, K., "Optimal audiovisual representations for illiterate users of computers," Proceedings of the 16th international conference on World Wide Web, p. 882, 2007.

In terms of our methodology, our experiences show that common best practices like 1:1 training, strict adherence to consistency, and TAP need to be reevaluated according to setting constraints. This study also made us question whether researchers should redefine the variables used to measure success in these contexts. For example, we propose the following two variables for others to consider: • Memory retention. How do we evaluate a user’s success when we are less concerned with efficiency (such as time on task) and more concerned with the extent to which the user remembers how to use a product? Should future usability studies collect longitudinal data and, if so, how? • Training methods: How do we understand what helps users learn and how such findings can improve the product itself? Should we accept flexible training methods, and if so, how can we maximize what we learn about the product’s usability while ensuring valid results? • Testing in groups: How do we train and test when users use devices in group settings? We conclude that usability studies need to be localized to suit each unique context, especially for products like the Talking Book that are designed for a global audience. As usability research progresses in this area, studies will be

9. Medhi, I., Gautama, S., and Toyama, K., "A comparison of mobile money-transfer uis for non-literate and semiliterate users," Proceedings of the 27th international conference on Human factors in computing systems, pp. 1741-1750, 2009. 10. Putnam, C., Kolko, B., Rose, E. & Walton, R. (2009). , "Mobile phone users in Kyrgyzstan: A case study of identifying user needs and requirements," in Proceedings of International Professional Communication Conference (IPCC '09), Honolulu, Hawaii, 2009. 11. Kolko, B. E., Wei, C, and J.H. Spyridakis. , "Internet Use in Uzbekistan: Developing a Methodology for Tracking Information Technology Implementation Success," Information Technologies and International Development, vol. 1, pp. 1-19, 2003. 12. Schmidt, C., Gorman, T., Gary, M., Bayor, A. "Impact of Low-Cost, On-Demand Information Access in a Remote Ghanaian Village." Information Technologies and International Development 2010. London

1440