Multilingual Web Usability Testing - A MUST Methodology Ankur Gupta, Amit Sangroya, Vasudeva Varma International Institute of Information Technology, Hyderabad, India
[email protected]
Abstract— Online marketers and retailers provide websites in various languages through translation in order to penetrate into local markets. Translation causes issues in terms of ease of navigation for the user and hence affecting the usability of these websites. Usability testing is a technique to refine the website by tracing problems in it, with the help of experiments, involving real users. The standard structure of usability testing becomes very rigorous in multilingual scenario and does not offer clear insights for evaluating multilingual websites. We attempt to develop an approach towards multilingual web usability evaluation and provide a perspective that reduces the level of abstraction in multilingual usability testing scenario. We term our paradigm of Multilingual Usability Testing as MUST.
I. I NTRODUCTION Online retailers and marketers offer websites in multiple languages to ensure their presence globally. These multilingual e-commerce websites offer channels to reach local customers in an efficient and effective manner. Websites provide an interface to the user to express his needs which is understood by the company if the expression of that need is comfortable and clear. As Steve Krug [1] in his intriguing work reveals that we use the web for scanning, satisficing, and muddling through and says that people won’t use your web site if they can’t find their way around it. System quality attributes determine its acceptability among the potential users which in general also lead to system reengineering. These attributes have no correlation with system functionality [2]. Usability is a quality attribute of the website which measures the ease with which users interact with the website. Formally, usability gauges the effectiveness, efficiency and satisfaction with which specified users achieve specified goals while using a software product. The most commonly cited factors that determine the usability of a website are the ease of navigating the site, the efficiency of design and the degree of error avoidance [3]. Multilingualism adds another dimension to the usability of an application by incorporating the cultural context to which the user belongs. By providing multilingual websites, companies intend to penetrate into the local markets in a more effective and efficient way. Companies provide websites in the native language of the user in order to better serve his needs. To optimize resources, companies resort to translation of these websites from one language, usually English, to the native language of the user. Website translation from one language to another doesn’t result in mapping of cultural norms belonging to user’s native language. As a result, the target audience doesn’t feel at home with the website in the translated language as the users of the original language are while using it. In general, the usability of a website
is considerably influenced by the quality of its language and ultimately its compliance with the culturally determined metaphors, attitudes and preferences of its target groups [4]. To ascertain that an application adheres to the expectations of the user, its evaluation is conducted. Usability testing is an approach to evaluate usability of an application in a systematic and structured way. In the realm of testing websites for usability, it offers a framework for determining the inhibitions caused in navigation and the ease of use to the target user. Usability evaluation of websites in the present state-of-the-art offers methodologies to test usability of a generic website while does not incorporate its multilingual feature. Evaluation of multilingual websites should be customized and refined keeping multilingualism of these applications in perspective. There should be a way of looking at things when a multilingual website is considered for evaluation. Conducting usability testing for each language and thereby ensuring the usability level of the entire web application is resource intensive and fails to exploit the multilingual nature of these applications. It’s like missing the forest for the trees. How to approach multilingual usability testing in a more systematic and structured fashion? To start with, we should have a way of looking at things, a paradigm, while undertaking the task of multilingual evaluation. From the commencement of the process of evaluation till its final completion, this perspective should be our framework for organizing aspects of user’s interaction with multilingual website. The correlation of usability with cultural context is firmly established by the inspiring work of Jacques Nantel and Evelyne Glaser [5]. Hence, the impact of language and culture should be incorporated and accounted for in any such paradigm. Usability testing in general is conducted by following few standard steps - objective definition, experimental set-up and so on. The literature of usability evaluation does not suggest a way to customize this procedure to reflect the multilingual aspect of the website. Moreover the care that we should take at each step of the process of multilingual website evaluation has not been elucidated in the research done so far. In this paper, we attempt to provide a perspective for multilingual web usability evaluation: • We describe the extensive survey of various websites which resulted in MUST methodology in section I. This study was also the source of motivation for us to design and articulate this paradigm. This study highlights the spectrum of issues that we can encounter in our process of evaluation. • The proactive knowledge of factors influencing the task
•
at hand aids in controlling the process in an effective and efficient way. We precisely explain the various aspects of MUST methodology in section III. This paradigm is an initiative to provide more structure to multilingual web usability evaluation and it’s an attempt at reducing abstraction level involved in this process. It, however, is not an exhaustive approach and has limitations. Finally we present the steps of MUST methodology in section IV. II. W EB S URVEY
To explore the issues and challenges in the evaluation of multilingual websites, we undertook a survey of few multilingual websites from various domains. We consider websites from various sample categories like news portals, ecommerce websites, logistics, e-banking and search engines to illustrate their specific features. We approach each website in these categories according to the nature of target audience and their variation across multilingual spectrum. The analysis of features of these multilingual websites help us in developing our approach towards multilingual web usability testing. This paradigm can then be leveraged when we systematically undertake the problem of evaluating their usability. For any category of websites, their target audience can be classified under two headings: ethnic users and international users. Ethnic users are those who belong to different regions of the same country while international users hail from different countries. The translation of multilingual websites from one language to another in order to provide the same set of services to culturally different target audience affects its usability. The degree of the effect depends on the distance in the cultural context of multilingual users. Hence, the most conspicuous factor determining the success of any multilingual website is the incorporation of cultural context that integrates its target audience. The structure, design and content of the website should be finessed so that the interface of the website doesn’t bewilder the user. To hone the website, its evaluation needs to be conducted through experimentation involving the sample users from these target users. International users of a website tend to be more remote in terms of culture than users of different ethnicities. Also, it is more cost-effective to find and collect the sample users from different ethnicities at a particular geographical location than it is in the case of a sample constituting international users. A. News Websites We surveyed three multilingual news websites Yahoo, Rediff and BBC. News websites in general vary in design because of prevailing customization for each language. Websites reporting news are the most dynamic and highly targeted in terms of the content. Multilingual news websites are separate from each other in terms of layout, design and content due to extensive localization. News in a particular language would be drafted including the cultural context of the users it is meant for. Here issues arising out of
translation of content are minimal because of predominant localization. However design and layout contain the major usability issues. Since the extent of localization is maximum in these websites, there is little correlation among them across languages. Considering this variation, these multilingual websites would demand exhaustive usability testing in each language. However the classification of target users among ethnic and international users still holds for these websites. We can assume that content localization in these websites already accounts for cultural context and hence only design needs to be validated for ease and efficiency of use by the user. We concentrated on three languages Hindi, Tamil and Telugu for Yahoo, Rediff and BBC as these languages belonged to three different ethnicities within India. Also, a survey could be easily conducted as sample users of these three languages are available in our lab. When testing usability in terms of design of these news websites, we obtained various critical incidents reported by the user for Yahoo and Rediff where the elements of the design obstructed the ease and efficiency of locating various categories of news items in these websites. However in the case of BBC, the simplicity and consistency of the design aided ease and efficiency of navigation. In this survey, we assumed the navigation in english language in each of these websites, as the benchmark, given that the users were accustomed to reading news in english. B. E-commerce websites E-commerce websites offer products and services to target audience. Amazon, FedEx and eBay are the three websites in this category, that we considered for our survey. Their portfolio of content has two major characteristics: one that remains static in terms of information content and the other that is drafted specifically for a group of audiences. The information that remains static across all languages of a multilingual portal can be put under the heading of global content. The information that is created specifically to target the individual audience of multilingual website, can be placed under the head of local content. The distinctive feature of global content is that it needs to be translated in each language of the multilingual portal to preserve its information content. Global content comprises of product catalogs, description, etc. which are available globally and remains same in terms of information content and hence the content being displayed globally need not be drafted from scratch. The issues arising out of global content would be purely translation issues. On the other hand, the local content is created keeping in mind the specific needs of the users of that language. Product reviews and other information assisting the customer in their buying decision falls under local content because they are drafted by the local audience to e-commerce portal. Local content incorporates cultural context because it is drafted by the designers belonging to the culture of the native user. The degree of localization is reflected in the amount of local content that a multilingual website contains. The problems of usability caused in the area of local content would be mainly content drafting issues
and not related to translation aspects. Amazon and eBay are different multilingual e-commerce websites as the website in one language has little correlation with the website in the other language because their design is customized for catering to local market, but they are different from online news portals in that they contain both local content and global content. Amazon is an online bookstore which offers a catalog of books and other products to users. Books that are available globally constitute predominant global content in Amazon. While the products that it offers specifically for a market constitutes its local content. FedEx has a close correlation in terms of design across its various languages as its business model offers little variation in terms of localization (Figure 1 and 2). All three websites that we considered for survey under the heading of e-commerce websites offer products and services to international users. Hence while conducting the usability evaluation for these websites the factors shared by the group of international users would be applicable for the target audience of these websites.
Fig. 1.
C. Search Engines Web search engines offering search facility in multiple languages is the simplest case of multilingual websites. These websites have very few pages that the user interacts with and also the design remains same across multiple languages. The content in these websites is predominantly the local content which is included from various websites in the native language of the target user. Hence these portals incorporate cultural context in terms of content and pervading localization. Since their design is highly correlated, testing these websites for usability becomes straightforward. We considered setooz.com in this category for gaining insight into their characteristics in terms of design and content. Setooz.com is an in-house search engine which provides online search facility in 13 international languages. The target audience of this multilingual search engine consists of purely international users. There are very few pages that users interact with to accomplish their task. The page displaying the result set corresponding to user’s query is displayed in a format which remains constant across all languages (Figure 3 and 4). We undertook usability evaluation for this website. Our approach was mainly dependent upon the availability of sample users rather than their relevance. Lacking any structured approach for conducting usability evaluation in multilingual scenario, we proceeded according to the standard steps of conducting usability evaluation which provided insight in shaping our paradigm. We observed that little design changes affected response. It reflected the need for slight customization or localization in layout and design catering to international users.
FedEx - Australia
Fig. 3.
Setooz - Swedish
III. MUST:OUR APPROACH
Fig. 2.
FedEx - China
The observation of these websites reveals few aspects of their design and the content within them. We also observe that websites falling under the same domain exhibit similar features in terms of the extent of localization and the characteristics of their target audience. We could deduce three intuitive aspects based on which these multilingual website can be roughly classified. After considering the domain of the website and the careful observation of their characteristics based on the domain, we could first perceive the website according to its design features. Secondly, the content of the
Such websites are provided by companies who incorporate extensive localization in their business model. The table below depicts the classification in terms of design:
Fig. 4. Single Home Small parts translated into another language(s). The translated sections often appear as links. These websites translate only a limited amount of text into another language.
Setooz - Finnish
Multi-Home The website has a single domain name, with a splash page presenting a choice of languages. Each language’s sub-site has the same layout and design.
Separate Each website has its own domain name (either a country level domain or a sub-domain). Level of variation in design depends upon the extent of localization.
Fig. 5.
Amazon - UK
TABLE I D ESIGN OF M ULTILINGUAL W EBSITES
website can be classified into two broad categories which would be helpful while dealing with problems arising out of content of website. Lastly, we can have two aspects of incorporating target audience. This aspect provides insight in decision making while recruiting sample users and conducting the experiment. A. MUST:Based on Design Now we shape our paradigm according to our observations from the survey and the popular and meticulous work of others in the field of usability testing. To commence our process of development of MUST paradigm, we first look at the design aspect of it. Mathew Hillier [6] in his motivating work describes the design of multilingual websites. In his paper, while exploring the role of cultural context in multilingual website usability, he presents three ways of classifying various multilingual websites, depending upon their design (Table I). They are: single home sites, multi home sites and separate sites, where each belongs to the same organization. The single home sites provide one extreme of design variation in multilingual websites. In such websites only limited portions are translated into another language(s) and these sections appear as links. Separate websites present another extreme of design variation where there exists very little correlation between the designs of two or more languages of multilingual website. As Shen, Woolley and Prior [7] discuss the influence of globalisation on userinterface design and propose a more diverse culture-centred, design-based system called ’Culture-Centred Design’ (CCD).
Fig. 6.
Amazon - China
During our study of FedEx and Amazon websites for usability, we could find that the design and layout of FedEx remained same across languages while there was a significant variation in design of Amazon websites across its languages (Figure 5 and 6). This observation was intuitive when we consider the domain and business model of these websites. This classification of design helps us while articulating the problem definition. While illustrating the usability problem areas of websites, we could consider the domain of website that we are testing for usability and also take into account the classification in terms of design. This would facilitate us in looking at the process of usability testing and devising solutions. As the work of Vanessa Evers and Donald Day [8] point out the role of design in user’s interface acceptance, the consideration of design features right from the commencement of the process becomes essential. The paper by Ito and Nakakoji [9] also reinforces this point. B. MUST:Based on Content The second aspect of MUST is to classify content in terms of the type of content appearing in the multilingual
website. Websites offering products to local market specifically contain some amount of localization in terms of the content drafted in that particular language only. While other languages of website do not contain that content because company does not offer these products in any other market. Hence usage problems arising out of the local content are due to drafting issues by content writers rather than the translation issues. On the other hand, a multilingual website also contains a type of content which remains constant in terms of the information it provides. This content needs to be globally available without incorporating any change in it. We term such content as global content. Problems arising out of the global content are purely translation issues. The division of content in a multilingual website under these two headings also reflect the extent of localization a company marshals. If the amount of localization is more then local content would be more in comparison to global content across languages in a multilingual website. Hence we could classify roughly the content in a multilingual website as (Table II): • Global Content that is customized by translation to different languages for effectively targeting global clientele. • Local Content, which is authored in each local language catering to the domestic market. After defining the problem arising in terms of usability of website, we design the usability test. While planning the usability test and preparing the task list we must recognize the role of content in affecting a task purpose. Peter B. Checkland in his popular book ”Systems Thinking, Systems Practice” [10] clearly states that the level of tolerance a user has to the text of a web page i.e. the context of language will vary, given a task purpose. Turk [11] also supports this fact. Local Content Drafted in each language Incorporates cultural context Content writing issues
Global Content Translated to different languages Static in information content Translation issues
TABLE II C ONTENT T YPE
C. MUST:Based on Target Audience The third aspect of MUST comprises of the user type of multilingual site we are considering for usability testing. Knowing the cultural proximity of target audience aids in their selection and recruitment for conducting usability testing. It also helps in deciding the kind of methodology we wish to adopt for test design. Users can be classified into two categories (Figure 7): ethnic users and international users (Table III). Ethnic users comprise of those target audience who belong to the same country but have different cultural context owing to the separate regions they hail from. They speak regional languages like Tamil, Telugu, and Hindi etc. International users, on the other hand, are those users who reside in different countries and speak different languages like French, German, Chinese, etc.
Multilingual website
Ethnic Users
Tamil
International Users
Hindi
Telugu
German
Japanese
Remote evaluation
Lab Experimentation
Fig. 7.
French
Sample Users and Evaluation Methodology
Ethnic Users Different regions but same country Experimentation
International Users Different countries Remote evaluation techniques
TABLE III C LASSIFICATION OF U SERS
IV. A PPLICATION OF MUST METHODOLOGY TO U SABILITY TESTING With this new perspective for multilingual scenario, we refine the standard structure of usability testing in order to account for classification based on design, content and target audience. Keeping in mind these groupings, we approach each step of usability testing for multilingual scenario in an effective way. We present each step of usability testing as it appears through MUST paradigm (Figure 8) Initially we require the knowledge of website domain before starting the test. The characteristics of the domain that this multilingual website inherits should be illustrated. These features help in classification in terms of design and also the extent of localization aids in ascertaining the extent of two types of content in the website. The nature of target audience of multilingual website; ethnic or international, helps in the choice of evaluation methodology. The overall classification of the websites based on the MUST paradigm, have been mentioned in Table IV. The evaluation parameters and the results of our study has been mentioned specifically (Table IV). 1) Define the objective The problem statement or objective of the test needs to be clearly defined. It reflects the purpose of conducting the test and drives the remaining steps. The definition of problem for multilingual scenario includes the location of problem in terms of design and content. 2) Test Design The choice of evaluation methodology is guided by the type of target audience that the multilingual website has. For international users, conducting the experi-
Domain of Website
News, E−commerce, Search Engine, etc
Problem Definition
Correlated websites, separate websites
Evaluation Plan
Sample Users
Experimental Setup
Ethnic Users or International Users
Lab Experiment or Remote Evaluation
user while utilizing the website. The sessions could be remotely conducted using user-reported critical incident method for remote usability evaluation in the case of sample users involving native speakers for multiple languages. In the case of formal experiment, a trained facilitator observes the user activity and records thought process using Thinking Aloud protocol and classifies the problem area according to MUST paradigm. 6) Data Analysis Usability testing is conducted to ascertain whether the user feel the application efficient and easy to use. Hence the data comprises user’s comments, suggestions and remarks. The data collected from the test is analyzed for problems and summarized based on priority, problem area and its intensity. Focused analysis of results helps in identifying and fixing the problem areas in multilingual web usability.
Data Analysis
V. R ELATED W ORK Interface Revision
Re−Test
Fig. 8.
Structure of Multilingual Usability Testing
ment in a lab is not feasible because of difficulty in collecting sample users from different countries at the same geographical location. In this case the techniques of remote evaluation are more appropriate. When the target audience comprises of ethnic users, the experiment is conducted in the presence of a trained facilitator. 3) Develop the task list After articulating properly the problem area and objective of conducting the test, the task lists are prepared accordingly. 4) Identify the sample users It is important that the sample users are selected with utmost care for the test in order to make the test effective and optimized. The definition of the problem narrows down the problematic area of multilingual website. For instance, a sample user could be a native speaker of the language of multilingual website in the case of separate websites and predominant amount of localization in the problem area targeted by the test. On the other hand, a user aware of translation issues could be incorporated in the scenario involving a multihome website and predominant global content in the problem area dealt by the test. 5) Experiment Setup The test is conducted to monitor the activity of the
Jacques Nantel and Evelyne Glaser have studied the impact of language and culture on perceived website usability. Our objective was to establish a paradigm which recognizes the predominant role of language and cultural context while evaluating usability of multilingual website. Huatong Sun [12] explores various strategies towards website localization. The author evaluated current practices and developed an approach for searching the efficient and effective strategies for localization. The study examines the cultural dimensions of website localizations. In our paper, we approach the problem of testing websites for usability from a process-oriented view and focussed on the steps of evaluation rather than any particular aspect of design or content. However, our attempt is to reduce the abstraction level in the overall task of multilingual usability testing rather than proposing a concrete methodology for a particular aspect of it. Usability evaluation can be categorized under the following three broad headings: Inquiry, Inspection and Testing. The task of filtering out relevant methods for evaluating a particular type of interface could be rigorous. The following methodologies are significant in the usability testing of websites: In User-Reported Critical Incidents [13], users selfreport own critical incidents. Usability data, centered around critical incidents self-reported by users, are captured in dayto-day task situations, while user and evaluator attend the process at different times and different places. Questionnaires [14] are written lists of questions that are distributed to users. It is defined, in a more structural way as a method for the elicitation, recording and collecting information. They are distinct from surveys in providing structured approach over an ad hoc analysis with an advantage of flexible administration. This frugal method recount traits of myriad of target audience while providing insight to their behavior of usage.
Website Yahoo Rediff BBC Amazon FedEx eBay Setooz.com
Domain News News News Bookstore Logistics E Commerce Search
Target Users Ethnic Users Ethnic Users Ethnic Users International Users International Users International Users International Users
Design Separate Separate Corelated Separate Corelated Corelated Identical
Predominant content Local Content Local Content Local Content Global Content Global Content Global Content Local Content
TABLE IV OVERALL C LASSIFICATION BASED O N MUST EVALUATION PARAMETERS
Journaled sessions [15] are often used as a remote inquiry method for web usability evaluation. It supplements usability testing with usability inquiry by providing an opportunity to observe users while they use the web application and at the same time interact with them about their experiences. Users perform several tasks with the web application, much as in formal usability tests and their actions are recorded with the journalizing software. Upon completion of the series of tasks, the results are available for evaluation. Eye tracking [16] equips testers in tracking what participants look at during the course of a usability test. Eye tracking equipment uses various distinct technologies, ranging from skin electrodes, marked contact lenses, image processing cameras, and reflector trackers. We had a usability lab at our institute with the eye tracker available which proved to be cost-effective. Surveys [17], [18] are ad hoc interviews with users, where a list of questions is asked and the users responses are recorded. Surveys differ from questionnaires in that they are interactive interviews, although not structured like contextual inquiries nor formally scheduled and organized like focus groups. Keeping in perspective the advantages and disadvantages of various approaches we could mature our paradigm for effective and optimal usability testing of multilingual websites. VI. C ONCLUSIONS This work is an attempt to extend the techniques and methodologies of usability testing to the realm of multilingual websites and provide optimal solution for usability testing. It explores how we can incorporate solutions while addressing cultural context of different countries or of different ethnic groups within a country. In multilingual scenario, the attempt to conduct formal experimentation with all users present at the same location and at the same time may not be cost effective. Methods used for remote usability evaluation like user-reported critical incident method could be more effective for testing usability in multilingual scenario involving multiple native speakers from different locations. The problems could be identified in a more structured way and solutions could be devised in a more optimal fashion while addressing multilingualism or multiculturalism through MUST paradigm. In future, we intend to explore more methodologies for classification of design and content of multilingual websites and also refine our paradigm to scale to maximum multilingual websites undergoing usability testing.
R EFERENCES [1] K. Steve, Don’t Make Me Think!: A Common Sense Approach to Web Usability. Indianapolis, IN, USA: Que Corp., 2000, foreword ByBlack,, Roger. [2] J. Mehdi, R. Alexander, and van der Linden Frank, Software architecture for product families: principles and practice. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2000. [3] C. Flavin, M. Guinalu, and R. Gurrea, “The influence of familiarity and usability on loyalty to online journalistic services: The role of user experience,” Journal of Retailing and Consumer Services, vol. 13, no. 5, pp. 363–375, September 2006. [4] R. Ferreira, Culture and e-commerce: culture based preferences for interface information design. Virginia Polytechnic Institute and State University, 2002. [5] N. Jacques and G. Evelyne, “The impact of language and culture on perceived website usability,” J. Eng. Technol. Manag., vol. 25, no. 1-2, pp. 112–122, 2008. [6] M. Hillier, “Multilingual website usability: Cultural context,” in Proceedings of the International Conference on Electronic Commerce, 2002. [7] S. Siu-Tsen, W. Martin, and P. Stephen, “Towards culture-centred design,” Interact. Comput., vol. 18, no. 4, pp. 820–852, 2006. [8] V. Evers and D. Day, “The role of culture in interface acceptance,” in IFIP Conference Proceedings. Proceedings of the IFIP TC13 International Conference on Human-Computer Interaction, 1997, pp. 260 – 267. [9] I. Masao and N. Kumiyo, “Impact of culture on user interface design,” pp. 105–126, 1996. [10] P. B. Checkland, Systems Thinking, Systems Practice. Chichester, UK: John Wiley, 1981. [11] T. A., “A world wide web of cultures or a ’world wide web culture’, in cultural attitudes towards technology and communication.” in Proceedings of the second International Conference. Murdoch University, Australia, 2000, pp. 243–256. [12] S. Huatong, “Building a culturally-competent corporate web site: an exploratory study of cultural markers in multilingual web design,” in SIGDOC ’01: Proceedings of the 19th annual international conference on Computer documentation. New York, NY, USA: ACM, 2001, pp. 95–102. [13] J. C. Castillo, Hartson, H. Rex, Hix, and Deborah, “Remote usability evaluation: can users report their own critical incidents?” in CHI ’98: CHI 98 conference summary on Human factors in computing systems. New York, NY, USA: ACM, 1998, pp. 253–254. [14] W. Foddy, Constructing Questions for Interviews and Questionnaires: Theory and Practice in Social Research. Reading, Massachusetts: Press Syndicate of the University of Cambridge, 1993. [15] J. Nielsen, Usability Engineering (Interactive Technologies). Elsevier Science & Technology Books, 1993. [16] D. JS and R. Janice, A Practical Guide To Usability Testing (Human Computer Interaction. Greenwood Publishing Group Inc., 1993. [17] S. Priscilla and D. D. A, How to Conduct your own Survey. New York: John Wiley and Sons, 1994. [18] A. P. L. and S. R. B., The Survey Research Handbook. Chicago: Irwin Professional Publishing, 1995.
