Measuring Public Value UX-Based on ISO/IEC 25010 Quality Attributes: Case Study on e-Government Website Ashok Sivaji1, Norfarhana Abdollah2, Soo Shi Tzuaan3, Chuan Ngip Khean4 Zulkifle Mohd Nor5 Siti Hamimah Rasidi6 and Yoong Siew Wai7 MIMOS UX Lab1,2,3,4,5 Commercialization and Market Research 6,7 MIMOS Berhad Kuala Lumpur, Malaysia
[email protected] Abstract—e-Government can be simplified as the use of electronic technology by government to provide information and services to the citizens. In theory, it shall be one of the effective ways for government to reach and interact with the citizens. In line with these, the usability and user experience of e-government website should meet or exceed citizen’s need. This first objective of this study is to measure user experience of citizen using a jobs-seeking website managed by the Malaysian government. The next objective is to determine suitable UX methodologies to measure the same. The ISO/IEC 25010 quality characteristics have been applied as the guidelines to measure the usability and user experience of the local jobs portal. This paper is written to address the user experiences issues of the registration process which is one of the common tasks to be completed as an attempt to obtain account in a job-seeking website service provided by the government. User experience test conducted with undergraduate users have shown various usability issues. Thus, recommendations obtain and analysis of the results shall be useful to guide the improvement of current version of jobs portal to better serve the good cause of its existence. Keywords - e-government; usability; informal user experience test; ISO/IEC 25010
I.
INTRODUCTION
In Malaysia, Electronic Labor Exchange (ELX) is a one stop service provided by the Malaysian Government under the purview of Ministry of Human Resources. The services including Job Clearing System (JCS) which is now known as JobsMalaysia provide many valuable services to the Malaysian citizens such as registration of employee and employers; jobs matching and jobs searching [1]. A study conducted on ELX in 2011 also found that there is an urgent need to improve eGovernment services in terms of functionality, usability and security which in turn could improve user satisfaction and increase usage as well as loyalty [1]. There are many targeted users of the ELX. One of the user segments is the undergraduate students who will eventually form the skilled worked force of Malaysia. Another study shows that online recruitment is expected to continue to gain prominence [2]. In order for the Ministry of Human Resource
Malaysia to provide this service to the undergraduates, considerable focus is required to influence the perception of these target users. Study by [2] has shown that user friendliness, information provisioning and website usability are significantly related to the perception of the undergraduates. The scope of this study however is on the undergraduate students who are also the first time users of ELX. II.
RELATED WORKS
A. e-Government Services Authors in [1] has recommended the involvement of citizen in monitoring and evaluating Government services to better help the Government to identify problems and implement innovative solution based on citizen experience. A study performed to understand the user satisfaction on the adoption of e-Government services in Malaysia found that the elements such as utility, reliability, efficiency, customization and flexibility are significant to influence the satisfaction level of the citizens [3]. A recent study by [4] on 16 Turkish Government websites found that although the sites are reasonably good at supporting requests and usability, the sites are far from satisfactory levels at providing platforms for citizen engagement, responsiveness and dialog. It is recommended that the public value perspective to be incorporated in the evaluations of websites and other public efforts [4]. As shown in Fig. 1, user experience and content should drive website design.
Figure 1. Relating website, users and public values [4]
This approach is also known as user-centered analysis (UCA) and user-centered design (UCD) among the HCI community. B. Measuring Malaysian Public Value with Informal UX Test The first objective of this study is to measure public value through the engagement of users in the evaluation of eGovernment service. This could be achieved through informal usability testing with the intended users of the e-Government services. As the name suggests, lab based usability testing is carried out in a controlled lab with users. Conversely informal usability testing is carried out in a more relaxed environment to simulate how users would use a system in natural environment. In order to ensure that public value is measured effectively through engagement, informal usability testing is believed to be more beneficial compared to previous studies which were carried out formally in the laboratory [5-10]. There are few reasons for this approach. Firstly, as witnessed in previous studies, many respondents suffer from high power distance during usability study [6-8]. As a result, they were not comfortable and tend to get nervous to think aloud when the studies were conducted in laboratory, resulting in some portions of the data gathered redundant. For instance, respondents answered “ok” when feedback was requested which is meaningless to the usability analyst. These data had to be disregarded as the researchers could not interpret these results. Informal usability testing, when conducted in a natural environment will increase the engagement between the practitioners and respondents. In other studies, the effect of high power distance is reduced by depending on advanced and expensive eye tracking equipment for the analysis of the usability study [8], [10]. Thus, it could be seen that informal user experience testing is a more practical and affordable method to ease data collection. C. Internet Usage Trends among Malaysian In order to gauge the Malaysian mental model, it is important to understand trends among Malaysian users and some recent initiatives by the Malaysian Government. A recent report from Malaysian Communications and Multimedia Commission (MCMC) shows that the 3G and Cellular Mobile subscription has grown by 42 times since 2006 [11] . This shows that the demand among Malaysians continue to growth. As a response to this demand, in the 2014 Budget speech tabled by the Prime Minister and Minister of Finance Datuk Seri Najib Tun Razak on 25 October 2013, the Government has implemented the High-Speed Broadband (HSBB) project under the National Broadband initiative [11]. The Internet User Survey 2012 from MCMC shows that the top 3 locations respondents of internet access are from home (63.1%), workplace (34.8%) and on-the-go (24.3%). The same survey found that Malaysian use multiple devices to access the internet. From here, it is clear that Malaysian would expect optimal user experience regardless of which device or location the internet being accessed. This increases a new challenge to the design of fixed-width websites. As reported by [12], fixedwidth websites ignores users with more capable hardware giving them a limited user experience. In other words, the tradition design strategy of ‘one size fits all’, whereby the typical minimum target resolution of 1024x768 to fit all users is not sustainable for a fast growing country like Malaysia.
D. ISO/IEC 25010 Usability Characteristics In this study, similarities between “citizen experience” [1] and the term “user experience” (UX) which is widely used in the literature of Human and Computer Interaction have been draw. The aim of our study is to measure public value through the engagement of users (undergraduate) in the evaluation of eGovernment service; focusing on online recruitment. In order to achieve this, we seek to perform the testing in an accredited Software Testing facility in Malaysia. As of January 2014, Standards Malaysia has recognized MIMOS as the first lab in Malaysia to receive MS ISO/IEC 17025: 2005 certification in software testing which includes usability and user experience [13]. The usability characteristics are measured against ISO/IEC 25010:2011 (E) [14]. This standard is a combination and revised of previous standards such as ISO/IEC 9126: 2001 and ISO 9241-11:1997 which were widely used by Software Engineering and Human Computer Interaction community respectively. The Quality in Use model defines five characteristics namely effectiveness, efficiency, satisfaction, freedom from risk and context coverage; while the usability sub characteristics of the Product Quality model are appropriateness recognizability, learnability, operability, user error protection, user interface aesthetics and accessibility. III.
METHODOLOGY
A. Optimal UX Methodology Selection The second objective of this study is to select a suitable methodology to measure the e-Government website. The variables such as usability defect count, usability defect severity, resource requirement, cost, workload, quality metrics and coverage factor has been studied extensively in [5] across the software development lifecycle (SDLC). It has been revealed that Lab based Usability Testing (UT) is expensive, able to measure quality characteristics yet has a lesser test coverage than heuristic evaluation (HE); while HE seems economically more viable, it is subjected to false alarms as the evaluator attempts to think on behalf of the users and they are not the actual users of the system. For instance in this study, the evaluator may not have similar motivation to seek for a job as compared to an undergraduate who is the final semester of study. In another study by User Experience Professionals Association (UXPA) [15] found that the top 5 UX methods applied by practitioners globally comprised of 1- HE (74%), 2Informal (68%), 3 - Lab (54%), 4 - Remote and Moderated (42%) and 5 - Satisfaction Surveys (34%). Based on the Hybrid Usability Methodology [5] and the aim of this study that is to measure public value through engagement of users while reducing the effects of power distance, informal UT seems to be the most optimal method to measure usability and UX of a jobs portal based on the quality characteristics of ISO/IEC 25010:2011(E) [14]. This decision is achieved by using the hybrid usability methodology selection decision tree [5] and UX Calculator from Mi-UXLab [16], [17]. As shown in Fig. 2, when the following selection are made: SDLC stage: Advanced, ISO: 1 or 100% indicating the highest weightage being placed on measuring quality attributes, the calculator returns in green highlight to perform 0% HE and 100% UT.
TABLE I. QUALITY IN USE AND PRODUCT QUALITY CHARACTERISTIC Model
Characteristic, Definition & Measurement Characteristics
Figure 2. UX calculator => 100% usability test and 0% heuristic evaluation based on high weightage on quality attributes [5], [16], [17]
B. UX Quality Characteristics for Task Completion Measure Table I summarizes the characteristics, definitions and measurements related to the quality in use and product quality characteristics of the jobs portal. One of the important measures for usability testing is task completion rate. In Table I, effectiveness, efficiency and learnability are measured with respect to the success of task completion, amount of time taken and the level of assistance required to complete performing a task. In terms of the jobs portal, some of the common task that would be performed by first time users includes registration into the system, finding new vacancies, updating personal information, searching and applying for a desired job. The scope of this paper is only on the registration task which is important for job matching. C. User Recruitment Targeted towards Malaysian Demography The sample size for this study is calculated based on reference [21] and [22]. Sample size of 16 is recommended to compare one design of website and baseline the results for future release design [21]. This study was conducted with 23 final semester undergraduates who are first time users of the jobs portal and moderated by usability practitioners from MIMOS UX Lab in an informal manner. The undergraduates are from various fields of studies such as engineering, psychology, sports science and mass communication. From the survey demography and Fig. 3, it is found that the samples spend an average of at least four hours per day using the internet, with significant amount of time in social networking and educational information gathering. From Fig. 3 as well, participation of samples in e-Government services stands at 30% in April 2014 which is an improvement of about 10% over the study by Malaysian Communications and Multimedia Commission in 2013 [23]. Despite the improvements, the outcomes of this study (refer to Results section) would reveal some of the reason for the low participation which will enable e-Government stakeholders to take relevant action. All of the users used either laptop or desktop to access the internet and 93% of them also went online using smart phone. This shows that the subjects recruited for this study is consistent to the multiple devices usage behavior demonstrated by Malaysians as reported in [24]. Most of them preferred informal and friendly communication style; this shows that the decision to use informal usability testing is suitable. Users also displayed high tendency towards systems that are friendly,
Product Quality
Quality in Use
Effectiveness
a
Efficiency a Satisfaction Freedom from risk Context completeness Operability Learnability a User error protection
Definition & Measurements
Extent to which user succeeded to use & complete the services offered Amount of time spent in completing the services offered Degree to which systema satisfies user needs in terms of Usefulness, Ease of Use [9], [18] Degree to which systema mitigates risk to health and safety risk and efficient operation The extent to which systema is usable by first time user (non-expert) Degree to which a product or system has attributes that make it easy to operate and control [14] The level of assistance required to use and complete the services offered Extent to which systema minimizes the possibility of user error, with inbuilt facilities for detecting and handling [19], [20] a. Measure based on time completion rate
Figure 3. User’s main online activity
provided with good user guidance and support rather than in a formal and structured manner. They also believed in having a good balance between work, friends and family. Most of them disagreed to the question of “My career is usually more important than spending time with my family”. D. Moderators and UX Tools Selection The moderators used in this informal usability testing have between 5-10 years’ experience as a usability / UX moderator. They have attended usability training and their practices of usability are standardized in a usability work procedure that is. MS ISO/IEC 17025: 2005 accredited [13]. Similar to Morae by TechSmith, to facilitate the qualitative and quantitative data collection, the Mi-UXLab user experience testing tool, formerly known as URANUS system [16], [17], [25] was used to measure demographic survey and also necessary ISO25010 characteristics (in Table I) while conducting the informal usability testing. Mi-UXLab has been acknowledged and/or used by HCI educators, UX practitioner and industrial ergonomist since 2011 for data gathering and analysis for various UX research in the field of electronic and mobile Commerce (e-Commerce/m-Commerce), e-Government, industrial ergonomics and cultural UX [5-7], [10], [16], [17], [25-27]. In this study, Mi-UXLab facilitated informal testing in multiple places such as in a lecture hall, library and living room settings to simulate how undergraduates would indeed carryout the job seeking process in real situations.
IV.
RESULTS AND DISCUSSION
A. Effectiveness Effectiveness is measured by observing the level of success of task completion. Out of the 23 users, only 5 users were able to complete the registration. From this, a mix of positive and negative feedback were received. Since it was the first time these users were exposed to an online job portal, they were indeed excited and felt grateful that the government has taken a good initiative to help undergraduates to apply for a job. They did find the portal as a useful means in applying for either a government or private sector job. On the other hand, for the 18 users who failed to complete the task, numerous feedbacks were also gathered regarding the registration process. In terms of effectiveness, users found that too many mandatory information required (fields indicated by *) as part of the registration made it difficult or even impossible to complete the registration at once. The users did suggest minimizing mandatory information significantly. For instance, for the demography portion, users felt that the system should only request for the Malaysia National Registration Identity Card number (abbreviated NRIC No.) as it shall contained all other relevant information pertaining to the individual respectively. B. Efficiency Many users also experienced a long time for task completion (low efficiency) which distracted their attention and made them feel uneasy during the registration. The users found that the system requested too many information about their personal details, educational background, interests, and so on. This triggered the users to abandon the registration part. Some of their feedbacks include: • • • •
“it took such a long time to fill up the job vacancy, ”, “too much of questions to be answered especially in the section of education background and interest” “to register a new account is convenient, however the time taken can be improved without which this could turn away potential job seeker”, “the slowness of the system somehow distracted me and i feel uneasy and not confident in completing the forms. ”
C. Learnability Out of the 23 users, 5 or 22% of users who were successful in registration found that once they have completed the registration task, they may understand the system and could then use it. However for the remaining 78% of the users, they found that the system has a high learning curve. They recommended that the system provided more informative feedback and direct user guidance. They also found the system confusing as for novice users, they were expecting for the system to list eligible jobs with minimum salary at each level of education. D. Operability Since it is normal for user to use more than one device nowadays to access the internet, the users as well tried to access the jobs portal using their smart phones and realized that the system layout is inappropriate and difficult to operate on such smaller screen devices. During the testing, it is observed that some of the information presented would get distorted when the jobs portal was viewed from their devices (different
screen sizes). This would limit the user experience to the device being used. As reported by [12], users will always expect optimal experience regardless the devices they used. From the testing, it is observed that the user experience being limited by irresponsive design of the jobs portal layout. Jobs portal layout is clear when accessed using 24’’ desktop and 12” netbook. However, there were users who use Samsung Galaxy Tab 3(7”) and Apple iPhone 5 (4”) who encountered a limited user experience as they were not able to view similar information due to differences of the screen size. This shows that the system partially supports multiple device access which is among the trend reported among Malaysians by Malaysian Communications and Multimedia Commission [23], [24]. Smartphone users encountered were not even given an opportunity to register as a new job seeker as they could only access the login page, which was meant for users who have already registered. The website however provides an instruction that user needs to use desktop computer to register. This negatively impact the user experience as the jobs portal shall be accessible and operable regardless which type of device being used, as long as there is available internet connection. E. User Error Protection Another major issue faced by the users is that once they have forgotten their password, the jobs portal does not provide a convenient way to retrieve it. During the evaluation, one user who managed to register but later tried to retrieve the password, couldn’t do so, despite the website providing two options as shown in Fig. 4. 1) Reset using security question Using this option, it is still inconvenient as user needs to enter security question but unfortunately that user also did not remember it. Then the other option is to manually call the operator for assistance, as shown in Fig. 5. 2) Reset by SMS For this method, user will be charged 15cents for sending a SMS and 20cents for retrieving incoming SMS from the system, as illustrated in Fig. 6. After few attempts, although the user was charged, the website failed to retrieve the password. It was surprising to the user that the reason given was that the number is not even registered in the first place. Eventually, the user gave up and decided to consider other job portals in the future. This shows failure to provide user error protection and charging users for poor service results in user turning away to competitor websites. F. Satisfaction and User Error Protection In this study, word clouds have been used as a technique to illustrate user’s overall feedback regarding their experience of account registration using the jobs portal. Word clouds are useful to summarize a representation of list of texts or words [18] . Study by [9] has found that word clouds provides a quick and simple visual representation of the level of satisfaction. In other words, this technique is capable of eliciting users’ subjective feedback regarding their experience while using the website [9].
Option 1 Option 2
Figure 4. Options for password retrieval
context of using the jobs portal by a first time user in performing the registration task using a desktop screen with a high network bandwidth. Within this context, qualitative feedbacks have been collected for improvement of the system in terms of effectiveness, efficiency, learnability, user error protection and freedome from risk. In the future study, we recommend similar study to be carried out in other realistic scenarios such as using smaller screen devices, such as smartphone, tablet, or ‘phablet’ from a lower bandwidth to simulate access from home, public areas etc. I. Proposed Model for Measuring Public Value UX Based on the results and findings, the UX Testing Framework (as shown in Fig. 8) is proposed for measuring public value UX in line with research objective 2. It could be seen as an extension to Fig. 1, whereby the usability and UX findings could be feedback to the design and development team. V.
Figure 5. Option 1-Reset password with security question
CONCLUSION
This paper has reported the qualitative results based on ISO/IEC 25010 usability characteristics of a jobs portal. The scope reported in this paper is on the account registration. The users found that the system requested too many detailed information from them and the registration steps were difficult, complicated and exhaustive to be completed at one time.
Figure 6. Option 2-Reset password with SMS
Based on the results obtained in Fig. 7, two outstanding words are ‘Confusing’ and ‘Useful’. This could be interpreted as the jobs portal is indeed useful; however in terms of user satisfaction it has raised a lot of confusion. Users also found that although in general the website is accessible, it is somehow difficult to be used and time consuming.
Measuring Public UX [4]
How
UX Methodology [5-10]
Which UX Methodology
Malaysian Culture [8] [12] [11] [23] [24] [28]
HUM [5-7] [10] [16] [17] [25] [26]
UXPA [15]
Standard [9] [13] [14] [18] [19]
Informal User Experience Test
Figure 8. UX testing framework for e-Government
Context Completeness
Freedom from Risk
User Error Protection
Operability
UX Metrics
Learnability
Mi-UXLab [16] [17]
Which Tool
Satisfaction
H. Context Completeness Context completeness is the degree to which a system can be used with effectiveness, efficiency, freedom from risk and satisfaction [14]. From this study, we have specified the
Objective
Efficiency
G. Freedom from Risk Freedom from risk is defined as the degree to which a system mitigates the potential risk to economic status, human life, health or the environment [14]. Although the users found that the website is useful, they also found that the system elicit higher cognitive load to complete the registration task. Hence, attempts to improve the system to reduce cognitive load is an effort to mitigate health and safety risks. In terms of economic risk mitigation, some of the potential risks of the system include inefficient operation. Hence, improving system efficiency is utmost important.
Figure 7. Word Cloud generated based on user’s feedback
Effectiveness
Users commented that they feel dissatisfied while filling up their details as many of them feel disengaged. There was clear time delay or lagging when the website was attempting to display error message. Although the website has field validation error handling, the time lag for the error messages to be displayed caused confusion leading to users feeling lost and less confident with the system.
Thus, the feedbacks obtain from the informal usability testing using the ISO/IEC 25010 usability characteristics are useful to help improve the current jobs portal to better serve the undergraduates. As shown in Fig. 1, these feedbacks indeed represent public values which are crucial to the improvement of e-government websites. Similar studies using the ISO/IEC 25010 usability characteristics could be conducted to generate public values to improve other egovernment websites as shown in Fig. 8. One of the limitations of this study is that the user segment is only constrained to the undergraduates. Similar studies could be conducted for other user segments such as existing work force and also unskilled workforce. Besides, other testing shall be conducted in the future once the improvement have been made for current jobs portal with anticipation that the user experience and the results will be significantly improved. This study is also limited to only the registration task within the job portal and a subset of usability-quality metrics. Other metrics such as accessibility, aesthetics and appropriateness recognizability could be considered in the extension of this study. REFERENCES [1]
Z. Mohd.Noor, H. Kasimin, A. Aman and N. Sahari, "An adoption model of electronic Government services in Malaysia: Electronic labor exchange (ELX)," Jurnal Pengurusan, vol. 33, pp. 87-97, 2011. [2] W. Ming-Yen Teoh, S.-C. Tan and S. Choy Chong, "Factors Influencing perceptions of university students towards internet recruitment," Asian Academy of Management Journal, vol. 18, no. 1, pp. 123–142, 2013. [3] E. S. Alias, S. H. Mohd Idris, N. Sahari @ Ashaari and H. Kasimin, "Evaluating e-Government services in Malaysia using the EGOVSAT model," in International Conference on Electrical Engineering and Informatics, Bandung, Indonesia, 2011. [4] N. Karkina and M. Janssen, "Evaluating websites from a public value perspective: A review of Turkish local government websites," International Journal of Information Management, vol. 34, no. 3, pp. 351–363, June 2014. [5] A. Sivaji, M. R. Abdullah, A. G. Downe and W. F. Wan Ahmad, "Hybrid usability methodology: Integrating heuristic evaluation with laboratory testing across the software development lifecycle," in ITNG '13 Proceedings of the 2013 10th International Conference on Information Technology: New Generations, Washington, DC, USA, 2013. [6] S. Rashid, S. Shi–Tzuaan, A. Sivaji, H. S. Naenib and S. Bahrib, "Preliminary usability testing with eye tracking and FCAT analysis on occupational safety and health websites," Procedia - Social and Behavioral Sciences, vol. 97, pp. 737–744, November 2013. [7] A. Sivaji, S. Shopian, Z. Mohd Nor, N.-K. Chuan and S. Bahri, "Lighting does matter: Preliminary assessment on office workers," Procedia - Social and Behavioral Sciences, vol. 97, pp. 638–647, November 2013. [8] A. Sivaji and W. F. Wan Ahmad, "Benefits of complementing eyetracking analysis with think-aloud protocol in a multilingual country with high power distance," in Current Trends in Eye Tracking Research, Springer International Publishing, 2014, pp. 267-278. [9] N. Abdollah, S. Shi Tzuaan and A. Sivaji, "Eyes on OSH – Usability testing with eye-tracking and user preference rating," in Advances in Visual Informatics, vol. 8237, Springer International Publishing, 2013, pp. 323-334. [10] G. Kim Nee, Y. Y. Chen, F. W. Lai, S. C. Daud, A. Sivaji and S. Soo, "A comparison of usability testing methods for an e-commerce website:
[11] [12]
[13]
[14]
[15] [16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
A case study on a Malaysia online gift shop," in 10th International Conference on Information Technology: New Generations, Washington, DC, USA, 2013. Malaysian Communications and Multimedia Commission, MCMC Clarifies Report on Broadband Speeds in Malaysia, 2014. B. Gardner, "Responsive web design: Enriching the user experience," Sigma Journal: Inside the Digital Ecosystem, vol. 11, no. 1, pp. 13–20, 2011. Archived News (04/02/2014) : Standards Malaysia Recognises MIMOS As The First Malaysian Lab To Receive MS ISO/IEC Accreditation Certificate For Software Testing BERNAMA - Official Portal of Nationall New Agency of Malaysia International Standard, Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality Models, International Standard ISO/IEC 25010, ISO/IEC 25010:2011 (E), Switzerland: ISO/IEC 2011, 2011. P. Sherman, "Survey, UPA 2009 Salary," Usability Professionals' Association, Bloomingdale, 2009. A. Sivaji and S. Shi–Tzuaan, "Understanding, enhancing and automating HCI work practices: Malaysian case studies," Procedia - Social and Behavioral Sciences, vol. 97, pp. 656–665, 6 November 2013. A. Sivaji and S. Shi-Tzuaan, "Website user experience (UX) testing tool development using Open Source Software (OSS)," in Southeast Asian Network of Ergonomics Societies Conference, Langkawi, Kedah, 2012. R. Kaptein, D. Hiemstra and J. Kamps, "How different are language models and word clouds?," in 32nd European conference on Advances in Information Retrieval, Berlin, Heidelberg, 2010. D. Albayrak and K. Çağıltay, "Analyzing Turkish e-government websites by eye tracking," in Joint Conference of the 23nd International Workshop on Software Measurement (IWSM) and the Eighth International Conference on Software Process and Product Measurement (Mensura), Ankara, 2013. A. Sivaji, A. Azween, A and A. G. Downe, "Usability testing methodology: Effectiveness of heuristic evaluation in e-government website development," in Modelling Symposium (AMS), 2011 Fifth Asia, Kuala Lumpur. Blink, "Usability Sample Size Calculator," Blink Interactive, Inc, 2011. [Online]. Available: http://www.blinkux.com/usability-sample-size/. [Accessed January 2014]. W. Hwang and G. Salvendy, Number of people required for usability evaluation: the 10±2 rule, Communications of the ACM 53.5 (2010): 130-133 "Communications and Multimedia : Pocket Book of Statistics Q4 2013," Malaysian Communications and Multimedia Commission, Cyberjaya, Selangor, 2014 "Statistical Brief Number Fifteen, Internet Users Survey 2012," Malaysian Communications and Multimedia Commission, Cyberjaya, Selangor, 2014. L. Jo Elliott, "Teaching undergraduates about human factors through research in usability," in Human Factors and Ergonomics Society Annual Meeting, 2013. S. Bahri, S. N. Shopian and A. Sivaji, "Alertness, visual comfort, subjective preference and task performance assessment under three different light's colour temperature among office workers," Advanced Engineering Forum, vol. 10, pp. 77-82, December 2013. S. Shi Tzuaan, A. Sivaji, L. Tek Yong, M. H. Tahriri Zanegenh and L. Shan, "Measuring Malaysian m-commerce user behaviour," in International Conference on Computer and Information Sciences (ICCOINS), Kuala Lumpur, Malaysia, 2014. T. Clemmensen, M. Hertzum, K. Hornbæk, Q. Shi and P. Yammiyavar, "Cultural cognition in the thinking-aloud method for usability evaluation," in International Conference on Information Systems, Paris, 2008.
