Conceptual Framework and Models for Identifying and Organizing Usability Impact Factors of Mobile Phones Dong-Han Ham1, Jeongyun Heo2, Peter Fossick3, William Wong1, Sanghyun Park2, Chiwon Song2, Mike Bradley3 1 2 3 School of Computing Science MC R&D Centre Product Design Engineering Middlesex University LG Electronics Middlesex University The Burroughs London Seoul Korea Bramley Road London NW4 4BT UK {jy_heo, sanghyun, chiwon79} N14 4YZ UK {d.ham, w.wong}@mdx.ac.uk @lge.com {p.fossick, m.d.bradley} @mdx.ac.uk Usability can be considered both from a design and evaluation perspective (Folmer et al., 2003). Usability is one of a range of non-functional requirements, such as safety and security, which should be satisfied as part of the design process. Therefore, it should be properly specified during requirements analysis and designed during the architectural and implementation design phases. Conversely, usability is the concept that needs to be evaluated from a user-centric point of view. User perception of usability is influenced by many design factors including visual appeal, hedonic qualities, logical task sequences, and pleasure in use, as well as contextual factors including the users’ environment (i.e. context of use). Thus, it can be said that usability is not an absolute concept determined by the design activity only, but rather the relative concept that can be affected by unspecified factors.
ABSTRACT
Usability has been regarded as a critical factor affecting the quality of mobile phones. Many studies have examined usability impact factors of mobile phones on the basis of software usability concepts. However, considering mobile phones as multi-media and information appliances, a new usability concept and associated factors should be developed. This paper proposes a conceptual framework which has five views to reflect different aspect of interactions between users and mobile phones, and from which various usability impact factor models can be derived. Five views include user view, product view, interaction view, dynamic view, and execution view. Furthermore, we developed a hierarchical model which organizes usability factors in terms of goalmeans relations. Through two case studies, we could verify the usefulness of the framework and model. Lastly, we developed a set of checklists that are helpful to measure the usability of mobile phones, thereby increasing the practicality of the framework and model.
To evaluate usability in a more systematic way, many studies examined factors or dimensions constituting usability (Bevan, 1999). For example, ISO/IEC 9241 (1998) defines three dimensions: effectiveness, efficiency, and satisfaction. Another example is those described in Nielsen (1993): learnability, efficiency of use, memorability, errors, and satisfaction. These dimensions can be classified into two main groups: objective and subjective dimensions. An objective dimension generally measures how well users’ tasks are supported by applying task performance measures like task completion time and the number of errors. However, objective dimensions do not always predict the user’s assessment of usability because it does not reflect users’ feeling or satisfaction. Subjective dimensions therefore needs to be assessed to provide a holistic and complete usability measurement.
Author Keywords
Usability, Mobile user interfaces, Interface evaluation ACM Classification Keywords
H1.2. Models and principles: User/machine systems. INTRODUCTION
It has been reported that usability is one of the most important attributes affecting the quality of mobile phones and thus users’ satisfaction (Ketola and Röykkee, 2001). Usability has been defined in various ways, but the concept of usability defined in ISO/IEC 9126 (1998) is widely accepted (Hornbaek, 2006). According to the definition, usability refers to ‘the capability of the (software) product to be understood, learned, used and be attractive to the user, when used under specified conditions.’ Although it is the definition focusing on software systems, it can be applied to mobile phones in consideration of features specific to mobile phones.
Usability can be measured in various ways; however, they can be categorized in three methods: usability testing, usability inquiry, and usability inspection (Zhang, 2003). It cannot be said that one method is the best in all situations. Hence it is necessary to choose an appropriate method, taking into consideration evaluation purposes, available time, measures to be collected and so on.
OZCHI 2006, November 20-24, 2006, Sydney, Australia. Copyright the author(s) and CHISIG Additional copies are available at the ACM Digital Library (http://portal.acm.org/dl.cfm) or ordered from the CHISIG secretary (
[email protected]) OZCHI 2006 Proceedings ISBN: 1-59593-545-2
We have described the general concepts concerning usability so far, without considering the features peculiar to mobile phones. To examine the usability of mobile phones, it would be useful to understand the user interface of mobile phones, tasks to be completed, and the 261
context of use of mobile phones, therefore it is useful to briefly describe them in the following section.
2004). The three types are: logical user interface (LUI); graphical user interface (GUI); and physical user interface (PUI). In this paper, LUI (e.g. menu structure and navigation structure) is defined as interface related to information contents and layout for task execution. GUI (e.g. icon and font) is concerned with graphical or visual items presenting information which users need to perform tasks. PUI (e.g. keypad and microphone) means tangible elements supporting physical operation needed for carrying out tasks.
Mobile Phones and Tasks
Mobile phones are a typical example of mobile or handheld devices. These devices are portable, selfcontained information and communication systems, and are characterized by three features: (1) they are used primarily in a user’s hands, (2) they are operated without cables, and (3) they support the addition of new application and internet connection (Weiss, 2002).
A key feature and constraint of user interface is that mobile phones have too little screen to display a lot of information at the same time; therefore, information organization and navigation are critical usability issues. The second significant feature is that a physical button or key has generally more than one control function. Thus the functions of a single key are dependent on types of modes. The third is that processing power and available memory are limited.
User Interface of Mobile Phones
Figure 1 shows a typical user interface of a mobile phone (LG, U2050 User Guide). Ketola and Röykkee (2001) divided user interface elements into seven categories: input (e.g. softkey, alphanumeric keys, and navigation tools), display (e.g. icons and indicators), audio and voices (e.g. ringing tones and microphone), ergonomics (e.g. touch and feeling, and slide), detachable parts (e.g. SIM card and battery), communication method (e.g. Bluetooth), and applications (e.g. making a call, games). User interface is one of the interaction elements affecting usability. Other interaction elements include external interface and service interface. External interface contains user support, accessories, and supporting software. Service interface refers to service provider’s services.
Interaction with Mobile Phones
As mobile phones have increasing number of functions, users can do various tasks using mobile phones. The most frequently used functions include making a call, sending a message, personal information management, listening MP3 files, changing settings, and taking pictures, playing games. Other functions are watching TV, remote control of systems on home automation, internet banking, personal computer functions, etc. In the forthcoming ubiquitous computing environment together with the continuing patern of convergence mobile phones are expected to play a key role accessing a wide range of services and will therefore be an essential device within contemporary life. The usability of mobile phones needs to be evaluated taking account of their context of use. Certainly, user interface elements are important factors affecting the usability of mobile phones. However, usability is also influenced by other factors from user groups and their preferences, task types, social constraints and so on. Citing the terminology from the area of software quality, we need to focus on quality-in-use rather than external or internal quality (Bevan, 1999). For this reason, usability impact factors should be examined with systematic viewpoints addressing all the aspects concerned with the interaction between users and mobile phones. Research Problems
To date, several studies have examined various kinds of factors affecting the usability of mobile phones. Examples are: efficiency; object layout; accessibility; consistency; screen size; navigation; battery life; feedback messages; user guidance; naming/labelling; etc. Such studies helped us understand what could constitute usability impact factors; however, very few attempts have been made at developing a conceptual framework and impact factor model for organizing the factors, which usability engineers actually need. Additionally, most of studies concerned with usability of mobile phones have applied software usability concepts originating from the study of static personal computers (PCs). Although these
Figure 1. User interface of mobile phones.
The seven categories described above are useful to understand the characteristics of mobile phone user interface. However, the categories can be generalized into the three user interface types that are more helpful to study users’ interactions with mobile phones (Kiljander,
262
approaches have been helpful to identify factors affecting usability of mobile phones, it seems to fail to reflect mobile phone characteristics comprehensively.
Lastly, execution view offers methods or techniques to evaluate usability impact factors. One assumption of this framework is that any combination of five different views can result in a meaningful model of usability impact factors.
This paper firstly proposes a conceptual framework from which several models of usability impact factors can be derived. And then a hierarchical model of impact factors, which was developed on the basis of the framework, is described. Lastly, we explain two case studies that were conducted to verify and improve the proposed framework.
Axioms
Usability is emergent property
Our position is that usability cannot be absolutely evaluated. Instead several evaluation results based on several viewpoints can indicate it
Perceived usability has abstraction hierarchy characterized by goal-means relations
User view reflects this axiom
Usability is affected by all the things of mobile phones
Product view reflects this axiom
Functionality restricts usability and at the same time is one critical factor of usability
Interaction (task) view reflects this axiom
Usability changes along time
Dynamic view reflects this axiom
Usability is dependent on user groups
User and Dynamic view reflects this axiom
Usability impact factors have different meaning to users in terms of their preference or importance
Execution view reflects this axiom (quantification scheme)
Designed usability is the outcome of constrained optimization
Execution view reflects this axiom (weighting value consideration)
Good usability is when perceived usability is greater than expected usability
Execution view reflects this axiom (a sort of checklist)
Most of critical usability problems can be found in analytical evaluation without actual observation
Execution view reflects this axiom (a sort of checklist)
Usability has a meaning only under specified context of use
Evaluation results using a set of checklist should be interpreted with context of use given
DEVELOPING A CONCEPTUAL FRAMEWORK
Here we need to consider the meaning of conceptual framework and methodological framework. Conceptual framework prescribes features or requirements that need to be represented in a model. In contrast, methodological framework prescribes how to develop a model in a proceduralized way (Vicente, 1999). In this regard, the proposed conceptual framework is just intended to specify information requirements that a model of mobile phone usability impact factors should represent; it is not concerned with processes to develop a model. To develop a conceptual framework, two points need to be considered. Firstly, a conceptual framework should be based on axioms about usability of mobile phones (which are described later). The second point is that a conceptual framework should provide several views that could be useful to develop a model of usability impact. The idea underlying the second point is similar to an architectural framework, which specifies different views helpful in the design of software applications. Axioms for Usability (of Mobile Phones)
In this study, axioms for usability of mobile phones mean a set of facts that can be assumed based on previous studies on this topic, our experience, and opinions from user interface engineers in mobile phone companies. Because the axioms directed our research approach, we have sought to develop a conceptual framework addressing them. Table 1 summarizes these axioms and how we addressed them. Conceptual Framework
Based on the discussions above, we developed a conceptual framework (Figure 2 (a)). The framework is composed of five views, that represent different aspects of interaction between users and mobile phones, and one in particular is related to the actual use of the framework. The five views are: user view, product view, interaction view, dynamic view, and execution view. As stated previously, we intend that the framework could be a basis for developing various usability impact factor models.
Our Approach
Table 1. Axioms and our approach.
Figure 2(b) illustrates how to use different views to derive usability impact factor models. This study developed two models: a hierarchical model and an extended model. This paper focuses on the hierarchical model and will explain the hierarchical model in detail and briefly describe the extended model later. The hierarchical model uses and relates three views: user view, product view, and interaction view, whereas the extended model additionally considers dynamic view to the hierarchical model. It should be noted again that our proposed models are only examples of what could be derived from the conceptual framework. Several models
User view addresses users’ perceived usability and thus interprets usability from the user point of view, such as cognition and emotion. Product view is concerned with usability exhibited by mobile phone itself. Therefore this view is related to technical design knowledge and areas of mobile phones. Interaction view considers tasks a core of usability. Dynamic view is concerned with various user groups and the change of usability along time phase.
263
with emphasis on specific views or their relations can be developed.
obtained by the functional combination of usability criteria. Usability criteria can be measured by applying measurement methods to the properties of mobile phones. These properties have specific value as usability data. Figure 4 explains five abstraction levels of usability impact factors.
Information needs
Usability
Prioritizing can be applied
3()"&4&")#)0,(
Usability Indicator
(a) Five views of conceptual framework Conceptual Distinctions Identify
Identification of impact factors Derive
Different view
List of impact factors
Measurable Concept
Form
Model of impact factors
Use
User view
!
Hierarchical model
Product view
!
Hierarchical model
Interaction view
!
Hierarchical model
Dynamic view
!
Extended model
!
Checklist Quantification scheme
Execution view
Entity
Usability Criteria
Usability Criteria
!"#$%&"'"()* '")+,-
!"#$%&"'"()* '")+,-
Property of Mobile Phone
Property of Mobile Phone
Checklist can be used for this measurement
Figure 3. How to link different impact factor groups.
(b) Use of conceptual framework Figure 2. Conceptual framework. HIERARCHICAL MODEL OF IMPACT FACTORS
This section describes how we developed the proposed hierarchical model on the basis of the conceptual framework, what the main characteristics of the model are, and how it can be used for usability evaluation.
Usability (Quality in Use)
Emergent feature to be characterized with several context factors
Usability Indicator
Hypothetical, abstract conceptual constructs that are not directly measured
Usability Criteria
Sub indicator that can be directly measured through at least one specific usability property
Usability Property (Metric)
Metrics that can be obtained by observing usability data or using a formula that defines a function
Usability Data
Usability Variable that can be obtained from mobile phones, user manual, user tasks, and so on
Figure 4. Five levels of hierarchy model.
Linking Different Groups of Usability Impact Factors
If we consider the functioning of five levels of usability impact factors, we can say that each of five levels is roughly identical with each levels of abstraction hierarchy that has been proved very effective in complex systems (Rasmussen, 1985). Thus we can have the following mapping relations: usability–functional purpose, usability indicator-abstraction function, usability criteria-general function, usability property-physical function, usability data-physical form. As in the abstraction hierarchy, adjacent levels of the hierarchical model have M:N relations.
Three views of the framework indicate that usability impact factors can be categorized into three groups. The first group is human perceived usability (user view) where typical examples include effectiveness, efficiency, and memorability. The second is property exhibited by mobile phones (product view) and examples are reliability, durability, performance, and aesthetics. The third is performance on the tasks (interaction view) and examples are task supportability and error prevention. Here the arising problem is how to link these different impact factor groups, that is, how to evaluate perceived usability by measuring property exhibited by mobile phones and referring to the performance of tasks. Figure 3 illustrates our approach to address this problem. Our approach is hinted by software measurement process framework (ISO/IEC 15939, 2001) and abstraction hierarchy which is popular concept in the area of cognitive systems engineering. To link different levels of impact factors, we adopted concept of goal-means hierarchy relations from abstraction hierarchy. Usability can be interpreted using usability indicators. Usability indicators can
Task Scenarios
.%(/)0,(#1 /,'20(#)0,(
Organization of impact factors
Three abstraction levels, which are located in the middle of the hierarchical model, are particularly important to evaluate usability. The three levels are described in more detail. Usability Indicator
While usability cannot be accurately and fully evaluated in any way, it can be estimated or evaluated by some usability indicators which provide a basis for decision making. On the basis of the literature we reviewed, we concluded that usability of mobile phones can be estimated by five indicators, which are further divided into two dimensions. The five indicators are:
the an by be
264
effectiveness, efficiency, learnability, satisfaction, and customization. Of those, the first three indicators are related to task performance dimensions and they are easy to quantify. But the later two indicators, related to emotion human factors, are not easy to quantify. It should be noted that the usability indicator lies at the abstraction level that does not depend on the kinds of systems or the features of systems. Thus it serves to bridge the gap between perceived usability and usability dimensions dependent on actual systems. In contrast, usability properties and criteria are dependent on actual systems. While the usability property is the abstraction level across all kinds of mobile phones, usability criteria are directly related to specific mobile phone product. Thus usability properties and usability criteria connect general characteristics of mobile phones and particular features of a mobile phone product. For this reason, we need more explanation about various design areas of mobile phones to understand usability properties and usability criteria. Figure 5 shows detailed design areas. S/W area
Logical User Interface (Task Flow/Interaction)
Application software, menu structure and contents, interaction methods, operation sequence, task organization and flow.
Graphical User Interface (Presentation aspect)
Indicator, icons, soft box, menu, main screen (window), field
Physical User Interface
H/W area Device Hardware
Graphical User Interface (Presentation aspect)
Shape, Color, Size, Metaphor, Naming/Labeling, Location, Font
Shape, Color, Size, Brightness, Texture, Balance, Heaviness, Sou Naming/Labeling, Resolution (Camera)
Accessory
nd,
Shape, Color, Size, Texture, Balance, Heaviness, Function implementation
Device Hardware
(Install) software functions, (manual) contents, shape, size, co
lor, etc.
Figure 6. Factors at usability property level. Logical User Interface (Task Flow/Interaction)
Minimalism, Consistency, Error Recovery, Flexibility, Minimal Me mory Load, User Guidance, Error Prevention, Navigation, Informativeness , Information Organization (Grouping), Task Automation, Key Mappin g, Adaptability, Multitasking, Internationalization, Task Support Understandability (Interface Metaphor), Aesthetics, Screen layou Cultural Specificity, Adaptability
Graphical User Interface (Presentation aspect)
Alphanumeric keys, control keys & devices (navigation keys, joysticks, rollers, wheels, soft keys, menu keys, backstepping keys, scrolling keys), up/down side keys (volume keys), power key, special -purpose keys, Microphone, digital camera, sensor, touchpad, touch screen, flat panel display, earpiece, buzzer, vibration mo tor, flashlight
Physical User Interface
Ergonomics, Affordance, Accessibility, Changeability, Luxuriousn Convenience, Attractiveness
Device Hardware
Capacity, Performance, Durability, Reliability, Resource Utiliza Changeability, Functionality, Ergonomics
Accessory
Supportability, Understandability,
t,
ess,
tion,
Installability , Changeability
Figure 7. Factors at usability criteria level.
Body, headset jack, front screen, camera lens, battery, battery cover, battery terminals, battery latch/lock, SIM card socket, SIM card terminals, charger connector, cable connector, handstrap hole
Use manual, battery charger, headset, USB cable, supporting software
Input method, Softkey mapping, Dedicated key mapping, Menu contents, Menu structure, Operation sequence, Feedback, Visual cue, Scrolling method, Naming/Labeling, Navigation method, Function existence, Function automation
Physical User Interface
Overall Hierarchy
Transmitter -receiver, amplifier, filter, oscillator, memory, ASIC processor
Accessory
Logical User Interface (Task Flow/Interaction)
From the description above, we can depict overall hierarchical model as shown in Figure 8. To emphasize again, the main characteristics of the hierarchical model are the M:N goal-means relations between levels and the distinction of design areas. Thus the model seeks to elucidate how the invisible relations between different user interface types or design areas affect the usability of mobile phones.
handstrap ,
Figure 5. Design areas of mobile phones. Usability Property
At the level of usability property, we measure observable features of mobile phones, which have an influence on usability, by directly observing mobile phones or additionally applying some measuring functions (see Figure 6). Usability data below the level of usability property offer actual value for usability property. One thing to note is that usability property can be measured without carrying out tasks. This is another point to distinguish usability property from usability criteria.
The proposed model is similar to the models proposed by Doyanee et al. (2002) and Bosch and Folmer (2004). But the main difference between those models and our model is that they focused on software, not mobile phones. Therefore they do not reflect the features specific to mobile phones. Another difference is that they do not have explicit goal-means relations, taking account of different design areas. Lastly, they were not developed analytically on the basis of a comprehensive conceptual framework like the one developed in this study.
Usability Criteria
As shown in Figure 7, usability criteria evaluate how well usability property is designed to enhance the usability of mobile phone in consideration of task scenarios. Thus usability criteria are aimed to offer usability-related information that is useful to assess how actual, observed features of mobile phones contribute to usability indicator. For example, we can measure ‘minimalism’, which can highly influence on ‘effectiveness’ and ‘efficiency’, by considering softkey mapping, menu contents, and scrolling method.
Usability
Indicator level
Effectiveness
Efficiency
Learnability
Satisfaction
Customization
Criteria level
Logical UI Criteria
Graphical UI Criteria
Physical UI Criteria
Device H/W Criteria
Accessory Criteria
Property level
Logical UI Property
Graphical UI Property
Physical UI Property
Device H/W Property
Accessory Property
Figure 8. Hierarchical model of usability impact factors.
265
Use of Hierarchy
EXTENDED MODEL
The proposed hierarchical model can be used for various purposes. Here we describe two purposes: interpreting evaluation items and analysing usability problems.
Based on the results of the first case study, which is described later, we revised the hierarchical model and added three more considerations. This led to an extended model. As the focus of this paper is not an extended model and there is limited space, we only give a brief description of the extended model.
Interpreting Evaluation Items
Using the hierarchical model, we can interpret usability evaluation items in a more systematic way. For example, in the left of Figure 9, we can say that “functional support of Bluetooth” needs to be evaluated by looking at two factors at the level of usability property (i.e. “functional existence” and “function implementation”), and it is associated with “task support” and “functionality” that have some impact on “effectiveness”. In this way, the hierarchical model helps to understand what mobile phone features are related to an evaluation item and how the evaluation item influences on usability. The second example of Figure 9 can be explained likewise. Thus we can say that a factor related to evaluation item “sound quality of MP3” at the level of usability property is “function implementation”. This factor influences on “performance” and further on “satisfaction” factor. Functional support (e.g. bluetooth )
The second consideration is “pleasure” as an indicator. As an information appliance and multimedia device, mobile phones have increasing features that need to be explained from the “pleasure” point of view. The “pleasure” in mobile phones is composed of three benefits: practical, emotional, and hedonic benefits (Jordan, 2000). Practical benefits refer to functional benefits resulted from performing tasks. Emotional benefits occur when a product affects the mood of users. Hedonic benefits define sensory and aesthetic pleasure associated with the product.
Functional quality (e.g. sound quality of MP3)
Usability
Usability
Effectiveness
Satisfaction
Logical UI Criteria Task Support
The first consideration in the extended model is the variety of user groups. It is obvious that usability is highly dependent on user-related factors. Moreover, different user groups have their core tasks and specialized tasks. Hence, usability needs to be evaluated differently, taking into consideration the characteristics of a user group. There are several criteria categorizing user groups, such as age, social status, ethnicity and occupation; however, there is no absolute answer to what is the best criterion. What is important is that different hierarchical models reflecting the characteristics of each user group need to be developed.
Device H/W Criteria Functionality
Logical UI Property - Device H/W Property Function existence Function implementation
Device H/W Criteria Performance
The third consideration is the time varying change of usability, which is emphasized by the dynamic view. In this study, we divided the phases of mobile phone use into three states: purchase, familiarization, and acceptance. To reflect such a dynamic view, we adopted the three dimensions of design, proposed by Donald Norman (2004). They are visceral design, behavioural design, and reflective design. Visceral design refers primarily to the initial impact and its appearance. Behavioural design is about look and feel from the total experience of using a product. Lastly, reflective design is about thoughts afterwards, how it makes one feel, and the message it tells others about the owner’s taste. Figure 11 illustrates how we relate the three design dimensions to the time phases of mobile phone use. At the phase of selection, visceral design aspects have greater impact on usability than behavioural or reflective designs. In contrast, at the phase of acceptance, reflective designs are regarded as the most influential factor.
Device H/W Property Function implementation
Figure 9. Interpreting evaluation items. Analysing Usability Problems
The hierarchical model can be used for analysing what are the causes of a usability problem and how the problem negatively affects usability. Figure 10 shows one example, the problem of “the soft key menu ‘options’ varies depending on the file view”. The possible causes of this problem at the level of usability property include “dedicated key mapping”, “navigation method”, and “naming/labelling”. These causes can have an influence on “navigation” and “user guidance”, and further on “effectiveness”. Hard to understand: The soft key menu
“options ” varies depending on the file view Usability
Effectiveness Phase 1: Selection (Visceral and Behavioural Bias)
Behavioral
Visceral
Reflective
Logical UI Criteria Navigation, User Guidance Phase 2: Familiarisation (Behavioural Bias)
Logical UI Property Dedicated key mapping (global key), Navigation method, Naming/La beling
Phase 3: Acceptance (Reflective Bias)
Figure 10. Analysing usability problems.
Behavioral
Visceral
Visceral
Behavioral
Reflective
Reflective
Figure 11. Time phases of mobile phone use.
266
The purpose of the second case study was to examine the practicality of the hierarchical model by using the full set of checklist. Another purpose was to provide a simple example about how to apply task-centred usability quantification methods using the weighting value obtained in the first case study.
CASE STUDIES st
The 1 Case Study
One purpose of this case study was to assess the practicality of the proposed conceptual framework and hierarchical model. The second purpose was to collect data to be used for calculating weighting value of tasks, usability indicators and criteria. As the second purpose is not concern of this paper, we do not describe it any more.
Five HCI experts participated in this case study. One expert conducted three tasks (making a call, sending a text, and adding a name and number to address book) and two experts observed and recorded his actions using a video camera. The other two experts evaluated usability by applying the checklist, referring to the task execution. After finishing the tasks, five experts discussed the usefulness of the checklist and the hierarchical model, and suggested points that need to be considered for improving the provisional checklist. Our experience with the two case studies indicated that our approach to identify and organize usability impact factors was viable.
In this study, 22 undergraduate students majoring in product design at Middlesex University, London in the United Kingdom, served as participants, and their age ranged from 19 to 25. In terms of their experience in mobile phone use, there was no big difference. Two participants were grouped into one team, and one participant conducted a task while the other observed it. After one participant finished the task, the roles were then reversed. This was repeated for seven tasks and their actions were recorded using a video camera. After finishing all the experimental tasks, they were interviewed using a questionnaire survey.
A Set of Checklists
Based on the second case study, we developed a revised of the complete checklist reflecting the hierarchical model, which can easily be used for evaluating usability of mobile phones. Figure 12 shows a part of the checklist, which is concerned with LUI. This checklist associates usability criteria with their relevant usability properties and metrics systematically.
The seven experimental tasks were: making a call, adding a contact to an address book, inputting a text message and sending the text, using PIM (Personal Information Management), using WAP (Wireless Application Protocol) services, taking a picture, and changing a phone setting. For the experiment, the participants used their own mobile phones. It therefore should be noted that the data collected for each task is different because tested mobile phones did not provide same user interfaces and set of functions. This difference would not be significant to examine the usefulness of the proposed framework and model because their use is not dependent on the specific user interface types or functions of mobile phones.
Criteria Level
1 Achievem ent
Yes/N o
Score
Engagement: Time to make call
Time (Secs)
3 Error rate
Input method/ D edicated key Mapping
n (Actions)
/ Operation Sequence Actions was repeated successfully
Yes
No
2
2 Flow
4 C onsistency
Yes/N o
-1
1 ~ 10 4
11 ~20 3
21 ~30 2
31 ~ 40 1
0
1 ~3
4~ 6
7 ~9
0
-1
-3
-5
Yes
5 N avigation
Menu Sturcture
Excellent 4
6 Affordance
Mapping of Soft key with interface
7 Multi-tasking (tim e)
U ser is asked to perform while performing
Multi-tasking (B ench m arked)
Time (Secs)
an additional task (e.g. U ndertaking a puzzle)
8 Minim alisation (no. of actions) description
n (Actions) Script:
9 Inform ativess
Feedback
> 40 0 >9 -7 No
2
-1 V.Good 3
Good 2
Average 1
Poor -1
Excellent 4
V.Good 3
Good 2
Average 1
Poor -1
1 ~ 10
11 ~20
21 ~30
31 ~ 40
> 40
4
3
2
1~ 3 0
4~ 6 -1
7~ 9 -3
1
0
10 ~12 -5
> 12 -9
N ote: the inputting of a number does not count. Excellent 4
V.Good 3
Good 2
Average 1
Poor -1
10 Inform ation Grouping
Excellent
V.Good
Good
Average
Poor
11 Quality of Annotation
Excellent 4
4
3 V.Good 3
2 Good 2
1 Average 1
-1 Poor -1
Figure 12. Checklist to evaluate logical user interface CONCLUSIONS
There are a lot of factors affecting usability of mobile phones. Usability needs to be designed and evaluated, taking account of all factors in a unified way. This paper proposed a conceptual framework for identifying and organizing usability impact factors of mobile phones. The framework provides five views which represent different aspects of interaction between users and mobile phones. They include user view, product view, interaction view, dynamic view, and execution view. The framework intends to be a basis from which various usability impact factor models can be developed.
From the questionnaire survey and informal interview with the participants, we realized the importance of a ‘pleasure’ as a usability indicator and the need of integrating ‘learnability’ and ‘customization’ into ‘efficiency’. More literature reviews and discussions with usability engineers in mobile phone companies supported them. Thus, as described previously, we reflected them in the extended model. nd
Metric
Operation Sequence was successful (i.e C all was made)
We tested only LUI aspects of mobile phones in this case study because we thought that it was sufficient only to test LUI for examining the usefulness of the hierarchical model. For this, we developed a provisional checklist to evaluate LUI on the basis of the hierarchical model. In total 11 evaluation criteria were used for evaluating the usability of LUI. Through this case study, we could confirm that the checklist reflecting the hierarchical model was useful to evaluate the usability of LUI and to collect usability-related data.
The 2
Property Level
Logical User Interface
Case Study
Based on the framework, we developed a goal-means hierarchical model of usability impact factors. In this model, usability impact factors that have different abstraction levels are organized in terms of goal-means relations. This model also classifies factors by design areas, such as LUI and PUI. This hierarchical model can
After finishing the first case study, as described earlier, we revised the hierarchical model and developed the extended model. Furthermore, we developed a provisional complete checklist addressing all the five design areas, referring to the checklist of LUI that was used in the first case study. 267
effectively be used for several purposes, such as interpreting usability evaluation items and diagnosing the causes of usability problems. Additionally, we developed an extended hierarchical model, paying particular attention to dynamic aspect of usability and different user groups.
Folmer, E., Van Gurp, J., and Bosch, J. A framework for capturing the relationship between usability and software architecture. Software Process Improvement and Practice 8, 2 (2003), 67-87. Hornbaek, K. Current practice in measuring usability: challenges to usability study and research. International Journal of Human-Computer Studies 64, 2 (2006), 79102.
In order to verify the proposed framework and hierarchical model and to obtain information for improving them, we conducted two case studies. These studies supported the usefulness of the proposed framework and model. To support the use of the hierarchical model, we developed a set of checklist evaluating usability factors by design areas under a task scenario. Furthermore, though we did not describe in detail in this paper, we developed a quantification scheme in this study, which gives a usability score of a mobile phone, applying various weighting value assignment methods. Thus the scheme takes into consideration users’ preference of usability factors.
ISO/IEC 15939. Software engineering-Software measurement process framework (2001). ISO/IEC 9126-1. Software engineering-Software product quality-Part 1: quality model (1998). ISO/IEC 9241. Ergonomic requirements for office work with visual display terminal-Part 11: guidance on usability (1998). Jordan, P. Designing Pleasurable Products: An Introduction to the New Human Factors. Taylor & Francis (2000).
Although structuring various usability impact factors in terms of goal-means relations was shown to be useful, the goal-means relations themselves were not empirically validated. To enhance the practicality of the proposed model, they should be thoroughly examined by conducting a questionnaire survey with large number of users and by applying advanced statistical methods. Lastly, a detailed methodological framework to be complementary to the proposed conceptual framework remains as a matter to be developed.
Ketola, P. and Röykkee, M. The three facets of usability in mobile handsets. In Proc. CHI 2001 Workshop (2001). Kiljander, H. Evolution and usability of mobile phone interaction styles. Unpublished Ph.D. Thesis at Helsinki University of Technology (2004). LG Electronics. B2050 http://www.lge.co.uk
financially
supported
by
User
Guide.
Nielsen, J. Usability Engineering. Academic Press (1993).
ACKNOWLEDGMENTS
This research was Electronics Inc.
Handset
Norman, D.A. Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books (2004).
LG
Rasmussen, J. The role of hierarchical knowledge representation in decision making and system management. IEEE Transactions on Systems, Man, and Cybernetics 15 (1985), 234-243.
REFERENCES
Bevan, N. Quality in use: Meeting user needs for quality. The Journal of Systems and Software 49 (1999), 89-96.
Vicente, K.J. Cognitive Work Analysis. Lawrence Erlbaum Associates (1999).
Bosch, J. and Folmer, E. Architecting for usability: a survey. The Journal of Systems and Software 70 (2004), 61-78.
Weiss, S. Handheld Usability. John-Wiley & Sons (2002).
Donyaee,, M, Seffah, A., Kline, R., and Striva, J. An integrated framework for usability measurement. In Proc. 12th International Conference on Software Quality (2002), 1-10.
Zhang, Z. Overview of usability evaluation methods. http://www.usabilityhome.com/ (2003)
268
