A Model for Mobile Television Applications Based

A Model for Mobile Television Applications Based on Verbal Decision Analysis Isabelle Tamanini, Thais C. Sampaio Machado, Marília Soares Mendes, Ana Lisse Carvalho, Maria Elizabeth S. Furtado, Plácido R. Pinheiro University of Fortaleza (UNIFOR) – Graduate Course in Applied Computer Science (ACS) Av. Washington Soares, 1321 - Bl J Sl 30 - 60.811-341 - Fortaleza – Brasil {isabelle.tamanini, thais.sampaio, mariliamendes, ana.lisse}@gmail.com, {elizabet, placido}@unifor.br Abstract - The emergence of Digital Television - DTV brings the necessity to select an interface that may be used in Digital Television for Mobile Devices. Many different candidate solutions for interaction applications can be possible. This way, usability specialists prototyped interfaces that were analyzed according to users’ preferences through usability test, considering criteria classified in accordance with users' preferences and alternatives, which were obtained from a ranking modeled by verbal decision analysis. Results revealed great influence of the application's functions evidence in the ease of navigation.

I. INTRODUCTION In domains that represent a new paradigm of interactivity (as Digital Television - DTV, smart home and tangible interfaces), the decision of the most appropriate interaction design solution is a challenge. Researchers of the Human-Computer Interaction (HCI) field have proposed in their works the validation of the design alternative solutions with users before developing the final solution. Taking into account the users’ satisfaction and their preferences is an act that has also gained ground in this kind of work when designers are analyzing the appropriate solution(s). Recent research reveals that the understanding of subjective user satisfactions is an efficient parameter to evaluate application interfaces [1]. In domain of interaction design for DTV, we assume that it is necessary to consider both international aspects for supporting the accessibility for all and digital contents for supporting a holistic evaluation (content and user interface) of the TV applications that show a content through their user interfaces. Structured methods generally consider quantitative variables (such as: amount of errors, number of executions of the function “Help” by the user, time spent to find a new function, etc). Research has evaluated qualitative aspects, like user satisfaction and his emotion when manipulating the technology, through the observations and comments obtained during the usability tests [2]. The users generally are encouraged to judge the attractiveness of the application interface, and from these judgement, evaluators elicit qualitative preferences [3]. The aesthetic quality of a product influences in users’ preferences but other qualitative aspects may influence in their judgments more, so that transcend the aesthetic appearance [4]. When subjective questions were understood, another problem deliberated for the research of the technical report is about traditional way of evaluation. It is quite a little hard to create new project alternatives and new ways to consider these alternatives because of the low flexibility. For example, designers evaluated two interface solutions when they applied usability tests, and selected one of them to implement. During the system development, more three design solutions emerged as a result of new usability pattern.

How can designers consider these new alternatives? How do they evaluate if a new solution is better than the old one? Traditionally, usability tests should be applied to all alternatives. With a multicriteria model, these decisions are efficient and only some alternatives would be evaluated. In this project, three interface solutions for Mobile Digital Television Application were evaluated qualitatively by applying a verbal decision analysis methodology. This strategy maps correctly the provided information of user’s preferences, which helped the judgement of the project solutions. Provide a holistic evaluation of interactions situations and more information to understand and organize subjective questions. The analysis of design solutions becomes more flexible. The ranking generated by the model is a tool which makes easier to insert new alternatives and judgements for interfaces. In order to be able to use the model, hypotheses were elaborated. These hypotheses consider important characteristics for mobile DTV interaction. From these hypotheses, criteria were established as well as usability tests were applied in order to obtain information about user’s preferences. The method ZAPROS III [5], which belongs to the Verbal Decision Analysis framework, was used. The method must be applied to problems that have qualitative nature and that are difficult to be formalized. Problems called unstructured [6]. The Aranaú tool [7] was developed with the aim of applying the verbal decision analysis method ZAPROS-III, helping on modeling unstructured problems. II. HYPOTHESES AND EVALUATION SCENARIO The following hypotheses were the basis to elaborate the multicriteria model which adheres to the reality of evaluations for applications of mobile Digital Television: • Hypothesis 1: The evidence of the functions in the application facilitates the use and influences in the effort spent by the user to localize himself in the application; • Hypothesis 2: The user experience with applications which have similar ways of navigation will influence the choice of an interface that has facility of use, accuracy and user satisfaction; • Hypothesis 3: The locomotion of the user while manipulating the device will influence in the choice of the interface which requires less precision to navigate between the options and screens than the others interfaces, so that facilitates the navigation of a person while manipulating the application; • Hypothesis 4: The involvement with the content influences in the user choice, so that, if the content is interesting, it may be decisive for the user to choose the interface;

• Hypothesis 5: The emotion felt by the user when using the interface exercises a considerable influence in the choice. Once the hypotheses were defined, usability tests with three mobile DTV prototypes were elaborated ( Fig.s 1, 2, and 3). The tests helped to elicit user preferences. The results of the tests were entered as data for the model. The usability tests were applied with young users who had wide experience with palms, DTV and desktop computing devices. The users evaluated were 12 university students and the duration of

Fig. 1. Prototype 1. Similar to TVD applications

the test for each user was between 20 and 30 minutes. Two different locations were used: the usability laboratory (LUQS) and a natural environment (field study). Interface Designers and Usability Specialists were present during the tests process. For each user, the test began showing them a sample portal application for digital TV [2]. This was done to the user know how the digital TV works. The next step is the execution of the usability tests with mobile DTV application. The tests begun as follows: before using the applications, the users were interviewed (using a questionnaire) in order to study their experience and opinions. They were also informed about how the test would be conducted. During the use of the application, the users were observed and specialists marked a checklist of questions. The user should execute four scenarios, and after each one, he should fill out a questionnaire about the context of the tested scenario. In addition, the users were monitored with cameras while using the application (with software that collected and stored results). The scenarios considered subjective aspects of each user. For example, if the user was a soccer fan who had a lot of experience with desktop applications, one example of scenario could be to execute sports programs in each one of the three prototypes. Should the solution for the project chosen by this user be something similar to desktop applications or the user who executes the scenario in movement would choose a solution similar to a DTV portal? These questions are directly connected to the hypotheses listed in the beginning of this section. Next we present a summary of the ZAPROS III method in order to get a better understanding how it works out. III. VERBAL DECISION ANALYSIS

Fig. 2. Prototype 2. Similar to Palm applications, navigation with scroll

Fig. 3. Prototype 3. Similar to Desktop applications, navigation with scroll

The ZAPROS III method belongs to the Verbal Decision Analysis – VDA framework. It combines a group of methods that are essentially based on a verbal description of decision making problems. This method is structured on the acknowledgment that most of the decision making problems can be verbally described. The Verbal Decision Analysis supports the decision making process by verbal representation of the problem [8]. ZAPROS III was developed with the aim of ranking given multicriteria alternatives, which makes it different from other verbal decision making, such as ORCLASS [9], and PACOM mainly due to its applicability. It uses the same procedures of elicitation of preferences, however with innovations related to the following aspects [5]: 1. The procedure for construction of the ordinal scale for quality variation and criteria scales are simpler and more transparent; 2. There is a new justification for the procedure of alternatives comparison, based on cognitive validations [10]; 3. The method offers both absolute and relative classification of alternatives. The method ZAPROS III can be applied to problems having the following characteristics [5]: The decision rule must be developed before the definition of alternatives; There are a large number of alternatives; Criteria evaluations for alternatives definition can only be established by human beings;

The graduations of quality inherent to criteria are verbal definitions that represent the subjective values of the decision maker. The decision maker is the key element of multicriteria problems and all necessary attention should be given in order to have well formed rules and consistent and correctly evaluated alternatives, always considering the limits and capacity of natural language. Thus, the order of preference will be adequately obtained according to the principle of good ordering established by Zorn’s lemma [11]. The application of the method ZAPROS III for problem modeling can be applied following this three-step procedure: a) Definition of Criteria and their Values: Once the problem is defined, the criteria related to the decision making problem are elicited. Quality Variations (QV) of criteria are established through interviews and conversations with specialists in the area and decision makers. b) Organization of Ordinal Scales of Preference An ordinal scale of preference for quality variations for two criteria is established based on pairwise comparisons. The preference between these two criteria is chosen according to the decision maker and the obtained scales of preferences are denominated Joint Scale of Quality Variation (JSQV) for two criteria. When carrying out the comparisons, it is assumed that there is an “ideal” alternative based on the decision maker’s preferences. From this ideal case, questions are made to the decision maker who will answer according to his preferences in relation to the other criteria values. In this way, the scale can be elaborated either by direct answers of the decision maker or by transitive operation [12], which helps diminish the quantity of necessary comparisons. The transitivity also helps the check of independence between criteria and groups of criteria as well as the identification of contradictions on the decision maker’s preferences. Dependence of criteria and contradictions should be eliminated by formulating new questions to the decision maker and remodeling the criteria (with possibility of carrying out a new formation of natural language and identification of other quality variations) [13]. c) Comparisons of Alternatives The ranking of the alternatives is constructed by comparisons between pairs of alternatives. Considering a group of alternatives, the elaboration of a partial order for these alternatives follows a three step algorithm: Step 1: Formal Index of Quality (FIQ); Step 2: Comparisons of Pairs of Alternatives; Step 3: Sequential Selection of non-dominated nuclei. By these phases (a, b and c), a problem modeled based on the ZAPROS III method results in an ordering of alternatives. The ordering gives us a quantitative notion of the order of preference, in an absolute way (in relation to all possible alternatives) as well as a relative way (in relation to a restricted group of alternatives) [14]. The next session will present the modeled study case and the validated hypotheses. IV. COMPUTATAONAL RESULTS The criteria used in the evaluation were established with the assistance of specialists in the usability of mobile DTV of the Usability and Quality of Software Laboratory (LUQS, of University

of Fortaleza). The specialists wanted to analyze the aspects that had greater influence on choosing a determined project interface. According to the hypotheses previously given, the following criteria were verbally defined: 1. Functions Evidence, which indicates whether the users are able to find easily the functions of the system. The user will prefer an interface which its use is probably easier for him. 2. User’s familiarity with a determined technology, which implies that if an interface is similar to one in a determined technology familiar to the user, this interface is preferable to him, since he is used to it. 3. User’s locomotion while manipulating the device, which infers whether the interface allows a good spatial orientation for the user, which doesn’t demand too much attention of the user while manipulating it. The user may choose an interface that has this feature instead of one with a familiar appearance or with excellent content evidence. 4. Content Influence; when the user uses an interface that has a content which attracts him, he may prefer this interface, although he is more interested in the content then in the interface itself. 5. User Emotion; considering that if the user feels fine when using the interface, he will want to use the device always more. It means that a good emotion makes the user want to use the interface. With conditions defined, the method ZAPROS III can be applied using the Aranaú tool, which implements Verbal Decision Analysis, according to our following presentation. Fig. 4 shows the definition of the criterion “Content Influence”. Table 1 shows the values of all criteria related to the aspects on which the definition of the attractiveness levels among the interfaces is based.

Criteria

TABLE I CRITERIA AND ASSOCIATED VALUES Values

A - Functions Evidence

B - User’s familiarity with a determined technology

C - User’s locomotion while manipulating the device

A1. No difficulty was found on identifying the system functionalities; A2. Some difficulty was found on identifying the system functionalities; A3. It was hard to identify the system functionalities. B1. No familiarity is required with similar applications of a determined technology; B2. Requires little user familiarity with applications of a determined technology; B3. The manipulation of the prototype is fairly easy when the user is familiar with similar applications. C1. The user was not hindered in any way when manipulating the prototype while moving; C2. The user was occasionally confused when manipulating the prototype while moving; C3. The spatial orientation of the application was hindered when the user was moving.

D - Content Influence

E - User Emotion

D1. There is no influence of content on choosing the interface; D2. The content exerted some influence on choosing the interface; D3. The content was decisive on choosing the interface. E1. He felt fine (safe, modern, comfortable, etc.) when using the interface; E2. He felt indifferent when using the interface; E3. He felt bad (uncomfortable, unsafe, frustrated) when using the interface.

Fig. 5. Example of preferences elicitation using Aranaú Tool.

Fig. 4. Example of the definition of criterion “Functions Evidence” using Aranaú Tool.

The order of preference among the criteria values was established by observing the results of the tests during its application. For example, it was observed that when the users were moving and trying to execute a task in a determined prototype, they complained that it was difficult to move and manipulate the device at the same time. After the tests, the responses to the questionnaires were gathered and evaluated. Questions like “What prototype did you prefer? And Why?” indicated the order of preference among the project alternatives and also which criteria values were decisive for the choice. Fig. 5 shows an example of preference elicitation on Aranaú Tool. The Joint Scales of Quality Variation (JSQV) for two criteria was gradually elaborated and validated with the gathered information of the tests. After this, the Joint Scale of Quality Variation for all criteria is elaborated. The obtained JSQV, from better to worse values, is: a1 a2 b1 b2 c1 e1 d1 e2 d2 b3 d3 c2 c3 a3 e3. After the construction of the JSQV for all criteria, the comparisons of the interfaces were made. Each alternative was studied in order to define which criteria values materialized the prototypes. The usability tests also supplied important information about how the users described the interfaces (for example, the

majority of users said that access to content using prototype 3 (three) was quite easy – the criterion value “B1”, but it required a lot of familiarity with Desktop applications – the criterion value “A3”). Finally, the established relationship was: Prototype 1 - A2 B1 C2 D1 E2 (Alternative 1); Prototype 2 - A2 B3 C1 D1 E1 (Alternative 2) and Prototype 3 - A2 B1 C1 D1 E2 (Alternative 3). Each Quality Variation - QV of JSQV is numbered in ascending order from 1 (one) to 9 (nine). The sum of the determining QV numbers for each alternative is the Formal Index of Quality - FIQ [4]. Fig. 6 presents the FIQ value of each alternative and the resultant ranking. With the FIQ values, the ranking of the prototypes is organized by assumption that the alternative with the lowest FIQ value represents the highest rank and the best alternative. The alternative with the highest FIQ value is the least preferred prototype. V. DISCUSSION The resulting rank and the preference scale between two criteria values prove that the evidence of the functions in the application facilitates the use and influences the choice of the interface, which is easier to use, more exact and more satisfying for the user. This

Fig. 6. Alternatives Ranking calculated by Aranaú Tool.

influence is a determining factor for the choice of the most preferable prototype solution. The value A1 was the top value in the scale of criteria values, demonstrating the Hypothesis 1. The criterion value B1 came just after A1 and A2, showing that user’s experience with similar types of applications of navigation influences the choice of the interface. So, Hypothesis 2 was proved. Considering the Hypothesis 3, since the criterion value C1 was less preferred than A1 and B1, we observed that when the user is moving and interacting with the interface, he chooses the interface that is easiest to use (the one he has more affinity and ease of access to more interesting content). The interesting content leads the user to choose the interface he had greatest affinity. This affinity is determined by the degree of similarity with applications commonly used by the user. The criterion value B1 came just after A1 and A2, showing that affinity is a determinant less important than the content accessed no matter how important or attractive this content is. Hypothesis 4 was proved because D1 is located after all others best criteria values. But the involvement with the content is important, so D2 and D3 are in the middle of scale. Hypothesis 5 “The emotion felt by the user when using the interface exercises a considerable influence in the choice” was demonstrated completely by criterion E (User Emotion). We noticed with the current model that criterion value E1 belongs to the first part of the scale, what means that it has a great influence in the user’s choice. It is important to integrate characteristics of criterion access to the content whit functions evidence. So that if the content is attractive, the user tends to choose the interface that more evidences the functionality to access the content. For example, if the user likes soccer games, he will feel fine if he could select a function to evident the sport in the application. The ranking of the alternatives showed prototype 3, similar to Desktop applications, as being the most preferred. Prototype 2, similar to palm applications, with a Formal Index of Quality value very close to the FIQ of prototype 3, demonstrated that their difference of attractiveness is quite modest. Prototype 1, similar to DTV applications, was proved inadequate for the content of DTV in mobile device. The high FIQ value related to the others prototypes shows the low attractiveness for this type of DTV application. Although ZAPROS III is for a large number of alternatives, this technical report analyses three alternatives. The model represents a guide to develop new solutions, which are going to integrate a new Rank. It will show the Verbal Decision Analysis flexibility related to the insertion of new alternatives in future works. VI. RELATED WORKS A discussion of user-centered development process considering real people’s needs is described in [4]. The authors use to mention that an interaction design made for a device does not fit to another one, but it does not specify any strategy to solve this problem. Behind, On the contrary of this study, that shows an experience which three solutions of design for different devices were defined and analyzed based on how users experience each solution. Our goal was not to find usability problems in the tests, but help designers to understand which criteria related to users’ experience could influence their decision and to discover users’ preferences. Reference [15] describes a multicriteria approach in which the

execution of its steps allows to identify the order of attractiveness of a list of usability interfaces for a certain interactive DTV application task, allowing the selection of the most appropriate interface to this new communication resource. However, the interface applications were not evaluated through executable prototypes. In this experiment and using the method ZAPROS III, a qualitative analysis about the users’ preference and their intentions of use with executable prototypes could be better appreciated. Reference [16] shows a multicriteria model with ZAPROS III applying three criteria: familiarity of the user with a determined technology, attractiveness of the task and locomotion of the user while manipulates the interface. VII. FUTURE WORK Given the complexity of HCI and the use of multicriteria analysis, our next goal is to develop a Collaborative Design method in order to assist the method ZAPROS III on the phases: Definition of Criteria and Organization of the Ordinal Scales of Preference, providing for more support on redefining and discovering criteria, and on clarifying the real preferences of users, designers and usability engineering. Results of usability tests are entries for Collaborative Design, which prepare supplementary material to accomplish the method ZAPROS III’s purpose. The design collaboratorium has been developed as a reaction against the failing capabilities of classical usability methods to cope with mobile and ubiquitous technologies [17]. The ZAPROS III method yielded results with respect to the criteria evaluated such that user familiarity with similar applications is a determining factor for ease of use of the interface. The usability interface for mobile Digital Television applications should strongly consider the most used applications by the target clientele. The method ZAPROS III proved to have characteristics such as flexibility, so that new project alternatives can be added, allowing researchers to have a better understanding of the needs and opinions of mobile DTV target users. The method has also helped usability specialists on understanding the relationship (order of preference) among the criteria frequently used for a project interface. The importance of formally evaluating the subjective aspects involves the analysis of which usability standards should be employed for this mobile DTV application. Prototypes using new usability standards are being developed in LUQS. Thus, the ranking supplied by this research will be increased and there will be new contributions in future studies. A research is also being developed to discover how to validate the hypotheses using quantitative metrics. What metrics are possible for each hypothesis? Could these metrics be entry points for information used to elaborate a multicriteria model such as ZAPROS III? The criteria used on hypothesis 5 are still being researched. Hybrid tests (qualitative/quantitative) are being developed using multicriteria. VIII. CONCLUSION It is important to point out that our intention was not to compare navigation techniques (as scrollbars, tap-and-drag, and so on) on mobile devices to identify the best one when users are performing navigation and selection tasks. Our goal was to help designers to

understand how criteria related to users’ experience could influence their preference for a final solution. In addition, we showed how to integrate two different areas (HCI and OR - Operational Research) describing an approach for evaluating the Interaction design in a subjective perspective of OR. It means that researchers interested in making qualitative analysis of the interaction can use this proposal, which leads to more objective results. ACKNOWLEDGMENT The authors are thankful to Celestica of Brazil for the support they have received for this project. REFERENCES [1] [2]

[3] [4]

[5]

K. Chorianopoulos and D. Spinellis, User interface evaluation of interactive TV: a media studies perspective, Univ Access Inf Soc; 5: 209–21, 2006. E. Furtado, F. Carvalho, A. Schilling, D. Falcão, K. Sousa, F. Fava, Projeto de Interfaces de Usuário para a Televisão Digital Brasileira. in: SIBGRAPI 2005 – Simpósio Brasileiro de Computação Gráfica e Processamento de Imagens, 2005. Natal, 2005. N. Tractinsky, A. S. Katz, D. Ikar, What is beautiful is usable, Interacting with Computers, Volume 13, Issue 2, December 2000, Pages 127-145. A. Angeli, A. Sutcliffe, J. Hartmann, Interaction, Usability and Aesthetics: What Influences Users’ Preferences?, Symposium on Designing Interactive Systems, Proceedings of the 6th ACM conference on Designing Interactive systems, Pages: 271 – 280, 2006. O. Larichev, Ranking Multicriteria Alternatives: The Method ZAPROS III, European Journal of Operational Research, Vol. 131, 2001.

[6] [7] [8] [9] [10] [11] [12] [13]

[14]

[15]

[16]

[17]

H. Simon and A.Newell, Heuristic Problem Solving: The Next Advance in Operations Research, Oper. Res., vol. 6, pp. 4-10, 1958. I. Tamanini, P. R. Pinheiro, A. L. Carvalho, Aranaú Software: A New Tool of the Verbal Decision Analysis, Technical Report, University of Fortaleza, 2007. O. Larichev and H. Moshkovich, Verbal Decision Analysis For Unstructured Problems, Boston: Kluwer Academic Publishers, 1997. A. I. Mechitov, H. M. Moshkovich, D. L. Olson, Problems of decision rules elicitation in a classification task, Decision Support Systems, 12:115–126, 1994. O. Larichev, Cognitive validity in design of decision-aiding techniques, Journal of Multi-Criteria Decision Analysis, 1(3): 127–138, 1992. P. R. Halmos, Naive Set Theory, Springer, 116 p., 1974. O. Larichev, Psychological validation of decision methods. Journal of Applied Systems Analysis, 11:37–46, 1984. J. Figueira, S. Greco, M. Ehrgott,(Eds.), Multiple Criteria Decision Analysis: State of the Art Surveys Series: International Series in Operations Research & Management Science, Vol. 78, XXXVI, 1045 p, 2005. H. Moshkovich and O. Larichev, ZAPROS-LM– A method and system for ordering multiattribute alternatives, European Journal of Operational Research, 82:503–521, 1995. K. S. Sousa, H. Mendonça, M. E. S. Furtado, Applying a Multi-Criteria Approach for the Selection of Usability Patterns in the Development of DTV Applications, In: IHC'2006, 2006, Natal. IV IHC'2006, 2006. A. L. Carvalho, M. Mendes, P. Pinheiro, E. Furtado, Analysis of the Interaction Design for Mobile TV Applications based on Multi-Criteria, International Conference on Research and Practical Issues of Enterprise Information Systems (CONFENIS 2007), October 14-17, Beijing, China. S. Bødker, The Design Collaboratorium – a Place for Usability Design. ACM Transactions on Computer-Human Interaction, Vol. 9, No. 2, , Pages 152–169, June 2002.