Usability testing of an Electronic Programme Guide and Interactive TV applications Pedro Concejero, Santiago Gil, Rocío Ramos, José Antonio Collado, Miguel Ángel Castellanos Human Factors Group. Telefónica Investigación y Desarrollo Emilio Vargas, 6 E-28043 MADRID (SPAIN) Tel: +34 91 337 3943 Fax: +34 91 337 4491 E-mail:
[email protected] Abstract This paper describes the usability testing of the Electronic Programme Guide (EPG) and some Interactive TV applications developed by Telefónica I+D for the Spanish satellite-TV operator Vía Digital. Fifty six people participated in the tests. In order to have a representative sample of the general population, participants were sampled following sex and age quotas from a total of 346 people randomly contacted by telephone. The paper also presents the results of the small telephone survey about TV viewing habits and the opinions about the advances in TV technology. The environment where the trials were made was a domestic setting, and people were asked to use a real working prototype, and to perform typical user tasks, as, e.g., searching for a particular program in the EPG. After each trial, the experimenter took note of the problems faced by the user, and whether the task was successfully completed, and then asked the subject to fill a short questionnaire about the usability of that particular application, including questions about difficulty, presentation speed and aesthetic appearance. At the end of the tests, subjects were also briefly interviewed to obtain the global opinion about the system, and including questions about some marketing issues of digital TV applications. Detailed results about the usability of the Programming Grid, and of the Theme and Channel Search functions are provided. After these results, design guidelines and user interface alternatives were produced, especially for the Programming Grid, which was considered by most subjects the most difficult part of the EPG. Two interactive TV applications, Tele-banking and Weather Channel, were also tested. These applications presented different difficulty levels for the users, and in particular, the two user interface approaches of the two interactive TV applications produced important differences in the usability of the system.
1
Introduction
Background This paper describes the results and conclusions from the usability tests of the Electronic Program Guide (EPG) developed for the Spanish satellite TV operator Vía Digital by Telefónica Investigación y Desarrollo (Telefónica I+D). These tests were carried out in November 1997. Telefónica I+D was formed in 1988 to contribute to the technological innovation of the parent company Telefónica de España, by performing research and development activities. Over the last few years, the line of work of Telefónica I+D has evolved to conform with the objective of becoming a services creation lab. This objective is driven by the emergence of interactive services which are increasingly becoming a mayor force in the telecommunications market. Telefónica owns a significant share in the Spanish satellite TV operator Vía Digital, for whom Telefónica I+D is developing many applications. The Electronic Program Guide (EPG)
Figure 1: First level menu of the EPG
Figure 2: The usability lab for the tests
One of the applications being developed is the EPG, which is the user interface for selecting the channels and for getting information about the programming on each of them (ETSI ETS 300 707, TR 101 288). Satellite TV provides many more channels than traditional TV, thus making troublesome for the user to find the desired programs. Satellite TV provides also an excellent platform to provide simple interactive applications, as, e.g., telebanking, weather forecast information, traffic conditions, etc. The main concern in the development of these applications has always been their usability, since the target population is very general, and especially because it adds new functions to a currently very simple system, the traditional TV. Therefore, the stress was put on the representativeness and the validity of the results. This is the main reason for the complex way in which subjects were chosen, using a quota sampling procedure carried out by a marketing research bureau. This was also used to get some knowledge of people's attitudes towards TV in general, and satellite TV in particular. A small report on the results of this survey is presented here.
Another added research objective was obtaining commercial information which could be useful for the marketing of the services. Nowadays, these issues are closely related to the usability issues, and the trials represented a good chance of collecting this information. 2
Materials and procedures
2.1
The usability lab
The trials were carried out in the usability lab of Telefónica I+D, where a normal sitting room was simulated as best as possible. This setting is depicted in Figure 1. Subjects were sitting on an armchair in front of a TV set, at a distance 5 times the height of the display. The lab was illuminated as in a domestic setting, using normal lights. There was a light behind the TV set to improve the perceived image quality. The experimenter was sitting close to the subject, as in the figure, and took notes or helped him or her whenever required. The trials were video-taped using a device which split one single image into four sources of video from the trials. These are depicted in figure 2. Clockwise from the upper left corner, the images present: a global view of the lab, the TV screen, an image from the back of the subject and last, the image taken from a mini-camera on top of the subject, and which recorded his or her hands using the remote control.
Figure 3: Image from the videotape with the four views. 2.2
Description of the trials
The trials begun with a short briefing to the subject, to inform him or her about the aims of the trials, and a training phase, explaining the general operation of the remote control and the basic components and functions in the Vía Digital EPG. It was emphasized during the briefing that the aim of the trial was collecting the subjective opinion of the usability of the user interface and the components of the system, and not a detailed performance measurement, so that the subject felt easy during the tests and operated the system in a normal way. After this introduction the subject performed an example task, in which he or she used all of the basic functions, and filled the first page of a questionnaire, to get used with this tool (code named part A in the trial). The part of the EPG involved in this training was not used
during the main trials. It was the option "In this moment" of the main menu, which was the easiest of all the possible functions in the system. After finishing every trial task, the subject had to fill in the questionnaire its subjective opinion about some task characteristics (these were the variables for the analysis): B1. The Program Grid This component presents the timetable of programs for each channel in the current day. User has to use the scrolling arrows in the remote control to move for and backwards, up and down in the grid. In the inferior bar, the menu functions are displayed. User had to search for the ending time of a football match in the sports channel. He or she had to use vertical and horizontal scrolling functions to find this information. B2. Thematic Search The user finds here a classification of channels as a function of themes (e.g., films, news, sports, children, etc.). Once chosen the theme, the system presents all channels in the category, and then displays the program grid for each of them. The tasks proposed was to search for the name of the film that it is going to be shown tonight in one channel (called "Solo cine").
B3. Channels Menu There is a channels menu in a top bar. Once selected a category in this bar, the channels are displayed in the box at the right side. Information about the contents of that channel is displayed in a box at the right side. The user had to use the option channels from the first level of the EPG, and then select the news channels, and watch the channel called "24 Hours". C1. Calendar This application presents a calendar so that the user can select a particular day, and then view the programming for that day. Apart from the traditional calendar, the interface provides icons for different parts of the day: morning, afternoon, evening and night. Subject had to use a calendar to find out the programming of another day.
C2. Weather Forecast This is an application which provides information about the weather in Spain. It provides this information for the whole country, for regions and for capital cities. The day of the forecast can be chosen. The user had to get the information about the weather in Vitoria (a Spanish city) tomorrow.
C3. Telebanking Application (BBV) This is an application which allows the user to compute the rates for loans, among other things. The interface is far more simple than for the other applications and components, offering only a few options. Subject had to compute the monthly fee for a loan to repair a house. He or she had to introduce some specific data about the loan (total amount, time to cancel, etc.)
Table 1: Tasks and screenshots used during the trials. The different tasks were devised so that they were representative of a person's normal way of using TV. The tasks were divided into two parts: part B presented basic components of the EPG, like channel selection, and getting information about the programming; part C presented interactive applications, like weather forecast and telebanking. To avoid interference between the two types of applications, and in order to avoid order effects in the measurements, parts B and C were counterbalanced separately. Table 1 presents the tasks used, and screenshots of the interfaces. During the trials, a number of subject's actions were registered by the experimenter in a register sheet, especially if the user needed help or not, which was the most important measure we wanted to analyze in the trials. After ending each task, subject filled a questionnaire to evaluate five attributes of the system, using a rating scale (0 –nothing at all to 10 –completely): - Facility of the task. - Utility to achieve the goals. - Speed of the system response, and whether it was considered acceptable or not. - The aesthetic appearance of the screens. - Help for the navigation offered in all the screens. After all the trials finished, the subject was asked to fill a separate questionnaire to obtain his or her opinion on the whole system. These global features were essentially the same as those asked for the particular components, and including a rank order of preference for them. Finally, after the subject had completed this questionnaire, the experimenter made a structured interview, asking commercial questions about this type of TV system 2.3
Subjects and sampling
The objective of the trials were having a representative sample of the general customer population for this kind of service, therefore a quota sampling procedure was chosen. The
objective was then having the same number of participant in each of the following categories: 3 age groups (14-30, 31-44 and 45-55 years old) and sex (male/female). Sex / Age (years old)
14 - 30
31 - 44
45 – 55
Total
Men Women Total
9 9 18
9 9 18
9 9 18
27 27 54
Table 2: Distribution of participants by sex and age. Sampling quotas Subjects participating in the trials were chosen by means of a sampling procedure, by means of a telephone interview. This was made based on a structured questionnaire, including questions on the use of TV and on the knowledge of the different companies operating the service being tested. The information was taken from 13th to 17th November 1997, with the target of the general population of Madrid (Spain), age 14 to 55 years. Result of the telephone call Completed Interviews Do not answer Wrong number Negative Not completed Out of quota TOTAL
Number of telephones 346 80 83 384 26 608 1.527
Table 3: Results of the telephone interview for the total number of calls made. Out of the total of completed interviews (the 22.7% of the total), 163 accepted to participate in the trials. This represents the 47% of the interviewees, but only a 10.6% of the total number of telephone numbers contacted for the trials. Only 1 or 2 calls were needed for the 90% of interviews within the quotas. Interviews took, as an average, 3.5 minutes, and it is very seldom that they took 7 minutes or more. Subjects were paid 5000 pts. for their participation in the trials. 3
Results
3.1
Tasks performance
One of the most interesting results for these trials was whether participants were able to perform the proposed tasks by themselves, or on the contrary they required the experimenter’s help, or consulting an external source of help, apart from the own TV system, to achieve the goal. This was registered by the experimenter in a sheet, and also registered in the viewing of the video-tapes from the trials. Global results can be seen in Figure 4. The most important result is that less than one third of the participants were able to end the task for the Programming Grid by themselves, i.e., without the experimenter's help. The rest of the applications have better usability, and the telebanking application is that with the highest percentage of success. There are no differences as a function of sex or age groups in the performance for the programming grid, but there are significant differences in performance for task B2 (thematic
search) as a function of age group, (χ2= 18.94, 2 d.f., p
