Designing Search User Interfaces

Designing Search User Interfaces Anders Salander

April 3, 2011 Master’s Thesis in Computing Science, 30 credits Supervisor at CS-UmU: Lars-Erik Janlert Examiner: Jerry Eriksson

Ume˚ aUniversity Department of Computing Science SE-901 87 UME˚ A SWEDEN

Abstract Searching for information has become a natural task in today’s society. Nowadays more than 2 billion people are connected to the Internet and using some kind of search functionality for finding information is a central activity. The amount of digital information that is produced puts high constraints on the design of search interfaces. This master thesis report presents a proof-of-concept design and implementation of a search user interface. The main focus of the project is usability, especially in the aspects of navigation and content presentation. There are many features that can support the user during the search activity and this interface has been developed on the basis of these features. The report describes an iterative user-centered design process that served as the foundation during the proof-of-concept development. The target user group has been involved throughout the project and the design has been evaluated on several stages in the design process and the proposed interface was well received by the target users. An interactive high-fidelity prototype was implemented in Adobe Flex 3 for the purpose of evaluation and demonstration.

ii

Contents 1 Introduction 1.1 Background . . . . . . . 1.2 The eX-IFS application 1.3 Target group . . . . . . 1.4 Assignment . . . . . . . 1.5 Method and Results . . 1.6 Limitations . . . . . . . 1.7 Report outline . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

2 Designing usable Search user interfaces 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 History and recent development of Search user interfaces 2.3 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Designing a search interface . . . . . . . . . . . . . . . . 2.4.1 Understanding the search process . . . . . . . . . 2.4.2 Design guidelines . . . . . . . . . . . . . . . . . . 2.4.3 Interface features . . . . . . . . . . . . . . . . . . 2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

5 . 5 . 6 . 6 . 8 . 9 . 11 . 12 . 17

3 Method 3.1 Design process overview . . . . . . . . . . 3.2 Research . . . . . . . . . . . . . . . . . . . 3.3 Gathering Data . . . . . . . . . . . . . . . 3.3.1 Contextual inquiry . . . . . . . . . 3.3.2 Interviews . . . . . . . . . . . . . . 3.3.3 Brainstorming . . . . . . . . . . . 3.3.4 Focus groups . . . . . . . . . . . . 3.4 Communicating with prototypes . . . . . 3.4.1 Low-fidelity prototyping . . . . . . 3.4.2 Hi-fidelity prototyping . . . . . . . 3.5 Evaluation . . . . . . . . . . . . . . . . . . 3.5.1 Think-aloud protocol . . . . . . . . 3.5.2 Quick and dirty evaluation: Asking 3.5.3 Long-table approach . . . . . . . . 3.5.4 Questionnaires . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . users . . . . . . . .

. . . . . . .

. . . . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . . . .

. . . . . . .

1 1 2 2 2 3 3 3

. . . . . . .

iii

. . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

19 19 21 21 21 22 23 24 24 25 25 26 27 27 27 27

iv

CONTENTS

4 Accomplishments 4.1 Research phase . . . . . . . . . 4.1.1 Observing users . . . . . 4.1.2 User interviews . . . . . 4.1.3 Focus group sessions . . 4.2 Design phase . . . . . . . . . . 4.2.1 Low-fidelity prototypes 4.2.2 High-fidelity prototype . 4.3 Final implementation . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

29 29 30 30 30 31 31 35 37

5 Results 5.1 The eX-IFS GUI . . . . . . . . 5.1.1 Layout . . . . . . . . . . 5.1.2 Start page . . . . . . . . 5.1.3 Search results page . . . 5.1.4 Component details . . . 5.2 Search interface characteristics 5.2.1 Search results . . . . . . 5.2.2 Search history . . . . . 5.2.3 Search path . . . . . . . 5.2.4 Search term suggestions 5.2.5 Limitations . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

39 39 39 40 40 40 42 42 43 43 45 45

. . . .

47 47 47 48 48

6 Discussion and conclusion 6.1 Assignment . . . . . . . . . . . . 6.2 Research and design . . . . . . . 6.3 Proof-of-concept implementation 6.4 Future work . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

7 Acknowledgements

51

References

53

A Summary of user observations and interviews

57

B Focus group

59

C Focus group analysis

63

D Workflow prototypes (low-fidelity)

65

E Workflow prototype user tasks

69

F Workflow prototype final survey

71

G In-depth paper prototype (low-fidelity)

75

H High-fidelity prototype user tasks

77

I

79

Summarized prototype evaluation

CONTENTS

J GUI evaluation questionnaire

v

83

vi

CONTENTS

List of Figures 2.1

2.4

The usability framework. The figure shows factors that may influence the usability of a product. Adapted from [2] . . . . . . . . . . . . . . . . . . . . . 7 The figure shows the important stages of the information seeking task. Adapted from [25] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 The figure shows H¨ olscher and Strube’s model and process with probabilities calculated over both research groups. (a) Overview of the information seeking process (b) Close-up of direct interaction with a search engine. Adapted from [14] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A categorized result list (left) and a common result list (right) [6]. . . . . . . 15

3.1

The figure shows an overview of the iterative user-centered design process [34] 20

4.1 4.2 4.3 4.4 4.5

Vertical workflow layout . . . . . . . . . . . . . . . . . . . . Horizontal workflow layout . . . . . . . . . . . . . . . . . . Pop-up workflow layout . . . . . . . . . . . . . . . . . . . . Wireframe overview of the in-depth paper-prototype layout First version of High-fidelity prototype . . . . . . . . . . . .

5.1 5.2 5.3 5.4

POC application overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . Component details overview . . . . . . . . . . . . . . . . . . . . . . . . . . . Search results presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Non-empty result set. If the search query string returns an empty result set the POC application automatically reformulates the query until the result set is non-empty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Historical view over the 10 most recent searches . . . . . . . . . . . . . . . . Path Breadcrumbs (within a search for: mp4) . . . . . . . . . . . . . . . . . Search term suggestions. The list is continuously filtered according to the query string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2 2.3

5.5 5.6 5.7

vii

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

32 33 33 34 36

. 40 . 41 . 42

. 43 . 44 . 44 . 45

viii

LIST OF FIGURES

List of Tables I.1

The scores for each evaluated part for respective prototype version. The maximum score is 8,00 for each column . . . . . . . . . . . . . . . . . . . . . 80

ix

x

LIST OF TABLES

Chapter 1

Introduction This chapter gives an introduction to the master thesis project through a short background description, followed by a presentation of the task and its limitations.

1.1

Background

Searching is a central part of many computer systems today. Typically, systems need to have the ability to find and display different kinds of information. A major challenge is that a search via a web-search engine, such as Google, can result in a span of zero to millions result hits. However, many systems are used to find specific information through a company database instead of information over the Internet. Typically, this limits the amount of information to some degree but there are still challenges and difficulties that must be overcome. A user needs to know what kind of information one can search for, be able to formulate a proper query, interpret the result list to find the expected information as well as being able to reformulate the query if no proper result was found. There are ways to simplify the search process to better support the users in their searching task. Saab Security and Defense Solutions is a high-technology company that develops complex military products and systems. One of their commitments to their customers is to provide the products with support, such as spare-parts. Some of their products have a lifecycle of more than 30 years; hence, it is important for them to know exactly what components that a product consists of as well as which products an individual component is a part of. The component hierarchy tree looks like this: – Project. This is the root of the component tree. A project can consist of systems, products, and components. – System. This is a collection of a large set of compound parts. The parts can consist of other systems, products, and components. – Product. This is a collection of a reasonable amount of compound parts. The parts in a product may be other products and individual components. – Component. Typically, these are the leaf nodes in the component tree. However, some components may consist of other components as well. 1

2

Chapter 1. Introduction

The information need is the same for every node type, that is, every node is handled like an individual component no matter if it is a project, system, product or component. Henceforth the term component will be used for these node types, however, to avoid confusion some of the above node labels may be used as a clarification when necessary. The organization has an existing computer-based desktop system for this. However, the lack of usability makes the system inefficient and results in user frustration.

1.2

The eX-IFS application

The initiator to this thesis project is the Through Life Support (TLS) department at Saab Security and Defense Solutions in Sweden. This is a company within the Saab Group and the TLS department’s main responsibility is after-sales errands on naval military systems. They have a need to develop a new web-based system for managing their customers’ naval products and systems. A system may consist of 50.000+ components at different hierarchical levels within the system; the department must be able to get information regarding every significant part in the system in case of a failing component. Their existing system does not match the usability needs of the users, hence the initiative for this project. The users’ everyday work with the existing and desired system is mainly based on searching and finding information regarding the components.

1.3

Target group

The target group of the eX-IFS application is primarily personnel working at the Through Life Support (TLS) department at Saab Security and Defense Solutions, Sweden. The users share a common set of characteristics, these are: – They use this kind of application frequently in their everyday work. – Their knowledge of the problem domain is very large; hence they are regarded as expert users. – Searching is a central part of their work assignments. – Typically, they know what to search for, that is, they use known-item search. The target group wants a more usable application for support in their everyday work, in comparison to their existing system.

1.4

Assignment

The primary focus in this thesis project is to design and develop a Graphical User Interface (GUI) for the eX-IFS application prototype. The GUI should provide the ability to easily perform searches over a collection of delivered systems in order to get information regarding the current status of a system and its embedded parts. The entire project is divided into two separate master thesis projects; one regarding database design and searching and one regarding the development of a usable graphical user interface (GUI) of a proof-of-concept (POC) application. The latter is the basis for this master thesis project.

1.5. Method and Results

3

The vision is to develop a functional application with real data that can be used in the TLS departments’ everyday work. The goal is to provide a partly functional prototype for the application to serve as a POC for future development and implementation. The selected platform for the prototype was an Adobe Flex front-end with a .NET WebService back-end; the latter is out of scope of this thesis project. That is, the author of this thesis report was responsible for the usability and design of the front-end as well as the POC implementation of the GUI.

1.5

Method and Results

In order to accomplish the specified goals of this thesis project, three primary areas where identified and explored. These are: – An in-depth research and exploration study regarding the research area of how to design usable Search User Interfaces. – An iterative and user-centered approach to develop the design. – A proof-of-concept implementation based on the research. The in-depth study of Search user interfaces (SUI) was conducted with the purpose to get a deeper understanding of the problem domain. The result of this research study is a chapter regarding certain important aspects and characteristics of SUI’s. In order to develop a design proposal that is accepted by the target-group, a user-centered approach was adopted. Real users from the target-group were available throughout the entire thesis work. This resulted in a design that was developed through continuous collaboration with the target-group. The implementation of the proof-of-concept (POC) application design was based on the results of the previous iterations of the design process. The application should be a webbased solution; hence a Rich Internet Application (RIA) prototype was developed. The chosen platform was Adobe Flex, a common RIA-platform. The main goal was to develop a POC interface that was intuitive and supported the users’ everyday tasks.

1.6

Limitations

The assigned task has a limited timeframe of 20 weeks for one person. This implies that several important areas and aspects regarding design, evaluation and implementation of the eX-IFS POC have been omitted during the thesis project. It is important to mention that the implemented POC application is not a fully functional application. This should be regarded as a concept proposal for further development. Due to the time constraints, no account was taken of graphical design, underlying algorithms or system architecture.

1.7

Report outline

– Chapter 2. Designing usable Search user Interfaces. This section is an in-depth literature study of Search user interface usability. The chapter touches different aspects of search user interfaces such as how users perform searches as well as what kind of functionality to incorporate to support the search process.

4

Chapter 1. Introduction

– Chapter 3. Method. This chapter introduces the scientific techniques and methods that were used throughout this thesis project. This includes a brief description as well as pros and cons of the methods. – Chapter 4. Accomplishments. This chapter presents a chronological overview of the different phases performed during the development of the eX-IFS GUI prototype. Each section and subsection touches important areas and aspects of the development. – Chapter 5. Results. This chapter gives a more detailed description of the results for each section. The individual results formed a basis for the final conceptual GUI prototype. – Chapter 6. Discussion and conclusion. This chapter focuses on personal reflection and discussion by the author regarding the thesis work. Some suggestions of future developments will conclude the chapter.

Chapter 2

Designing usable Search user interfaces This chapter is a literature study and aims to explain the concept of Search User Interfaces (SUI) with the focus on how to successfully design the user-interface for such applications.

2.1

Introduction

The fast paced development of the Internet has had a great impact on today’s society. In the beginning the Internet was primarily used by computer enthusiasts but now ordinary people use the Internet as an integral part of their everyday life, at home, at work or at school [40]. People tend to turn to the Internet in order to find information and the use of search-engines is now the second most frequent activity on the Internet [12], only beaten by e-mail. A major problem today is that information management makes the everyday life more complicated, this is due to the vast amount of information we encounter and process each day. The information management problem entails a high burden on our physical and cognitive resources. There are more choices to make, more information to search through and information is deliberately intended to influence our behavior. This complicates our information retrieval process. Electronic systems for information seeking must improve and have a high usability in order to better support users during information seeking tasks [25]. A research survey performed in December 2009 by PEW [5] showed that 74% of the American adults use the Internet. The survey also revealed that 88% of these users have the main goal of finding information through a search engine. One indication of the importance of a successful and usable Search Interface is shown in a study by Forrester Research, where 76% of firms considered search as “extremely important” while only 24% of them considered their own web site’s search functionality to be “extremely useful” [13]. According to [9] the total amount of searches performed worldwide on the Internet was over 131 billion during December 2009. This indicates an increase of 46% on the amount of searches compared to the corresponding period in 2008. During this period there was approximately 0.2 billion new Internet users registered, resulting in a total of about 2 billion Internet users worldwide [16]. How can one design a user interface that can handle the amount of information we use and produce today, and still maintain a high usability level? The aim of this literature study is to present different aspects of design and development 5

6

Chapter 2. Designing usable Search user interfaces

of search user interfaces. The study will not cover all details or aspects; instead it should be seen as an overview and a starting point for further research for those interested. The study is organized as follows: First a section regarding the history and recent developments in the concept of Search-user interfaces is explained, followed by a section about general usability. Then a section regarding the search process, design guidelines and important characteristics of SUI’s is presented. Finally, the chapter concludes with a section that contains a discussion of SUI characteristics.

2.2

History and recent development of Search user interfaces

The usage of Search User Interfaces has drastically changed since the advent of the Internet and World Wide Web. Previously searching was mainly performed by trained professionals such as librarians, researchers and journalists. The wide spread use of the Internet led to the development of search engines such as AltaVista, Yahoo and Google [12]. The development of search interfaces is not limited to only the target group. The userinterfaces, the searchable information as well as the nature of the queries have all changed over the last years. Early systems were often closed, that is, non-public and contentmonopolized. Unfortunately this discouraged the development of usable interfaces since there was a lack of competition amongst content-providers. Usually the interfaces were command-line based and the searcher had to learn and remember complex search-operators in order to perform successful searches [12]. It was uncommon to be able to search over full text articles; the searchable information usually consisted of abstracts, metadata and values such as titles. The main goal of these kinds of searches was often to find the name and location of the physical full text version [12]. However, since the general public got access to the amount of information available on the Internet, the importance of functional search interfaces has become crucial. The interfaces today are quite simple and new features for supporting the user in the search process are constantly being developed. Full-text searches are now common since the web often provides full text documents. Modern search user interfaces commonly have very advanced algorithms underneath the graphical surface. One powerful advantage of modern systems is that users have the ability to use both keyword based searches and in some cases even natural language in their queries [12].

2.3

Usability

Usability is about understanding how to design interactive systems to best support the users. However, it is not as simple as that. One must understand what the users want and need as well as how to best match these requirements in order to be able to design a UI with high usability. Beside this knowledge a usability expert or interaction designer must also have knowledge of the users’ work situation and the environment in which the system will operate [34]. There are many aspects of usability and the concept is somewhat subjective. Hearst [12] and Nielsen [30] refers to usability as how easy a user-interface is to use [12] while Faulkner [11] has a slightly different description and refers to usability as how satisfying the interface is to the users and how comfortable they are using it. The International Organization for Standardization (ISO) defines Usability as follows in ISO DIS 9241-11 [2]:

2.3. Usability

7

Figure 2.1: The usability framework. The figure shows factors that may influence the usability of a product. Adapted from [2]

“The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” In their definition, ‘effectiveness’ means whether or not the user successfully can carry out the intended tasks. The time of accomplishment is not regarded in this definition in contrast to an early definition by Shackles who included both speed and performance in ‘effectiveness’. That is, a system was considered effective only if the users were able to perform the intended tasks within specified time limits [11]. However, in ISO’s definition one dimension of time is included in ‘efficiency’. If one should compare two systems and one of them is faster than the other in performing the task, then the faster system is more efficient. Shackle make no distinction between ‘effectiveness’ and ‘efficiency’, instead he defines usability in terms of ‘effectiveness’, ‘learnability’, ‘flexibility’ and ‘attitude’ [11]. Bevan [2] points out that a strength of ISO’s definition is the flexibility in that it also accounts for the context of use. The context of use consists of the users, the tasks, and the equipment in use as well as the physical and organizational environment. Figure 2.1 shows that all these factors may influence the usability of a product. More recently Nielsen [30] identified five core components that defines usability, these are: – Learnability. How fast a user can understand the GUI and accomplish the intended tasks the first time a user encounters the design. – Efficiency. The improvement in time to accomplish tasks when a user has learned the GUI.

8

Chapter 2. Designing usable Search user interfaces

– Memorability. How fast a user can reestablish proficiency when the user returns to the design after some time of not using it. – Errors. The amount and severity of the errors a user encounters as well as how fast the user can recover from them. – Satisfaction. How pleasant it is to use the design. Nielsen disregards ISO’s broad description of context of use. Instead he focuses more on the system and the interface itself. Although he accounts for some context of use in the sense that he argues that in order to improve usability, one should evaluate new designs with a number of actual users. Preece et al. [34] complements Nielsen’s list with utility, meaning how well the system provides the right functionality according to what the users need or want to do.

2.4

Designing a search interface

It is hard to define the characteristics of a good UI for an information seeking system. It is highly dependent of the kind of answers users are looking for. A SUI must in some sense try to adapt parts of the UI, such as the presentation of search results, based on the question at hand. That is, different types of questions might benefit by formatting and displaying the search results with an appropriate representation. An example of this is a question regarding the current weather; this kind of question could benefit from a graphical representation instead of a text-based representation. Typically, web based search engines must often be able to handle a wide range of query types. They must support both known-item search and searches that aim to answer diffuse and complex problems [13]. Some example queries could be: – How is the weather in Stockholm right now? – What are some good examples of graphical design? – What are some promising untried treatments of breast cancer? Lee at al. [23] highlights the complexity of the definition for known-item search. An issue that complicates the concept of known-item search is that one could discuss the amount of knowledge the user must have regarding the known-item. Typically, researchers tend to articulate their own definition of the concept as well, to suit their particular research context. The definition used in this thesis is formulated by Kim and Allen [18]: “...to find a piece of information that is known to exist and to give a specific answer to the question...” However, it is not enough to just support these different types of questions, the system must also be able to produce and present a result set that is relevant to the users’ query and information needs [13]. Moreover, users with varying levels of computer skills will most certainly use the SUI differently; hence the system UI must support both expert-users as well as novices [14]. A SUI should be a tool that supports the users during the entire search process. It should be able to help users to concretize the information need, formulating the queries, help users understand the search results as well as showing the status and progress of the information seeking effort [12].

2.4. Designing a search interface

9

When discussing information retrieval systems, two terms are frequently used to characterize the quality of the search results, these are: precision and recall. The definition of these terms is [37]: – Precision. The ratio of relevant documents that “should” have been retrieved in relation to the total number of retrieved documents. – Recall. The ratio of relevant documents retrieved in relation to all relevant documents in the database. Typically, using phrased queries increases the precision at the expense of recall. That is, a search for “air pollution” is likely to generate a result set with less irrelevant documents than a search for “air” and “pollution”. However, the phrased query might overlook important results such as documents containing information regarding air quality or other topics related to air pollution [37]. One difficult decision a SUI designer has to make is how much automation the interface should provide. An interface might have multiple levels of automation at the same time depending on the complexity of the interface [1]. Bates [1] highlights that much research is focused on how to develop “optimal” information retrieval systems with completely automated search functionality. However, she argues that not all users would appreciate this kind of system since some users want to do and direct their own search. Users tend to want the speed and power of an automated system, but they want to feel in control of the system. This is supported by a study by Koenemann and Belkin [19] on how users experienced an information retrieval system with different levels of relevance feedback on novice searchers. The results showed that the users achieved better results when they had more control over the queries than the automated system. Bates [1] stresses that a designer should thoroughly think about the question of what capabilities the user should be able to exercise as well as what the system should be designed to do.

2.4.1

Understanding the search process

Marchionini [25] describes the information seeking process as a kind of problem solving activity. The basis for this claim is that either or both of the information sought (problem) or the search process (solution path) may be complex. The author presents a model consisting of five basic components. Even though the information seeking task is an iterative process the model is visualized in a non-linear fashion since there are several possible paths between the definition of a problem and the solution. A rough overview is illustrated in figure 2.2 and the five main components from [25] are described briefly below. – Define problem. This is the initial step in the information seeking task. The definition of a problem can vary much in complexity, a known-item search is often much simpler than for example trying to find information that supports an argument. Moreover the problem definition evolves during the whole problem solving activity as the problem may branch into different sub-problems. – Select Source. Regardless if the problem definition is simple or complex, a user must always select a source to begin the search process. It is shown that end users primarily tend to start with sources that have proved to be successful for them in the past. – Articulate problem. In order to perform a successful search the defined problem must be articulated. This often means that an end user must formulate a query or determine a starting point for the search in the system.

10

Chapter 2. Designing usable Search user interfaces

Figure 2.2: The figure shows the important stages of the information seeking task. Adapted from [25]

– Examine results. A system typically responds with a set of hits as a result to a user query. This list might be represented either graphically or in textual form as for example a ranked list. Based on the result set the user must make a decision regarding which function to call next. This is typically to view more detailed information regarding one (or more) of the result hits. It is crucial to the examination of results that full text information or images can be displayed since primary information is of high importance for the users.

– Extract information. After the examination of results a user must extract and manage relevant information in order to be able to verify that it can be applied to the defined problem. Some examples of managed tasks are copy, cut and paste as well as save or print the information.

The information seeking task presented above is described from a user perspective as it includes cognitive processes such as problem definition. However, this can be matched to other studies with a more systematic perspective. H¨ olscher and Stube [14] performed a set of research studies aiming at finding a common model for information retrieval via search engines. The study included two research groups, one with computer experts and one with novice computer users. Their findings showed that their model (illustrated in figure 2.3) could be applied to both user research groups. According to their study there was a probability of 81% that a user in any of the research groups would use a search engine to try to find the answer to a question. It also indicates that search engine interaction is a somewhat iterative procedure during information seeking (figure 2.3a); this is supported by a 42% probability of query reformulation when a user examines the page results. The study also highlights some important differences in search engine usage between the expert and novice users, this is however out of the scope of this thesis and readers are directed to [14] for further details. Nielsen [28] advises not to provide users with advanced search functionality since users often use it incorrectly. However, H¨olscher and Strube’s study shows that expert users tend to use advanced search techniques continually [14].

2.4. Designing a search interface

11

Figure 2.3: The figure shows H¨ olscher and Strube’s model and process with probabilities calculated over both research groups. (a) Overview of the information seeking process (b) Close-up of direct interaction with a search engine. Adapted from [14]

2.4.2

Design guidelines

In 1997, Schniderman et al. [37] found a problem with SUI’s. The UI’s was unnecessarily complex and there was no consistency in how different SUI’s were designed. This resulted in a paper with a presentation of eight guidelines that targets SUI usability. These guidelines along with a brief explanation are presented below: – Strive for consistency. Designers should pay careful attention to details such as layout, fonts, terms used in the UI and colors [37]. Hearst [12] stresses that SUI’s often show rich information of complex nature and that changing small details in the design can result in severe effects on the user-experience. – Provide shortcuts for skilled users. The design of a SUI should allow skilled users to work efficiently by implementing shortcuts to relevant functionality [37]. Some examples of such shortcuts could be to support immediate answers to directed and focused questions or to provide deeper level site links in the result-list (see section 2.4.3 for more details) [12]. – Offer informative feedback. A user should be informed of all aspects regarding the current search, such as displaying what is being searched for and showing the sources [37]. Hearst [12] presents a number of suggestions of valuable features that can be used to increase the user-experience by providing efficient and informative feedback. This includes showing search results immediately, highlight query terms, show query terms suggestions, allow sorting and support rapid response. – Design for closure. It should be possible for a user to see and understand things such as when all results in the list have been viewed or whether or not the search was made over the entire database [37]. Findings in [12] show how important the graphical design of the UI is. Good designs that appeal to the users seem to raise the overall acceptance of a site as this correlates to their perceptions of the UI quality and the user satisfaction. Users also tend to persist longer in search process in an appealing

12

Chapter 2. Designing usable Search user interfaces

interface [12]. However, a good graphical design is not enough the interaction design is also very important [34] and the UI should support the users’ actions and give valuable feedback [12]. – Offer simple error handling. A user should be able to easily update their search terms and parameters. If an error occurs it is important to present a concise and easy to understand error message with as little technical details as possible [37]. A few suggestions in order to improve the error handling are to not display empty result sets and to address the vocabulary problem. A more detailed description of these can be found in [12]. – Permit easy reversal of actions. A user should be able to go back to a previous action or state; this means that every action should be reversible. An easy way to accomplish this is to offer some kind of history mechanism in the UI that allows a user to re-issue a previously issued query [37]. – Support user control. A good design UI allows the user to always feel in control. Constraints in the UI like forcing a user to enter search parameters in a specific order should be avoided. Instead the user should be given the control over the UI and it is the user that should initiate actions [37]. However, Hearst [12] argues that there must be a balance between user control and system automated action. She mentions two important functions that should be considered, even though they might imply a tradeoff between opaque system control and transparent user control, when designing a SUI. These are rank ordering and query transformation. – Reduce short-term memory load. The human short-term memory (STM) is activation based, that is, information of this kind is easily lost [10]. In order to avoid unnecessary load of the STM a SUI should support functionality such as recent search history and compact presentation [37]. Two other examples of functionality that might help reduce the STM load is to suggest search action in an entry form and to integrate navigation and search. This can be accomplished by using for example a watermarked textbox and faceted navigation [12]. Although several researchers have proposed guidelines for search interfaces, Hearst [12] argues that even if guidelines might be helpful in interface design, guidelines alone are not enough. The main argument for this claim is that guidelines can be hard to follow. Guidelines often overlap and conflict with each other and they seldom provide concrete solutions on how to achieve the guidelines’ goals.

2.4.3

Interface features

This section is not supposed to cover all aspects of Search-user Interface features; instead it aims to highlight some important characteristics that might improve the usability of a SUI. History management Research has shown that people tend to have a need to revisit previously viewed information. A study by Cockburn and McKenzie [8] shows that approximately 81% of the URL’s that a user visit is in fact revisits. This behavior is also common in SUI’s where users tend to re-send search queries [12]. History is a common feature in web-browsers, SUI’s can benefit from this functionality as well by for example providing a list of the current user’s recent search-terms [12].

2.4. Designing a search interface

13

Komlodi [20] claims that improved history functionality in UI’s can be beneficial for users, since it would enable users to: – Get an overview of the search process – Better ability to manage tasks – Compare and relate result sets – Collect documents and information A number of researchers [21] present suggestions of important features in search history representation. One suggestion is that the history feature should be able to handle parallel tasks as users tend to switch task continuously. Results by Cockburn and Jones [7] show that the common functionality of history navigational aids in web browsers, that is the back and forward button, mismatch the mental model of the users regarding the session history. A browser’s history is typically stack-based, that is, it does not reflect the user’s actual history. If a user hits the back button a number of times and then follows a link from the current page, the browser overwrites the previous forward history with the new target page. Furthermore the screen real estate is important; a basic idea is that the history functionality should support the users’ task. Although the history is likely to grow over the session, its screen real estate should not be dominant. It should be constrained properly in relation to other parts of the UI [21]. Providing Suggestions Hearst [12] discusses several possible solutions for providing users with suggestions to enhance the SUI experience. Two variants, with different purposes in the search process, are briefly presented below: – Query specification in entry forms. There are several ways to provide suggestions to the users via an entry form. One common approach is to use watermarked text blocks, that is, the text block shows a short description of the type of query or query terms a user is expected to use. When the text block gets focus the watermark disappears and the user can type instead [12]. Another approach is to give users alternatives of possible and popular query refinements based on the typed query. The refinement suggestions may be presented as a list somewhere in the UI. This kind of feature has proved to be useful and appreciated by users [41]. – Query specification with dynamic term suggestions. Dynamic term suggestions mean that users get some kind of real-time alternatives during their query specification. White and Manchionini developed an Interface where the system generates a list of suggestions of possible relevant alternatives for the next query term. The suggestions were based on the users previous query terms and the list updated and displayed when a user pressed the space-key. The results showed that some users thought it was a huge time saver [42]. Another possible way of displaying suggestions is to use auto-complete functionality, like the SUI Live Search from Microsoft. Auto-complete is a way to show possible query term suggestions on a letter-by-letter basis. According to Hearst it is likely that this kind of functionality will become increasingly useful as the development of modern and more responsive term-suggestion features progress [12].

14

Chapter 2. Designing usable Search user interfaces

Result presentation The presentation of search results is very important in a SUI. A few aspects of result presentation are discussed below. – Results per page and search result ordering. Typically, search engines display about ten hits on the search results per page [12]. In a usability study from Google [26] they found that speed was of great importance to the user. They experimented with displaying a list of thirty search hits instead of ten. Analysis showed that there was an average of 0.5 second delay when displaying thirty hits. This rather small delay had a huge impact on the user experience as Google detected a 25% reduction in traffic on the site over a six-week period. Studies have shown that the ordering of the search results is crucial. A study by Joachims et al. [17] shows that users tends to primarily explore the first and second search result; beyond that there is a significant drop in interest for the user. Nielsen [28] found that people almost never look at the second page of search results. A consequence of a user not finding relevant information on the first result page could result in the user giving up or having to reformulate the query [12]. According to [28] the latter has proved to be difficult since a query reformulation often leads to even worse search results. – Search result presentation. Some of the challenges that search engines faces is that words used as search terms can have several diverse meanings [39] and that documents and web pages often discuss several different topics interchangeably [12]. Paek et al. [32] studied what kind of impact different summary lengths of result hits have on user performance. Three interface variations were tested, these were: • Normal view. A simple listing of search results, where a mouse click opens up the full text document. • Instant view. A mouse click expands the summary information with additional sentences, containing query terms, from the full-text document. • Dynamic view. As long as the mouse hovers over a particular result hit the summary expands a few words at a time to display more information. Their study showed that users performed faster and produced more accurate results using the instant view than with the other two. A common strategy for presenting search results is a common ranked list [39], that is, a list of ranked results. The ranking is based on relevance of the search hit calculated by some underlying algorithm. However, several studies [39] [6] have shown that one can improve the user experience by either using clustering or categorization techniques. Categorization is a technique for ordering search results in a logical and systematic manner. The basic idea is that one uses pre-defined categories to organize the information into different groups or topics [6]. A drawback with categorization is that it often requires a high amount of manual work with category assignment in order to successfully implement categorization [12]. However, Chen and Dumais [6] present a more automate solution using an on-the-fly web directory as source for the categorization. Figure 2.4 shows an example of a UI using a categorized search result list. A problem with assigning documents to a single category is that they often discuss many different topics that may fall under different categories. One way to solve this

2.4. Designing a search interface

15

Figure 2.4: A categorized result list (left) and a common result list (right) [6].

problem is to use faceted navigation, that is, one can use metadata such as tags in order to categorize documents into different topics. This would allow a user to find the same document under several categories relevant to its content [12]. Clustering is a technique that orders results based on internal similarities, such as words or phrases [12], that is, clustering is a way to organize information based on keywords in content itself [39]. An advantage with clustering is that it is highly automatable and requires little manual work. However, the advantage also has a downside since clustered search results do not have as high accuracy as categorized results [12]. Wang and Zhai [39] present a solution where they use query logs from past searches in order to cluster the information. Their findings show that their technique performs more accurately on average than ordinary clustering techniques. Both categorization and clustering outperforms an ordinary ranked list, both in speed and user experience [12]. Other interface features Hearst [12] presents several features that may increase the user experience of a SUI. A few of these are shortly presented below: – Query-term highlighting. A feature that has proved to be very useful for increasing the user experience in search-result presentation is to highlight the query-terms in the result hits. The highlighting can be done in several ways, such as reverse video, bold text or with a colored background. It can also be implemented so it is shown both in the short summary below the search hit as well as in the complete document that is displayed when a user navigates to a specific search hit. – Sitelinks. A feature that has increased in popularity is to present links to popular subpages of a search-hit. The subpage-links are often indented directly below the main

16

Chapter 2. Designing usable Search user interfaces

search-hit link. This kind of implementation might speed up the users’ navigation to the subpage presenting the desired information. – Shortcuts. Some search-engines try not only to find results where the answer to the query might be, they also try to answer directed and focused questions such as “what’s the weather in X” directly by displaying for example weather information in the resultlist. – Blended results and media types. Modern search engines find different kind of media content related to the search query. It is common to display diverse content types, such as images, in the search-result list. – Previews of document content. Recent development in some search engine’s searchresult lists include the ability for the user to preview a thumbnail image of the result target, without having to leave the search-result page. Navigation Navigation is an important part of a SUI. Nielsen [27] claims that even though interfaces, in his example a website, implement search functionality the UI still needs navigation and a strong sense of structure in order to be usable. During his research he found some important characteristics in order to enhance the user-experience and the use of search; some of these are: – Search should be a box. Usability studies [28] have shown that users often look for “the little box where they can type”. When Nielsen updated the search functionality on his website by exchanging a link with a search-box the search engine use increased by 91%. – Search should be easily available on every page. If a user feels lost on a webpage they tend to use the search functionality in order to get on the right track again. Since it is impossible to predict where and when a user will be lost the search functionality should be visible and available on every page in the structure so they don’t have to search for the search [27]. A common navigational aid in UI’s is breadcrumbs. These are often thought of as a secondary navigational feature, although its popularity increases. Critics argue that breadcrumbs have the disadvantage of taking up screen real-estate; however, they bring several advantages to a UI. Some advantages are that they show the current location, they allow users to navigate to higher navigational levels with one click and they take up very small amount of screen real-estate compared to many other navigational aids [31]. A study by Rogers and Chapparo [35] indicates that users tend to get a better understanding of a website’s structure if the site in question implements breadcrumbs. Instone [15] identifies three different types of breadcrumbs: – Location. This type of breadcrumb reflects a page’s relative position in the overall navigational structure, that is, where the user is. – Path. This type of breadcrumb reflects the users’ actual path to the current location. This can be useful in dynamically generated pages where a single page can be reached by different routes.

2.5. Discussion

17

– Attribute. These types of breadcrumb do not reflect either the path or location of a page. However, this is commonly used by e-commerce sites for describing characteristics of a product by using meta-data. According to Nielsen [31] the proper way to implement breadcrumbs is by implementing the Location-based type. The argument for this is that breadcrumbs should reflect the site’s navigational structure instead of the users’ history.

2.5

Discussion

There is no doubt that search user interface design and development is a very complex task. Many factors of highly different nature must be taken into account in order to succeed. The task gets even more complex by the fact that there are large uncertainties, like key factors such as the user’s computer skill-level and the different types of queries needed. This implicates that several tough design decisions often must be made based on assumptions. Section 2.4.2 presented a set of guidelines in SUI development; these guidelines will not be discussed further. Readers interested in details regarding those guidelines are referred to the sources; these can be found in the reference section. This section aims at highlighting some important aspects of SUI’s based upon the research literature presented in this chapter. All aspects will not be discussed here since this is a rather large area with many different approaches. This section focuses on important characteristics of SUI’s and should serve as a guide for those looking beyond the general guidelines presented earlier. Design and optimize for efficiency. A responsive interface is very important for a good user experience, SUI’s is no exception as speed seems to be an important factor for users during their information seeking task. A designer should also be aware that small UI changes can result in severe consequences for the business. As previously mentioned, Google [26] became aware of this during one of their user interface experiments, they lost a significant amount of traffic over a short period of time because of a small change in the UI. Another angle of efficiency is that the design of the SUI should support and guide users to find the answer to the query at hand. A good design should therefore try to use visualization techniques such as categorization or clustering since they often increase the efficiency for the users. A carefully designed SUI should also have an appealing graphical design; this is not only a way to increase the acceptance among users [12], one can also use graphics to guide the users during the search process. Get to know your user task and their domain of interest. Far from all SUI’s have to be as flexible as web search engines such as Google or Yahoo. The characteristics of the users’ information need within the domain could highly influence the SUI design. Knownitem search queries are enough in many websites and applications and the UI should be adapted to such prerequisites. The presentation of search results for directed queries could (and should) probably be designed much better if a designer adapted the design to the information at hand, the user needs and the domain instead of blindly follow guidelines. Many company SUI’s seem to lack this understanding since there is a large gap between the perceptions of the importance of searching compared to their own website’s usability [13]. Always think about the users, not the cool-factor. The basic idea with a SUI is to support the users during a very complex task. Even though there are many features that are cool and innovative from an interactive perspective, designers should always be aware of the users and

18

Chapter 2. Designing usable Search user interfaces

think twice regarding the functionality. Features that distract the users and dissipate their attention should be disregarded while features such as query term highlighting or instant view that clearly benefit the users should be considered in the design process.

Chapter 3

Method The project was performed using an iterative user-centered approach and this chapter provides a detailed description regarding the methods and technique used during the thesis. First is a section presenting an overview of the user-centered design processes, followed by a few sections that discuss the methods and techniques used in more detail. These are presented in chronological order with respect to the design process for this thesis, starting with techniques for gathering information, followed by sections for prototyping and evaluation. The chapter concludes with a section regarding details of the implementation.

3.1

Design process overview

Interaction design is about investigating the usage of a product and the problem domain; this is typically done by utilizing a user-centered approach on development. A user-centered approach is goal-driven in the sense that the users’ needs form the basis for the development process [34]. However, interaction design is also about trade-offs. That is, to balance the conflicting requirements and constraints [34]. This is supported by L¨owgren and Stolterman [24] who argue that every design process is unique and somewhat unpredictable. The authors’ main argument is that there are three variables that differ from project to project, these are: those responsible for the job, the conditions and constraints regarding time and resources, and the current situation. Furthermore, they stress that a designer’s primary assignment is to develop a “goodenough” design given the constraints for the project. Designers continually switch between overview and detailed features; they often have an understanding of what should be done and have a set of creative solutions for the problem at hand. However, they often face a situation that is both chaotic and concrete. This implies that thinking on various levels simultaneously is necessary and a designer cannot rely entirely on a rational model or framework [24]. A general lifecycle model for the interaction design process is presented by Preece et al. [34]. This model identifies four basic activities in user-centered design (see figure 3.1), these are: – Identifying needs and establishing requirements. In order to design a system that supports the target users, one must know who the users are and how an interactive product could support them best. The users’ needs form a basis for the product’s requirements and the following design and development activities. 19

20

Chapter 3. Method

Figure 3.1: The figure shows an overview of the iterative user-centered design process [34]

– Developing alternative designs. During this stage several alternative suggestions and ideas that meet the target-users’ requirements are presented. This is the core activity of designing and it can be divided into two separate sub-activities; conceptual design and physical design. A conceptual design includes a conceptual model that focuses on what the product should do, what it should look like as well as how it should behave. The physical design goes more into details of the product and describes design attributes such as colors, sounds, menu design and what images to use. – Building interactive versions of the design. A natural part of user-centered design is to let the users evaluate a set of design alternatives. The development of interactive prototypes facilitates the collection of feedback since many UI problems can be found by watching users interact with the prototypes. There are different levels of interactive prototypes; paper-based prototypes are fast and easy to develop while software-based prototypes are more complex and time-consuming to develop. However, each type of prototype serves its own purpose in the design process. – Evaluating designs. The process of interaction design requires a high level of user involvement throughout the entire process. In order to ensure that the users’ needs and requirements are met, the product must be evaluated in various stages during the development process. The evaluation focus on the users and their tasks, one typically studies criteria like the amount of errors, how appealing it is and how well it matches the specified user requirements. Preece et al. [34] present three important key characteristics of the interaction design process. These are: focus on users, specific usability criteria, and iteration. The user focus is crucial in user-centered design since their needs form the basis for all stages throughout the entire design and development phases. A set of specific usability criteria should be identified at the beginning of the project. These criteria guide the designer in the process of choosing between ideas, solutions, and different design alternatives. It is important to incorporate an iterative process. This is the key to the possibility of evaluations on different levels as well as refinements to the design based on user feedback. A greater understanding of the users and their environment is

3.2. Research

21

gained throughout the entire user-centered process; this is a powerful tool for developing a system that has high acceptability among the target-users [34].

3.2

Research

A usability expert or interaction designer (henceforth referred to as a designer) often find themselves in a situation where they need to design a product or a system that is to be used in an area of which they have no prior knowledge. However, a prerequisite for a successful design is that the product or system is tailored to fit the intended usage area. Hence, designers need to research the problem domain. This is usually done by collecting information regarding the intended users and the operational environment, a necessity for the development of a system that efficiently helps the users achieve their goals and fulfill their needs [11]. According to Preece et al. [34] it is crucial to understand the problem domain or else usability goals may be overlooked. This can be avoided by clarifying the usability and the user experience goals. L¨ owgren and Stolterman [24] argue that it is not unusual that a designer has to answer questions regarding what can be done and what should be done. In order to answer that question the designer must rely on the knowledge gained through the research phase. The authors present two approaches of research: – Exploration. The search for several different alternatives and ideas in order to solve the problem. – Investigation. The search for one solution by focusing on the current situation. Typically, both of these methods are used somewhere in the design process since they serve two different purposes. Exploration allows a designer to examine several different aspects and solutions to the problem. However, almost every design process ends with investigation, since the ultimate goal is to develop a product, system or specification that focuses on a specific problem [24]. The design process described in Section 3.1 is a basis for the research phase in this thesis project. Section 3.3 present methods for identifying users’ needs and establishing requirements. Techniques used to develop alternative and interactive versions of the design are presented in Section 3.4. During the research for this thesis project, the investigation part consists of observations, user interviews and focus groups sessions. The exploration part of the thesis is a literature study regarding search user interfaces (Chapter 2 - Designing usable Search user interfaces).

3.3

Gathering Data

The foundation of user-centered design is to know the users, their environment and what they are trying to do. Usability is not about providing the easiest solution; it is about providing the most appropriate solution for the users and their task at hand [11]. The following sections present a set of methods that can be used to collect valuable information for a forthcoming design.

3.3.1

Contextual inquiry

Contextual inquiry is an ethnographic approach used for design. The method is based on collaboration between a user and a designer [34]. Beyer and Holtzenblatt [3] introduced

22

Chapter 3. Method

contextual inquiry with the goal to strengthen the relationship between the user and the designer. Their model builds on an apprenticeship relation model, that is, the user serves as the master and the designer serves as the apprentice. In order to collect as valuable information as possible the apprentice observes the master while performing their natural working activities. However, the observation is not passive, instead the master explains what they are doing and why, in parallel with performing the activity. The apprentice can at any time interrupt the activity by prompting questions regarding some aspect of the task. There are many benefits with contextual inquiry [3]; some of these are presented below: – Doesn’t require teaching ability. Many users lack a teaching background. However, they are often experts on their natural working activities. The master-apprentice relation allows a designer to get a better understanding of the natural activities in the problem domain. – Reveals what matters. Often users are not aware of everything they do. How they perform an activity can be based on years of training and some sub-tasks are obvious to them due to their experience. Since the designer is allowed to put questions, the user can be forced to reflect over why they do these automated tasks. This may lead to more efficient workflows in the new design. – Reveals details. Important characteristics of the users’ activities can be revealed by contextual inquiry. This is due to the fact that the users perform and explain the task simultaneously. This prevents the user from generalizing the task to other similar tasks. – Reveals structure. Since users communicate basic underlying strategies and structures for the activities by performing them, a designer can get a better understanding for the different aspects of the activities by observations. If a designer has the opportunity to observe multiple users and instances of the task, the designer can easier form an idea of how one could perform the activities in a different and more efficient way. – Improves learning. Every event that happens while the designer is present is a starting point for discussions of past events. A designer often has a limited time to get an overview of the problem at hand; by using contextual inquiry a designer can take advantage of the user’s previous experience. The typical form of contextual inquiry is by doing a contextual interview. This is a combination of observation, discussion and reconstruction of past events [34]. Snyder [38] highlights the importance of watching the users in their natural work environment. She argues that contextual interviews are a very helpful method to get valuable information for the design task at hand.

3.3.2

Interviews

Preece et al. [34] describes interviews as a ”conversation with a purpose”. The arrangement of interviews can differ significantly. Which kind of approach and structure an interview session should have is highly dependent on what kind of questions to be answered, the adopted paradigm, and the session’s evaluation goals. There are four main types of interviews [34], these are: – Unstructured. An unstructured interview can generate a lot of useful data since the arrangement is more like a conversation on a specific topic. That is, both parties have

3.3. Gathering Data

23

the opportunity to control the interview session. There is no explicit manuscript for the topic at hand, instead the questions are open-ended and the interviewee is free to answer the questions as thoroughly as he or she wishes. If one use an unstructured interview it is important to be organized and have a plan for the topics to be covered. – Structured. The questions in a structured interview are predetermined and closed, like those in a questionnaire. This kind of interview works best if the goals are clearly understood and in order to work well the questions should be specific, short and clearly worded. All participants get the same questions and the questions often require a precise answer. – Semi-structured. A semi-structured interview is a combination of features from both structured and unstructured interviews. The interviewer has a script with basic questions for guidance in order to cover the same topic among the participants. There is a mixture of both closed and open-ended questions, and the typical approach is that the interviewer starts with preplanned questions and then probe follow-up questions until no new relevant information is presented. – Group. This kind of interview is based on a group of participants with representative users. The arrangement is that the whole group discusses the questions together. One popular variant of group interviews is called focus groups, and is discussed in more detail in section 3.3.4.

3.3.3

Brainstorming

Brainstorming is a technique for idea generation. A recommended group size of a brainstorming session is between five to fifteen people. The basic principle of brainstorming is that the participants collaboratively generate ideas on a specific topic or problem. One can generate ideas either by presenting a new idea or by further developing other participants’ ideas. It can be beneficial to have participants with diverse backgrounds or area of expertise in a brainstorming session; one idea can trigger new innovative ideas from participants with a different perspective on the current topic [33]. A prerequisite for a successful session is that one must create a relaxed environment where the participants feel comfortable. There are basically three rules that must be followed during a session [24], these are: – No one is allowed to criticize someone else’s ideas – One should not hold back or hide any thoughts; the goal for the group is to generate as many ideas as possible – Participants should be encouraged to combine or improve other ideas At the end of the session there is a need to systematize the results. One way to do this is to categorize the results collaboratively into suitable categories. During this process one should also clear repeated ideas and clarify under-specified notes into more concrete ones. This results in a structured set of ideas [24]. A disadvantage with brainstorming is that the problem is often too complex and cannot be solved by idea generations alone [33]. Furthermore, there is no evidence that brainstorming generates more or better ideas or solutions than if each participant spent equal time individually on the problem [24]. However, the technique can be used to generate ideas either for sub-parts or to the entire problem as well as give directional hints toward a solution [33].

24

Chapter 3. Method

3.3.4

Focus groups

The purpose of a focus group is to take advantage of the possibility that a group has the capacity to become more than the sum of its individual parts [22]. A focus group consists of a selection of people that share certain characteristics, relevant for the topic of interest. The size of a focus group can vary between three to ten people [34], however, Casey and Krueger [22] recommends a group size of six to eight people for best result. Discussions among the participants form the basis for the session; hence it is crucial with careful planning of relevant questions before the focus group session is conducted. Some important characteristics of a good questioning route are that the questions are openended, that they are sequenced and that they move from general to specific questions. That is, one question at a time is discussed and the questions should become more in-depth and detailed as the session progresses. A skilled interviewer is in charge during the session, the main responsibility of the interviewer is to lead the group to get the answers to the questions of interest. It is important that the participants focus on the topic at hand and not discuss irrelevant topics; this can be constrained by the interviewer. There are primarily three stages during a development process where one can use focus groups successfully [22], these are: – Early in the development. The main purpose is to gain a greater understanding regarding the problem domain. This is achieved by learning how the participants feel, think and talk about the area of interest. Designers can then use this information as a basis for the creation of prototypes of the program or product. – In the middle of the development. Focus groups in this stage can give designers early feedback regarding if the design is on the right track. Participants are asked to evaluate the designs as well as to compare and contrast each option, that is, what they like or dislike between the prototypes. – After release. The primary goal with a focus group in this stage of the development process is to evaluate the program. Questions regarding how one can improve the product and if it achieves the expected results is typically in focus during these sessions. Several focus group studies can be conducted on a specific topic. This allows for more than one target group to be studied. The results can then be compared and analyzed among the groups in order to get an even better understanding of different aspects of the problem domain [22]. A disadvantage with focus groups is that it might be hard to find suitable time and place for the participants to meet. It is also crucial that the interviewer is skillful so that the time is used efficiently and not wasted on irrelevant topics or issues [34].

3.4

Communicating with prototypes

Prototypes are a good approach to test new ideas for a design as well as to solve and communicate problems and opportunities within the development team and other stakeholders. The technique has proven successful since it allows a designer to evaluate multiple ideas in an early stage in the design process [38]. A benefit of prototyping is that it is fast and cheap in relation to the development of a fully functional system. The characteristics of a prototype can differ substantially; a developer may develop a prototype in a programming language while a designer can use sketches instead. Typically,

3.4. Communicating with prototypes

25

prototypes are divided into two different levels depending on how advanced it is and which technique used in the development; low-fidelity and high-fidelity prototypes [34]. Which kind of prototyping method to use depends on what part of the design that is to be evaluated; each method fills its own purpose [38]. Preece et al [34] differentiates design into two broad categories, these are: – Conceptual. This kind of design highlights a conceptual model of what the product will do and how it will behave. – Physical. The primary focus is on details of the design, such as menu structures, icons and other graphics. A design goes through both categories as the iterative design process consists of design, evaluation and redesign phases. Typically, the design evolves on a more detailed level the further into the design process one comes [34].

3.4.1

Low-fidelity prototyping

The characteristics of low-fidelity prototypes are that they are clearly a mock-up version of an interface, that is, they don’t look and behave as the final product. This kind of prototype is often very limited in scope and they are used early in the design process as conceptual proposals [34]. There are several approaches for this kind of prototype, such as storyboards, interface sketches or interactive paper prototypes [24]. These are described briefly below: – Interface sketches. Sketching is a technique to accomplish fast results of an idea or concept for an interface. The sketch illustrates a proposal of how the interface is supposed to look. Despite the simple technique of sketching, it is a powerful tool to communicate ideas among team members and to develop visions for the interface [24]. An interface sketch can be as simple as a wireframe representation of a system and as complex as a screenshot of a real system [38]. – Storyboards. This is a combination of interface sketches and scenarios. That is, a series of interface sketches form a walkthrough of how the system can be used to solve a specific pre-determined task [34]. An advantage of storyboards is the possibility to show dynamic properties of the interface and the ability to better communicate how the interface might be used. However, a disadvantage with storyboards is that it is very time consuming to revise if one has to make changes to the interface [24]. – Dynamic paper prototypes. This is a somewhat interactive version of a user interface. Paper prototypes are often drawn by hand and a prototype can have different levels of complexity. They can be as simple as a set of predefined state sketches that the user could end up in, but they can also incorporate more advanced features such as “real time” scrolling, drop down menus, pop up windows as well as drag and drop functionality. Interaction is simulated by the supervisor who acts as a computer and reacts to the user’s action during user tests. A dynamic paper prototype is fast and cheap to develop, however it can still produce valuable information regarding the interface [38].

3.4.2

Hi-fidelity prototyping

This kind of prototype is digital interactive versions of the interface, in contrast to lowfidelity prototypes that often are paper based. The prototype has the “look and feel” and the

26

Chapter 3. Method

responsiveness of a final implementation [24]. Hi-fidelity prototypes often involve some kind of programming in order to create the interactive features. This makes the prototypes more time consuming and expensive to develop than their low-fidelity counterparts. However, the accuracy of user testing increases since the system responds to the user interaction [36]. A hi-fidelity prototype can become very large and complex; sometimes it can be useful to develop a somewhat limited variant of the prototype. There are primarily two basic approaches for this [36], these are: – Vertical. A vertical variant only contains a subset of the prototype’s features and functionality; however, this subset is developed and presented with a high level of detail. – Horizontal. A horizontal variant contains a wide range of features and functionality; however, while the prototype offers many features the tradeoff is that the prototype has a limited level of detail on the information. Both of these approaches have a limited scope, however, they often provide enough functionality or details to be useful during usability testing and evaluation [36]. Other uses of hi-fidelity prototypes, besides testing technical and interactive features, is selling and communicating ideas to other stakeholders [34].

3.5

Evaluation

It is possible to gather rich and detailed information regarding a system by conducting studies of the actual use in the natural operation environment. This data can then be used in later stages in the design process, such as evaluation and design reiterations [4]. Typically users want a system to be easy to learn as well as efficient, effective, safe to use and satisfying. Evaluation is a crucial part in order to be able to develop a user-friendly system and it aims at finding out whether or not the users like it and understand how to use it [34]. Evaluation is a native part of the iterative design process and it should be performed in each stage of the design [11]. There are commonly two types of evaluation [11], these are: – Formative evaluation. The main purpose of this type of evaluation is to support the design process, that is, to formulate and refine a design. In order to accomplish this one must work closely with the users to gather their feedback of the system. Typically, this method is a qualitative evaluation which is used during early stages of the design process. – Summative evaluation. This kind of evaluation type is mainly used after the design or parts of the design are complete. Summative evaluation aims at answering questions regarding the impact, usability and effectiveness of the system. That is, it results in measurements regarding the performance improvements (if any) that the design entails. Both formative and summative evaluations give important measurements of different aspects of the design. Hence, one should not choose one or the other but rather perform both types of evaluation [11]. Preece et al. [34] introduces the DECIDE framework for evaluation. The basis for the framework is a checklist with the purpose of guiding novice evaluators through the evaluation process. The framework includes the following activities:

3.5. Evaluation

27

– Determine the overall goals that the evaluation addresses – Explore the specific questions to be answered – Choose the evaluation paradigm and techniques to answer the questions – Identify the practical issues that must be addressed, such as selecting participants – Decide how to deal with the ethical issues – Evaluate, interpret, and present the data The following subsections gives a short introduction to some methods and techniques used for evaluation of ideas, concepts and data during this thesis project.

3.5.1

Think-aloud protocol

A designer can get valuable feedback by observing users while they perform a set of tasks in the system. However, observation alone may result in a loss of crucial information since it does not give any insight regarding the users’ thoughts and expectations [11]. The think-aloud protocol aims to bridge this gap by encouraging the user to communicate their thoughts and expectations continuously during the observation. A challenge with the thinkaloud protocol is to avoid the silent periods that may occur if a user is unaccustomed or uncomfortable with communicating their thoughts verbally [34]. One way to avoid the problem is to allow the evaluator to prompt questions to the user during these periods [11].

3.5.2

Quick and dirty evaluation: Asking users

A fast approach to collect early feedback is to use a “quick and dirty” evaluation technique. The basic idea of this method is to meet users under a less formal setting than ordinary usability tests. Typically, the evaluator can get immediate feedback from the users by presenting the prototype to them; by watching and talking to users the evaluator can use the prototype as a basis for further discussion. A strength with this approach is that it can take place in a casual form anywhere and anytime, while still producing quick and valuable feedback [34].

3.5.3

Long-table approach

Typically, a large amount of ideas and comments emerge during a focus group session. The long-table approach is a method for systemizing the results into manageable chunks. The basic idea of this method is to collaboratively sort the comments and ideas into the predefined categories as well as create new categories where needed. The primary reason for the creation of new categories is that comments and ideas often overlap and regard two or more different categories. Instead of duplicating the comment, one can create new categories where they belong. After finishing the categorization each category is analyzed individually based on frequency, specificity, emotion and extensiveness [22].

3.5.4

Questionnaires

One common method for collecting information from users is to work with questionnaires. The results from a questionnaire are a good source for measuring the users’ attitude and opinion regarding a system [11]. The questions can range from simple binary (yes/no)

28

Chapter 3. Method

questions to Likert-scales and open-ended questions. Each type of question fulfill its own purpose; one should carefully decide when to use each question type based on the information need [34]. Some disadvantages with questionnaires are the complexity of designing a good questionnaire as well as the risk of a time-consuming analysis. However, the strength is that it can produce a vast amount of useful data [11].

Chapter 4

Accomplishments This chapter provides a description of how the work was conducted throughout this thesis project. The different phases of the development are explained in the following subsections.

4.1

Research phase

The primary goal with the research phase was to get a deeper understanding regarding the area of use and the user needs in order to have a foundation for the conceptual GUI design to the eX-IFS application. In order to accomplish these goals the research phase was planned and performed using user-centered techniques and methods. More specifically the research was planned to give more insight into the following aspects: – The target users – The usage context for the eX-IFS application – Problem areas in the existing system The basic idea was that the research phase results would be used in later stages throughout the thesis project. Since the result of the thesis project was meant to serve as a proof-ofconcept for a possible future development of a real application, the research was conducted with the aim that the project initiator could use the research results as a basis for that development. In order to collect the necessary information to the above listed aspects the following data-gathering methods was used: – Naturalistic observations – Contextual inquiry – Focus group sessions The gathered information was summarized and documented for use in later stages of the thesis project. The following sections describe the use of the data-gathering methods in more detail. 29

30

4.1.1

Chapter 4. Accomplishments

Observing users

The goal with this phase was to get a deeper understanding regarding the users and their work tasks as well as discussing problems with the existing systems GUI. Another motivation for this phase was to investigate which kind of information that was relevant to most users. This was considered highly important since the employees at the department have different user-roles and a responsibility for different stages of the support process. A total of 10 observation and interview sessions were conducted with a mean-length of approximately 1 hour per user. The observations took place in the users’ natural work environment and each observation session was combined with a user interview (see section 3.3.2 for details). The users were encouraged to first explain their work tasks and then to show how they interacted with the existing system in order to find the information they needed. During the observation notes and questions were continuously documented for later use.

4.1.2

User interviews

As described in the previous section, the user interview was conducted in conjunction with the user observations. The selected interview approach was contextual inquiry (see section 3.3.1 for a more detailed description). The primary reason for the use of this method was that the user observations were performed simultaneously. By watching users in their natural environment “silent information” that would have been lost outside of their workplace could be found and documented. When the users were finished with the interaction demo using the existing system (as described in section 4.1.1) the interview started. Since the users had so different user roles, no pre-determined questions regarding their work-tasks were included in the interview. Instead information collected through the user observations served as a basis for that type of questions. The questions asked throughout the interview mainly regarded issues with the GUI of the existing system, the lack of functionality to solve their work-tasks, the desires for functionality in the eX-IFS prototype as well as general usability questions. Since the interview took place in the users’ natural environment, the users had the possibility to show and discuss issues and questions on a deeper level since they could interact with the existing system during the interview. During the interviews continuous notes were taken and they proved to be highly useful during the design phase. A summary of the interview results can be found in Appendix A.

4.1.3

Focus group sessions

During the research phase two focus group sessions were conducted, one with employee’s at Saab’s after-sales department and one with students from the M.Sc. Programme in Interaction Technology and Design Engineering at Ume˚ a University. The sessions were carefully planned in advance and a Moderator Guide (Appendix B) was developed and used as a structure throughout the sessions. A conference room with a large table and a Whiteboard was booked on each location. Each session was divided into two parts with a total duration of approximately 3.5 hours (2x1.5 hour active time) including a coffee break. Before the actual session started the participants were asked to read a short introductory document that briefly described the thesis background, the session structure and the problem at hand. The motivation for this was to give the participants some insight regarding the topics to be discussed. The focus group session was audio recorded for later analysis.

4.2. Design phase

31

The first part was a brainstorming activity based on usability aspects identified during the research phase. This part was the same for both groups and aimed at finding new ideas of relevant functionality for the eX-IFS prototype. This part was further divided into six sub-parts where ideas should be generated around a specific topic, one at a time. The Whiteboard areas were divided likewise. Before the session started each participant got a pencil and a set of Post-It notes. When a participant came up with an idea he or she wrote it down on a Post-It and then placed the note in the column for the topic at hand on the Whiteboard. When the first part was finished all participants discussed and classified the results into more detailed categories. This served as documentation for later analysis. The second part differed among the focus groups and consisted of open-ended questions to allow more in-depth discussions among the participants’ area of specialty. The employees at Saab discussed questions regarding the existing system, desired functionality and future development of the eX-IFS prototype while the students discussed visualization, interaction and usability issues. During this part the Whiteboard area could be used to communicate and discuss ideas among the participants by using text, sketches or other graphics. The focus group sessions generated many interesting functions and the discussions highlighted many important issues and aspects for the problem at hand, including how to visualize large amount of data and how to design navigation-interaction for this kind of system.

4.2

Design phase

The main goal with the design phase was to generate creative solutions and ideas for a userfriendly GUI that matched the needs of the target group. Several user-centered methods presented in the research section of this thesis were used in order to accomplish this. These methods are discussed in the following subsections.

4.2.1

Low-fidelity prototypes

During the thesis project two iterations with dynamic paper prototypes and evaluation were performed. The motivation for this was to find a suitable overall layout as well as to find more specific functionality for the final prototype implementation. Workflow paper-prototypes The first iteration with low-fidelity prototypes aimed at finding a suitable design that matched the workflow of the users’ everyday task. Based on the user observations and interviews three different design proposals were created. Since the primary goal was to evaluate how well each design proposal matched the actual workflow instead of evaluating the actual design, little or no consideration was given to properties like color, font and graphical design. The prototypes were based on paper and drawn by hand, techniques for creating and evaluating paper prototypes were based on [38], some basic interactivity could be simulated during user evaluations. A short description of each individual workflow proposal is presented below (for a more detailed description of each proposal see Appendix D). – Vertical. The layout was structured in a vertical manner. The search-block and report-block were found in a Tab-control at the top of the page and the search-results or component details were presented directly below the tab-control.

32

Chapter 4. Accomplishments

The basic idea with this layout was to separate actions and content. That is, grouping functionality where a user has to perform a conscious act such as searching or generating reports from pure component information. An overview of the vertical layout can be seen in figure 4.1.

Figure 4.1: Vertical workflow layout

– Horizontal. The layout was divided into three main parts (search, content, report) placed in a horizontal manner. On the left-hand side of the layout the search-block was located, the content/search results were located in the middle and the ’generate report’-block was located on the right-hand side of the screen. The basic idea with this layout was to support a workflow where a user searches for a component, gets the search-results or component details presented in the content area and then has the possibility to generate a report relevant to the current component. Figure 4.2 shows an overview of the horizontal layout. – Pop-up. The layout was structured in a way where much of the functionality and information was hidden until needed. The component details are always visible in the center of the screen. The search-block (located to the left) and generate report-block (located to the right) is hidden until a user clicks the Search or Report “button”, then a drop-down block appears with respective functionality. Detailed information about a component is shown when a user hovers over the component header. The basic idea with this layout was to have the possibility to use the screen real-estate more efficiently and still allow a user to easily access detailed information for the current presented content without requiring a mouse-click. An overview of the pop-up layout can be seen in figure 4.3.

4.2. Design phase

33

Figure 4.2: Horizontal workflow layout

Figure 4.3: Pop-up workflow layout

34

Chapter 4. Accomplishments

In-depth paper-prototype Before the development of the in-depth low-fidelity prototype started, the results from the workflow prototype were summarized and discussed with the project initiator. A decision regarding which concept to further develop into an in-depth prototype was taken in consent with the project initiator based on the results of the workflow-prototype evaluation. The main goal of this iteration was to develop a prototype that had enough details to be able to find usability problems as well as evaluate if the design suited the users and their tasks. The design was primarily based on the horizontal workflow concept although some parts were extended and re-designed based on the results from the evaluation of the previous iteration. Figure 4.4 shows an overview of the in-depth paper prototype structure.

Figure 4.4: Wireframe overview of the in-depth paper-prototype layout

Evaluation of paper-prototypes All evaluation session were voice recorded for later analysis and each session concluded with a short discussion with the respective user to collect additional feedback. The low-fidelity prototypes were evaluated using scenarios and the think-aloud protocol (see section 3.5.1 for more details about the think-aloud protocol). The scenarios were task-based, that is, the users got a description of what tasks to perform. In order to accomplish this they interacted with the prototypes at the same time as they were told to communicate their thoughts by talking aloud. The evaluator was allowed to probe the user with questions during the session in order to get a deeper understanding of the user’s thoughts and interaction. The evaluation of the workflow prototypes aimed at gathering feedback and user opinions regarding the suggested designs. The primary goal was to investigate which of the prototypes

4.2. Design phase

35

that the users liked the most and eliminate the other two. In order to accomplish this, users filled out a questionnaire after all prototypes were evaluated; some example questions were: – Which prototype best supported your natural workflow? – Which prototype offered the best information display? – Which prototype had the most intuitive interface? All workflow prototypes were evaluated by three users; more specifically, the same three users evaluated all three workflow prototypes. The scenarios for this iteration were developed in a way so the user had to navigate through the interface structure several times in order to find a suitable workflow. The evaluation sessions were performed using a variation of the Latin Square design. A Latin Square is an NxN matrix. This implied a 3x3 matrix in this thesis project since there were three users and three concept prototypes to be evaluated. The main purpose for this evaluation technique was to minimize the biased effect of evaluation order. If all participants evaluate the prototypes in the same order, they gain a deeper understanding of the problem domain and knowledge of similar GUI features of previously evaluated prototypes. This could result in unfair results. However, evaluating the prototypes in different order for each participant can minimize the biased results and create a solid foundation for future work. A more detailed description of the Latin Square design can be found in [12]. The evaluation of the in-depth paper prototype was performed with five new users, that is, the three users from the workflow evaluation session were excluded from this evaluation phase. During this iteration another set of scenarios was used. These scenarios were developed to let the users focus on functionality instead of workflow. The number of users was chosen based on Nielsen’s [29] theory that an evaluation by five users is enough to find about 85 percent of all usability problems in a prototype. The setting was similar to the evaluation of the workflow prototypes, that is, using the talk-aloud protocol and questionnaires as primary data collection methods. However, the questionnaire was re-designed to better match the prototype scope, that is, the limitations in both functionality and scope of the managed information. Most questions were more detailed and focused on specific part of the interface, typical questions were: – How intuitive and usable is the interface? – Was it easy to find and understand the search functionality? – Was it easy to find the relevant information for a component? A summary of the evaluation analysis can be found in Appendix I.

4.2.2

High-fidelity prototype

After the two previous iterations an interactive prototype was developed and evaluated. The goal with this prototype was to get more accurate feedback regarding the interaction and GUI. The users could actually interact with the prototype for “real” during this iteration and get a feeling of how well it responds to their actions and fulfill their needs. This feedback was considered crucial for the future thesis work.

36

Chapter 4. Accomplishments

Figure 4.5: First version of High-fidelity prototype

Interactive Adobe Flex prototype The high-fidelity prototype was primarily based on the results from the evaluation of the indepth prototype. The interactive prototype was developed in Adobe Flex and ActionScript 3. The main motivation for this technique was that the final prototype would be a browserindependent web-solution. The choice of a technique that supported that functionality from the start allowed for the possibility to speed up the implementation a certain amount of code could be reused and extended for the final prototype. The interactive prototype had most of the desired functionality except for a PDFgenerator; however, the prototype was limited in the sense that it used a small amount of dummy information stored in a XML-file instead of using real data from the database. Figure 4.5 shows the first version of the POC GUI, this version was used during the evaluation sessions. Evaluation of interactive prototype The evaluation of the interactive prototype was similar to those performed during the lowfidelity evaluations, except that this iteration only regarded the final prototype variant. The users used the think-aloud protocol (see section 3.5.1 for further description) to communicate their expected result before, during and after interaction with the GUI. This evaluation was performed on five new users. That is, none of the users had previously evaluated the GUI. The users used a laptop running a webpage with the interactive

4.3. Final implementation

37

prototype embedded. The scenarios for the interactive prototype were a combination of the scenarios used in the workflow prototype and the in-depth prototype. By combining these scenarios the user had to explore both navigation and functionality to solve tasks in the scenarios. The session ended with a short discussion and a questionnaire just like in the previous iterations. The questionnaire for the in-depth prototype was re-used in order to be able to compare the results between the prototypes. The user-feedback was documented both during and after the session. When all sessions were finished the results and user-feedback were evaluated before the design of the final GUI prototype started.

4.3

Final implementation

The last iteration during the GUI development was a final implementation of the proof-ofconcept. This implementation was based on some code from the high-fidelity prototype and hence developed using Adobe Flex 3 and ActionScript 3. The final GUI was revised after the evaluation of the high-fidelity prototype and some functionality was added to resolve some issues found during the previous iteration. The changes regarded both navigation and tools to support the search process. An example of navigational issues was that some users wanted to use the browser’s back and forward buttons for navigation. Since this kind of application lives in the browser’s “Sandbox” this kind of behavior was non-standard for this version of Flex. However in the final prototype a variant of the Back-/Forward functionality was added using deep-linking in Flex. As for the search-process issues, one desired function was some kind of search history of the last searches. This functionality conforms to the theory in chapter 2 (Designing usable Search-user interfaces) and hence an area showing the user’s last 10 searches was added in the final proof-of-concept implementation.

38

Chapter 4. Accomplishments

Chapter 5

Results This chapter presents the achieved results during the thesis work as well as the final GUI design. The chapter presents some functionality and solutions in more detail with a description of how they conform to the theory of search interfaces in Chapter 2.

5.1

The eX-IFS GUI

The eX-IFS application targets expert users that need an application that supports them in their everyday activities. This has been the focus throughout this thesis project. The eX-IFS application work with dummy data and the GUI is based on the users’ needs and work activities. This implies that it might not be possible to implement a final application that includes every aspect of this POC without performing more research of how to retrieve the information from the corporate database. The main purpose of the POC is to develop and design an intuitive GUI that supports the users’ workflow as well as their search process. Searching is a central part of their everyday work activities, hence it is an important aspect of this thesis. The workflow, structure, composition and interactions are the result of the iterative usercentered design process performed during this master’s thesis project. The POC includes: – Navigation within the component hierarchy tree – Search for components by reference number or project name – Report generation with Alive PDF (limitations presented in section 5.2.5) Worth mentioning is that the eX-IFS RIA application should be embedded in an ordinary web page for easy access within the company network. The pictures in this report disregard the enclosing web page and only focus at the eX-IFS application GUI.

5.1.1

Layout

The users’ main activities are to search for a product, review its detailed information and generate a report for future follow-up. The layout consists of three main regions, these are: search, content, and report generation (see figure 5.1). Since the application is web based some guidelines targeting web design has been followed for the POC design, one of these is that the clickable parts are shown as hyperlinks. Note that some parts of the UI have 39

40

Chapter 5. Results

Figure 5.1: POC application overview

been colored for clarity only, otherwise no effort has been given to the graphical design as mentioned in chapter 1.

5.1.2

Start page

The start page is presented to the users immediately when starting the application, that is, when the users type in the URL for the eX-IFS application. The users’ natural entry point in the application is an overview of all delivered projects, that is, the root of the component hierarchy tree. From the root node a user should be able to traverse the component tree in a top-down approach all the way to the leaf nodes. Figure 5.1 shows the eX-IFS application entry point. The users’ have the possibility to go into a detailed view of each project or component by clicking the object id, that is, the hyperlink (see figure 5.3). This functionality is explained in more detail in section 5.1.3.

5.1.3

Search results page

The search result page shows the hits for the search query. Each search hit is presented with the reference number and a descriptive name in the header, where the reference number serves as a link to the components detail view. However, in some cases the reference number and the name is not enough to distinguish two or more components from each other. To solve this issue a set of hidden information is available for each component; this information can be displayed when needed. Another aspect of this functionality is presented in section 5.2.1.

5.1.4

Component details

The users’ need to be able to see certain details of every part in the system, hence, the detailed information is rendered in the same way for every part regardless of node type.

5.1. The eX-IFS GUI

41

Figure 5.2: Component details overview

That is, no matter if the desired part is a project or an individual component, each part has the same basic set of detailed information. Henceforth, each node-type is referred to as component. Figure 5.2 shows an overview of the component details information views. The detailed information consists of: – Overview. This is an overview of important basic information regarding the component. Typically, this information is of “good to know”-character, such as product status, End-of-life (EoL) status, and creation date. – Part of. If an error occurs in a component (or manufactured series of components) it is important to know where this component is located. From this view a user can get information regarding which projects as well as which components one level up the tree that includes the current component. – Consists of. The contrary to the “part of ” is the “consist of ”. This view provides the users with information regarding the sub-components that the component contains. The list of components is limited to one level down the components tree. – EoL-components. A very important user task is to investigate if a component or subcomponent is EoL. That is, if the estimated life-time of the component has passed. This view displays a list of all sub-components (one level down the component tree)

42

Chapter 5. Results

that is marked as EoL. This view also contains a bar chart over the amount of EoL components over time. – Documents. This view lists the documents related to the current component. This could for example be documents regarding tests or hardware composition. – Service/History. Users need to be able to get a quick overview of previous and active service errands. This view presents all service orders for the component as well as the current status for each service order. This view also contains some statistical information regarding the Mean-time between failures (MTBF). The statistics are displayed as a line chart. Worth noting is that the POC application GUI is based on the user needs, this implies that it might not be possible to extract all this data in a final application without more information regarding the database.

5.2

Search interface characteristics

The central activity in eX-IFS application is to search and find information regarding specific components in Saab’s naval products. This section presents some functionality that aims to support users during the search process.

5.2.1

Search results

The presentation of search results is important in this kind of system. The users must be able to find the desired component as fast and easy as possible. Some of the features of the search result presentation in the POC application that aims to support the users search process is presented below: – Result set. Each hit in the result list shows the reference number and the name of the respective component in the header. This information is the most significant information for the users. A user can click on the reference number to navigate to the detailed view of the component; this can be seen in figure 5.3.

Figure 5.3: Search results presentation

5.2. Search interface characteristics

43

If a performed search results in zero hits, the application will automatically try to refine the query by removing the last character in the query term until the result set is non-empty (see figure 5.4). It then displays the hits of the refined query along with a text that indicates that the query was reformulated.

Figure 5.4: Non-empty result set. If the search query string returns an empty result set the POC application automatically reformulates the query until the result set is non-empty

– Instant view. If a user performs a search that results in several similar hits, one can get more information regarding each hit by expanding hidden information. The basic idea of this functionality is to support the user in the decision making regarding the choice of component. Information regarding which information to show in the hidden part was found during user interviews and user testing. The instant view (discussed in section 2.4.3) approach simplifies the users’ decision without having to navigate through every component before finding the desired one. – Advanced search. Sometimes a user must use multiple search criteria; hence the GUI is prepared for advanced searching if needed. The advanced search allows the users to specify criteria such as operating country, technology platform, and project. The advanced search facility is hidden by default but can be made visible by the user when necessary. Some limitations of these features in the POC application is presented in section 5.2.5.

5.2.2

Search history

The GUI was supplemented with a search history function after the high-fidelity prototype user testing. Several users emphasized that it is a common procedure to return to a recent search and expressed a desire for this kind of functionality. This resulted in a list of the 10 most recent searches (see figure 5.5). Each item in the list is a link to the actual search, that is, clicking an item in the list will trigger the search functionality with the given parameters.

5.2.3

Search path

Since users navigate up and down the components hierarchy tree it should be possible to see where you have been, that is, how you got to the current component. The solution was to use path breadcrumbs. Each time a user clicks a component link in “consist of ”, “part of ”

44

Chapter 5. Results

Figure 5.5: Historical view over the 10 most recent searches

Figure 5.6: Path Breadcrumbs (within a search for: mp4)

or “EoL”, the current component is reloaded. The clicked item will serve as the new current component and its detail view will show. Figure 5.6 show the search path breadcrumbs. Whenever a user switches component, the search path is updated with the new component (if it does not already exist in the path), this implies that a user can trace the path taken to the current view. One can return to a previous component in the path by clicking an item in the search path. If the user performs a new search through the search box the search path will be reset to be able to display the future path within the latest search. During user testing several users tried to use the browser’s back and forward buttons to navigate through the previous searches. In the prototype that was evaluated this functionality was not implemented. However, after the user testing session an easy version of deep-linking was added to the prototype to allow user to navigate with the browser’s buttons as well.

5.2. Search interface characteristics

5.2.4

45

Search term suggestions

To support the users in their search process, a functionality that supported dynamic search term suggestions was implemented. The basis for this functionality was an auto-complete search box, that is, once a user starts typing in the search box a set of existing results that matches the user’s query string is presented. The list is filtered according to the query string and the suggested search terms become more specific as the user formulates the query. A user can choose to type the whole query string or select an appropriate item from the list of suggested search terms. Figure 5.7 shows the search term suggestion feature in the POC GUI.

Figure 5.7: Search term suggestions. The list is continuously filtered according to the query string

5.2.5

Limitations

Some of the limitations in the POC GUI are presented below: – Searching. A search in the POC GUI is limited to search terms consisting of a project name, reference number, or part of a reference number. This limitation is due to the lack of database connection in the POC GUI; the main reason for this is that the search logic should be implemented in the database and not the client application. The result of this limitation is that the advanced search functionality is only conceptual in the GUI and all possible scenarios that can produce empty result sets might not have been covered in the POC GUI. Another limitation is that there is no pagination or search result ranking algorithm in the POC GUI; however, the result sets are often quite limited so this limitation was disregarded in the project initialization. – Search history. The search history is a simple implementation, in the sense that it uses the previous query term to perform a new search. This functionality could be extended to save the state of the previous search, that is, to save the search path within each search in the history list. – Report generation. Since the POC GUI only deals with a limited set of XML data the generated reports use hardcoded dummy data, that is, no real data is used. However, the functionality shows that it is possible to generate real reports by implementing this functionality properly when one can use real data.

46

Chapter 5. Results

– Functionality. Some features of the POC GUI might be limited in functionality. This is due to the fact that it is a POC implementation. One example of a feature with limited functionality is the autocomplete search box. A user cannot select an item from the search term suggestion list using the keyboard; this is one example of a functionality that should be implemented in a final application. As mentioned earlier, the POC is based on the user needs and activities. It might not be possible to extract all data and information needed to use this kind of functionality in a real setting.

Chapter 6

Discussion and conclusion This chapter is a summary of the work and results of this thesis project. Important aspects of the different stages in the process will be discussed in further detail followed by a section of suggestions for future work.

6.1

Assignment

The assignment included both usability aspects as well as an implementation of a GUI for the eX-IFS proof-of-concept (POC) application. The main focus throughout this thesis project was to create a user-friendly GUI where searching was a central part of the user-interaction. The result of this thesis project (design and implementation) is not a complete application. A more thorough investigation regarding the problem domain and opportunities with this kind of application should be conducted. The assignment was very interesting and stimulating. However, there were several challenging parts throughout this thesis and a lot of obstacles were found during the project. Most of them regarded the possibility of completing the implementation phase of the final POC application. This is a direct consequence of Saab being a high-technology company with very high security policies on their computer environment. Several tools, such as WebORB, would have been highly useful during the implementation but had to be disregarded during the project because of these security limitations. Furthermore, other limitations that affected the thesis project was the lack of a prespecified requirement specification, lack of an entity-relationship diagram, and no pre-study regarding privileges and setup of the development environment as well as what kind of data that actually could be retrieved from the existing database. These obstacles had severe direct or indirect implications for the POC implementation and the time-frame for this thesis project.

6.2

Research and design

The in-depth study, “Designing usable Search User Interfaces” fulfilled its purpose to increase the understanding and knowledge of the problem domain. There were several possible approaches to the problem domain, one direction could have been to focus on search algorithms and the impact they have on search interfaces and user experience. This kind of information could probably be useful both for SUI designers as well as developers. 47

48

Chapter 6. Discussion and conclusion

Both data gathering and data analysis was made by a single individual. This implies that the collected data is mostly qualitative. The qualitative research is more sensitive for biased results than a quantitative method. However, the careful planning of each iteration in the interaction process should limit the possibility for misinterpretation of the results. The easy access and continuous discussions with users from the actual target group should also minimize the risk for misinterpretation of the user experience and usability issues found during this thesis project. More user testing and observation could have been performed; however this was not possible due to the limited time constraints for this thesis. The user testing sessions during this project was done in a research environment, that is, a controlled environment. However, a final implementation of the application should also be evaluated in the users’ natural work environment. This could increase the understanding about how users interact with the system in the context of use, as well as give valuable information of possible improvements in the GUI.

6.3

Proof-of-concept implementation

The POC application was planned to give a demonstration regarding how one can design a user interface for this kind of system. There was a desire for a fully functional application that could be used in the everyday operations on the TLS department. However, due to several different factors such as delayed development platforms, limited time constraints, complex it-architecture and restricted permissions to the company networks this could not be achieved. A limitation of the POC is that it is not integrated to the company database; instead XML-files serve as data source for the application. In order for the POC to communicate with the actual data source, that is, the company database, a WebService was required. However, the developed service was not fully compatible with the “Sandboxed” Adobe Flex implementation; hence a connection could not be established. It should be mentioned though that the POC is prepared for this kind of connection but it requires some changes in the underlying POC code. At the time of performing this thesis project Adobe Flex was the most mature RIAplatform. However, the development of other emerging techniques such as Microsoft’s Silverlight has significantly improved lately. One should perform a research study to evaluate which platform that is most suitable and beneficial for the company as well as the ease of integration with their IT-environment.

6.4

Future work

During the research for this thesis project, several possible extensions to this application were identified. It is likely that a structured coordination of these extensions could have a very positive effect on the everyday operation efficiency. An example of such an extension is integration with other existing systems as well as the ability to gather the information in one place. This could be beneficial since other user roles or departments could access the same information simultaneously. A more thorough pre-study that collects different departments’ needs with this kind of application should be performed and measured based on business value. If one wishes to use this kind of application it is recommended that the final implementation is designed and developed in a more object-oriented manner. The focus of this

6.4. Future work

49

thesis project was on usability and GUI design; this implies that no effort was spent on the underlying architecture, modularity or information flow. Furthermore, the development project should also have the required resources and permissions needed to develop this kind of system.

50

Chapter 6. Discussion and conclusion

Chapter 7

Acknowledgements I would like to take the opportunity to thank all people that supported me and in some way contributed to my work throughout this thesis project. Thanks to all people at Saab Systems for taking time for questions, discussions and usertesting. I would like to direct a special thanks to the project initiator and my supervisor Tobias Lindell. Thanks to my project colleague Mattias “MJ” Johansson, I enjoyed working with you and I appreciate all interesting discussions we had both on-site as well as off-site. Thanks to the people at Ume˚ a University, especially those of you that participated during the focus group session. A special thanks to my supervisor Lars-Erik Janlert in the Department of Computer Science at Ume˚ a University for supporting me during the thesis work and pointing me in the right direction. I would also like to thank my family and friends who supported and encouraged me during the thesis project. I couldn’t have done this without you. Some people that deserve special thanks are Anders Lundgren and Christian Olander that gave me valuable feedback on the report. And finally, thanks to Matilda who challenged me to finish this thesis, at last!

51

52

Chapter 7. Acknowledgements

References [1] Marcia J. Bates. Where should the person stop and the information search interface start? Information Processing and Management, 26(5):575–591, 1990. [2] Nigel Bevan. Human-computer interaction standards. Proceedings of the 6th International Conference on Human Computer Interaction, July 1995. [3] Hugh R. Beyer and Karen Holtzblatt. Apprenticing with the customer. Communications of the ACM, 38(5):45–52, May 1995. [4] John M. Carroll. HCI Models, Theories and Frameworks: Toward a multidisciplinary science. Morgan Kaufmann Publishers, 2003. [5] Pew Research Center. Pew internet. http://www.pewinternet.org/Static-Pages/ Trend-Data/Online-Activites-Total.aspx, [2010-03-30]. [6] Hao Chen and Susan Dumais. Bringing order to the web: automatically categorizing search results. CHI ’00 Proceedings of the SIGCHI conference on Human factors in computing systems, pages 145–152, 2000. [7] Andy Cockburn and Steve Jones. Which way now? analysing and easing inadequacies in www navigation. International Journal of Human Computer Studies, 45(1):105–129, December 1996. [8] Andy Cockburn and Bruce McKenzie. What do web users do? an empirical analysis of web use. International Journal of Human-Computer Studies, 54:903–922, 2001. [9] comScore. Press release: Global search market grows 46 percent in 2009. http://www.comscore.com/Press_Events/Press_Releases/2010/1/Global_ Search_Market_Grows_46_Percent_in_2009, [2010-04-01]. [10] Michael W. Eysenck and Mark Keane. Cognitive Psychology: A Student’s Handbook. Psychology Press Ltd., 5 edition, 2005. [11] Xristine Faulkner. Usability Engineering. Palgrave, 2000. [12] Marti A. Hearst. Search User Interfaces. Cambridge University Press, Blacksburg, Virginia, 2009. [13] Marti A. Hearst, Ame Elliott, Jennifer English, Rashmi Sinha, Kirsten Swearingen, and Ka-Ping Yee. Finding the flow in web site search. Communications of ACM, 45(9):42–49, September 2002. 53

54

REFERENCES

[14] Christoph H¨ olscher and Gerhard Strube. Web search behavior of internet experts and newbies. Proceedings of WWW9, 2000. [15] Keith Instone. Location, path and attribute breadcrumbs. 3rd Annual Information Architecture Summit, March 15-17 2002. [16] ITU. The world in 2010: Ict facts and figures. http://www.itu.int/ITU-D/ict/ material/FactsFigures2010.pdf, [2011-01-27]. [17] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. SIGIR ’05 Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 154–161, 2005. [18] Kyung-Sun Kim and Bryce Allen. Cognitive and task influences on web searching behavior. Journal of the American Society for Information Science and Technology, 53(2):109–119, 2002. [19] J¨ urgen Koenemann and Nicholas J. Belkin. A case for interaction: a study of interactive information retrieval behavior and effectiveness. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’96), pages 205–212, April 13-18 1996. [20] Anita Komlodi. Search history for user support in information-seeking interfaces. In CHI ’00 extended abstracts on Human factors in computing systems, CHI EA ’00, pages 75–76, The Hague, The Netherlands, 2000. ACM. [21] Anita Komlodi, Dagobert Soergel, and Gary Marchionini. Search histories for user support in user interfaces. Journal of the American Society for Information Science and Technology, 57(6):803–807, April 2006. [22] Richard A. Krueger and Mary Anne Casey. Focus Groups: A Practical Guide for Applied Research. Sage Publications, 3 edition, 2000. [23] Jin Ha Lee, Allen Renear, and Linda Smith. Known-item search: Variations on a concept. Proceedings of the American Society for Information Science and Technology, 43(1):1–17, 2006. [24] Jonas L¨ owgren and Erik Stolterman. Design av informationsteknik: Materialet utan egenskaper. Studentlitteratur, 1998. [25] Gary Marchionini. Interfaces for end-user information seeking. Journal of the American Society for Information Science, 43(2):156–163, March 1999. [26] Marissa Mayer. In search of...a better, faster, stronger web. velocityconference.blip.tv/file/2290442/, [2011-01-12].

http://

[27] Jacob Nielsen. Alertbox - search usability: Search and you may find. http://www. useit.com/alertbox/9707b.html, [2010-04-03]. [28] Jacob Nielsen. Alertbox - search: Visible and simple. alertbox/20010513.html, [2010-04-03].

http://www.useit.com/

[29] Jacob Nielsen. Alertbox - usability testing with 5 users. http://www.useit.com/ alertbox/20000319.html, [2010-04-04].

REFERENCES

55

[30] Jacob Nielsen. Alertbox - usability: 20030825.html, [2010-04-07].

101.

http://www.useit.com/alertbox/

[31] Jacob Nielsen. Alertbox - breadcrumb navigation increasingly useful. http://www. useit.com/alertbox/breadcrumbs.html, [2010-04-12]. [32] Tim Paek, Susan Dumais, and Ron Logan. Wavelens: a new view onto internet search results. CHI ’04 Proceedings of the SIGCHI conference on Human factors in computing systems, pages 727–734, 2004. [33] Gerhard Pahl and Wolfgang Beitz. Springer-Verlag, 2 edition, 1999.

Engineering Design: A systematic approach.

[34] Jennifer Preece, Yvonne Rogers, and Helen Sharp. Interaction Design: Beyond HumanComputer Interaction. John Wiley and Sons, Inc., 2002. [35] Bonnie Lida Rogers and Barbara Chaparro. Breadcrumb navigation: Further investigation of usage. Usability News, 5(2), August 2003. [36] Jim Rudd, Ken Stern, and Scott Isensee. Low vs. high-fidelity prototyping debate. Interactions, 3(1):76–85, January 1996. [37] Ben Shneiderman, Don Byrd, and W. Bruce Croft. Clarifying search: a user interface framework for text searches. DLib Magazine, Januari 1997. [38] Carolyn Snyder. Paper Prototyping: The fast and easy way to design and refine user interfaces. Elsevier Science USA, 2003. [39] Xuanhui Wang and ChengXiang Zhai. Learn from web search logs to organize search results. Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 87–94, 2007. [40] Barry Wellman and Caroline A. Haythornthwaite. The Internet in everyday life. WileyBlackwell, 2002. [41] Ryen W. White, Mikhail Bilenko, and Silviu Cucerzan. Studying the use of popular destinations to enhance web search interaction. Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 159–166, 2007. [42] Ryen W. White and Gary Marchionini. Examining the effectiveness of real-time query expansion. Information Processing and Management, 2006.

56

REFERENCES

Appendix A

Summary of user observations and interviews A total of ten user observations and interviews were conducted during the thesis project. The main purpose with these observations was to investigate the users’ needs and requirements, get an understanding of different user roles and their everyday work tasks, as well as to identify pros and cons with the existing application. However, since the final result of this thesis should be a proof-of-concept interface design, the main focus was to identify the most common activities among the user roles. The observations were conducted with users from the target group and they represented seven different user roles, among them project manager, configuration manager, system engineer, and hardware analyst. A summary of the most important results from these observations are presented below: Common activities: – Search for components – View information regarding the component and its sub-components. Important information is: • End-of-life • Component version • Part of/Consists of • Service orders – Conduct investigations of the component and its sub-components – Manual summarization of information regarding information, changes, and errors in a component (this is done manually based on a set of printed information of the component)

Problems with the existing application – The interface contains a tab-control with many tabs without fixed positions for each respective tab (the order changes depending on the currently active tab) 57

58

Chapter A. Summary of user observations and interviews

– Some tabs are empty and there is no indication whether or not a particular tab contains any information – It is not possible to navigate the component-tree upwards; one has to copy-paste the component-ID and perform a new search to be able to go to higher levels in the tree – The application feedback is useless, for example a search for a product that does not exist opens up a new window with bland information instead of an information message – Searching in some parts of the application is case-sensitive while searching in other parts of the application is not – The application lacks consistency regarding the use of right-and-left mouse clicks – The search functionality is limited since one can only use component-ID for most searches

Desired functionality: – A more flexible search functionality that supports more than searches for componentID – Functionality that supports easy traversing of the component-tree – Better overview of each component – Integration with other applications or databases (for example FRACAS) – Easy accessible information regarding End-of-life components (and sub-components) as well as some statistics of the components’ End-of-life status – The possibility to generate reports regarding the component (the desired reports differ among the different user roles)

Appendix B

Focus group The purpose of this document is to serve as a guide for the moderator during the focus group sessions. The session is divided into two parts, one general and one targeting the participants’ area of expertise. There should be a break for about 20-30 minutes between the different parts and some kind of refreshments should be available during this break. 1. Introduction (5-10 minutes) – Briefly explain the task and the expectations with the focus group session and run a short demo-session to give the participants insight in the process. – Handle out a Post-It block, a pen and a paper with background information and further instructions of the session structure to each participant. Give a short introduction about yourself and introduce the participants to each other. – The discussions should be kept as focused as possible on the topic at hand. Try to involve all participants in the discussions. The session will be audio recorded. The recording and the Post-It notes will act as documentations for later analysis. 2. Part 1 (75 minutes) Part one is a Brainstorming session. The goal is to generate new ideas for functionality and interaction for the eX-IFS prototype. (a) Searching (10 minutes) The only possibility for a user to find a system/product/component in the existing system is to search on article number. This means you have to use the article number that the desired system/product/component has in order to retrieve its information. How can the functionality be extended to offer the users a smarter search-function for large information sets? (b) Graphic design (10 minutes) The prototype might be able to present a large amount of data at the same time. However, many users only need certain information and functions depending of their user-role. How can the Graphical User Interface (GUI) be designed to be as usable as possible at the same time that it targets each user/user-role as precise as possible? 59

60

Chapter B. Focus group

(c) Reporting (10 minutes) Different user-roles have certain needs regarding reporting. Sometimes the users need to continue process the information in for example Microsoft Excel and sometimes they just have to create a report for the archives. The prototype should have a set of pre-defined report templates but new and different report templates might be needed in the future both within and outside of the after-sales department. One idea is to implement some kind of report generator where the users by themselves can define and specify SQL-queries through a GUI to retrieve the information they need. They might also have the possibility to define the graphics and layout of the report. What kind of functionality should such a report generator have? (d) Communication (10 minutes) Users of the eX-IFS prototype may have a need to communicate with other users based on certain information in a system/product/component (e.g. contact the person responsible for the electronics). How can this kind of communication functionality be solved? (e) Other functionality (10 minutes) Support-systems that allow the ability to manage systems/products/components are very important for many companies today. These kinds of systems often offer great possibilities for improving service and support errands both internally and externally. What functionality would be usable in this kind of systems? (f) Categorization and discussion (25 minutes) Discuss the generated ideas and re-categorize the results into more specific categories if needed.

3. Part 2 at Saab (75 minutes) The goal with this part is to encourage the participants to open discussions regarding their area-of-specialty. This group is specialized in the pros and cons with the existing system, what they need and desire/expect from a new system. (a) What are the main problems/issues with today’s system? The goal with this question is to identify the main problems with the existing system to be able to avoid them in the future. The main focus should be on information regarding usability issues, functionality issues, informational issues and integration issues. (b) What kind of functionality would be desired in the prototype? The goal with this question is to identify possible features and functionality that have been overlooked during user-observations/interviews. The participants should be allowed to discuss freely but the discussions should preferably focus on things somewhat relevant to the scope of the prototype. (c) What level of administration would be preferred in the prototype or a future system? The primary goal with this question is to get a sense of how much administration the users are willing to do in order to have an updated system. This includes administering of users and user-roles. (d) What kind of future developments could be possible or desired for this kind of system? The goal with this question is to get the participants to think

61

about the future developments of this kind of system. This could for example include possible integration with other systems, future functionality and so on in order to get a deeper understanding of the possible development needs and directions. The participants should be encouraged to discuss freely with little or no boundaries although discussions of realistic functionality is preferred.

4. Part 2 at Ume˚ a University (75 minutes) The goal with this part is to encourage the participants to open discussions regarding their area of specialty. This group is specialized in usability, visualization and development issues. (a) Information visualization The primary goal with this discussion is to get ideas regarding how to visualize the information for the users. The prototype might have to be able to process and give an overview of large system/productstructures with over 50.000 components. The participants should be able to discuss ideas freely as long as there is some kind of realistic solution. (b) Component versioning A component might exist in several different versions located in different systems/products. A user might have the ability to keep track of both the product version, the version history and where each version is located. The focus here should be to get ideas regarding how to simplify and visualize this for the users in a usable manner. (c) Modality and Information overload If the prototype is proved to be useful and able to successfully support the users at the after-sales department other departments might be interested of extending the prototype with different functionality and information. The goal with this question is to find inspiration regarding how to structure the GUI in order to support different user-roles and departments.

62

Chapter B. Focus group

Appendix C

Focus group analysis During the thesis project two separate focus group sessions were conducted, one at Saab and one at Ume˚ a University. The main purpose of the sessions was to benefit from each group of participants’ area of expertise. Both sessions were successful and gave a lot of valuable information for the thesis project. However, the focus group session at Ume˚ a University was somewhat limited; this was primarily a consequence of Saab’s restrictions regarding classified information and the project overall. The participants in that session couldn’t get enough information regarding the project to get a sense of the details of the project. A result of this was that many innovative ideas that could be highly successful in other projects but not in this particular one were presented. One example of this was the photo-match functionality, that is, one takes a picture of a component and then uploads it to a server that automatically tries to match the photo to all components stored in the database. Another example is the related search functionality such as ’other also searched for these components’. Both of these ideas could be useful in several other projects, however, the scope, restrictions and users’ work activities for this project make them more or less useless for the proof-of-concept. The participants in both sessions were very active and every participant got the chance to talk. Many interesting discussions occurred and the mediator only interrupted the discussions a few times when they got too much off topic. The most relevant ideas from the focus group sessions are presented below: Part 1 (Saab and Ume˚ a University): – Searching for more than component-ID – Filtering search results and other lists (such as End-of-life components) – Real-time term suggestions in searching – Show a short overview of the component to increase the “hit rate” for the correct component. That is, show information that enables the user to distinguish similar components from each other. – Provide several different report templates that enables users to generate different kinds of reports – A carefully designed detail-view of the component 63

64

Chapter C. Focus group analysis

– Make it possible to navigate to related information from within a component – Integrate actions with for example MS Outlook, that is, one could choose to receive alarms or e-mails when a specific component is updated or changed

Part 2 - Shortcomings of the current application as well as how to avoid them and better support the work activities in the POC-interface (Saab): It is hard to find information in today’s application, one often has to open up several instances of the application and perform several searches in order to find the desired information. This makes it very easy to lose the perception of exactly where you are in the application and the work process. The POC implementation should be easy to use and it should be easy to get an overview of the current application state (where you are in the application), where you came from, where you can go and to see the details of each component. One of the most important functions is that it should be possible to traverse the component-tree, upwards as well as downwards. The interface used today has low usability, and both users and decision makers are reluctant to use the application since it is so complicated and hard to find the relevant information. The everyday work activities at the TLS department are not supported in the application and it requires a significant amount of manual operations in order to compile and analyze information into reports. The POC interface should contain functionality that supports the user in their work activities. Some examples of this kind of functionality could be smart search functionality that does not generate empty result sets as well as the possibility to generate digital reports directly from the application. This would probably increase the service level to the customers as well as significant improvements in department’s efficiency. Part 2 - Usability, interaction and Visualization (Ume˚ a University): The application interface should be intuitive and easy to use. The interface should also be easy to learn and understand so users that only use the application sporadically can remember how to use it. Functionality that supports the users’ work activities should be provided; examples of such functionality are real-time search suggestions and filtering functionality. One could use buttons or hyperlinks to enable the possibility to traverse the componenttree, that is, one can navigate to a sub-/parent component directly from the detailed-view of the current component. One could also improve the user experience by implementing filtering functions that allow users to filter out relevant components in a list by providing significant parameters for the desired components. The application should follow general usability guidelines such as providing the user with understandable and relevant error messages if an error occurs.

Appendix D

Workflow prototypes (low-fidelity) This Appendix presents the workflow of each low-fidelity prototype. The following pages shows how to use each interface in order to find requested information. A summary of the scenario in the following pictures is∗: – Search for: “Radarenhet” – Select the correct unit from the list – Find information regarding the unit – Navigate to a related component, that is, a product in the component tree – Find information regarding the related component

∗ Note that the scenario in the following pictures is not the same scenario as during the evaluation sessions. The primary reason for this is that the following pictures are used only to get an overview of the prototype interfaces. The evaluation sessions were more in-depth and complex. The scenarios used during workflow prototype evaluations can be found in Appendix E

65

66

Chapter D. Workflow prototypes (low-fidelity)

Horizontal layout

67

Vertical layout

68

Chapter D. Workflow prototypes (low-fidelity)

Pop-up layout

Appendix E

Workflow prototype user tasks Moderator guide The purpose of this moderator guide is to give the users an introduction to the evaluation session. The information presented below should be read out loud to the user before the evaluation session begins. The evaluation is based on: – Think-aloud protocol – Probing – Questionnaires (for each prototype version) – Final survey (comparing the different prototype versions) – Closing discussions (after each evaluation session) Important information: – It is the prototype that is evaluated, not you! – The evaluation sessions will be voice recorded for future evaluation – The main objective is to present a conceptual design proposal to get valuable feedback regarding improvements for future development, not to present a final design Basic prototype information: – Based on previous investigation during the thesis project – The prototype is limited, both regarding information and design – The prototype is not in scale 1:1 – The interaction is limited and in some cases disregarded – The response time is slow (due to the manual computer simulation) Evaluation tasks 69

70

Chapter E. Workflow prototype user tasks

– Task 1 - Main focus: Find article information Task 1.1: Find the article with id ’RP667700-06’ (main article) Task 1.1.1: Find the following information: Country of export Task 1.2: Investigate if there exists EoL-components in the main article Task 1.2.1: Confirm the EoL-components (if any) Task 1.2.2: Investigate if the main article contains a EoL-component that exists in other systems as well Task 1.3: Investigate the MTBF of the main article Task 1.4: Find the information regarding export restrictions for the main article Task 1.5: Find a component/system that contains the main article Task 1.6: Investigate how many ’Servo units’ that the main article contain as well as the respective POS

– Task 2 - Main focus: Navigate the interface structure Task 2.1: Investigate how many failures the ’Radar unit’ (RP658900-06) in the main article had during 1999 Task 2.2: Investigate if the current article has any documents Task 2.3: Find a test regulation protocol for the current article Task 2.4: Return to the main articles overview page Task 2.5: Find a system where the main article is included Task 2.6: Investigate the degree of freedom for: • Case 1: Prototype == #1: 9LV435 FCS ASMD • Case 2: Prototype != #1: 9LV435 FCS MK3

– Task 3 - Main focus: Service and reporting Task 3.1: Find all work orders for 9LV435 FCS MK3 Task 3.2: Investigate if work order 2046 is closed Task 3.3: Find a ’radar unit’ that is located in Malaysia Task 3.4: Investigate the content of a product owner report Task 3.5: Generate a product owner report

Appendix F

Workflow prototype final survey 1. Rank the importance of the following interface features according to your opinion (1=Most important, 6= Least important) Easy to navigate structure (layout) Easy access to search field Easy to generate reports The possibility to use free-text search Good, clear and transparent information display A clear workflow For the following questions: Please observe that the order of the prototype list in this document might not reflect the order in which you evaluated the prototypes. 2. Which of the prototype versions do you perceive as most useful? (1=Most, 3=Least) “Horizontal” “Vertical” “Pop-up” List three good feature of the prototype you ranked as most useful: 1. 2. 3. List three bad features of the prototype you ranked as most useful: 1. 2. 3. 71

72

Chapter F. Workflow prototype final survey

3. Which of the prototype versions offered the best navigation features? (1=Most, 3=Least) “Horizontal” “Vertical” “Pop-up” Motivate your choice:

4. Which of the prototype versions offered the most transparent information display? (1=Most, 3=Least) “Horizontal” “Vertical” “Pop-up” Motivate your choice:

5. Which of the prototype versions offered the best functionality? (1=Most, 3=Least) “Horizontal” “Vertical” “Pop-up” Motivate your choice:

6. Which of the prototype versions offered the best workflow? 3=Least) “Horizontal” “Vertical” “Pop-up” Motivate your choice:

(1=Most,

73

7. Which of the prototype versions offered the best layout structure? (1=Most, 3=Least) “Horizontal” “Vertical” “Pop-up” Motivate your choice:

74

Chapter F. Workflow prototype final survey

Appendix G

In-depth paper prototype (low-fidelity) The in-depth low-fidelity prototype is highly similar in design as the horizontal layout in appendix D. The biggest difference of the two prototypes is that the in-depth version handles more information and it goes somewhat deeper into the design and navigational issues comparing to the workflow version. Because of the similarities in design this appendix only highlights a few of the design changes made between the evaluation iterations.

75

76

Chapter G. In-depth paper prototype (low-fidelity)

Appendix H

High-fidelity prototype user tasks Moderator guide The purpose of this moderator guide is to give the users an introduction to the evaluation session. The information presented below should be read out loud to the user before the evaluation session begins. The evaluation is based on: – Think-aloud protocol – Probing – Questionnaires – Closing discussions (after each evaluation session) Important information: – It is the prototype that is evaluated, not you! – The evaluation sessions will be voice recorded for future evaluation – The main objective is to present a conceptual design proposal to get valuable feedback regarding improvements for future development, not to present a final design Basic prototype information: – Based on previous investigation during the thesis project – The prototype is limited, both regarding information and graphical design – Some interaction features might not be fully implemented – The response time might not reflect the performance of a final version (due to the proof-of-concept implementation) 77

78

Chapter H. High-fidelity prototype user tasks

Evaluation tasks 1. You heard that some sub-components to a specific article has become End-of-life. Now you want to investigate whether or not this is correct as well as which other systems that are affected if this is true. You know that the article has a component-ID that starts with RP667700 and that it is located in the project 9LV453 FCS MK3E. Use the interface to perform the task/find the requested information. 2. You choose to return to the Start page and decide to search for the ’Radar unit’ with component-ID: RP141918-03. The interesting information is its sub-components as well as which systems that contains this component. Use the interface to perform the task/find the requested information. 3. You discover that that ’Radar unit’ is a part of the article with component-ID: RP667700-06. Suddenly you remember that there have been some problems with the ’Laser sensor’ in that component. You would like to investigate whether or not this article has some work orders and if so, if any of them has been marked as closed. Use the interface to perform the task/find the requested information. 4. You receive an e-mail with the information that something is wrong with the ’Servo unit’ in the article with component-ID: RP667700-11. This is the second time in a short period and you would like to find information regarding how many errors this component has had during the last 3 years and how long time it seems to be between the errors. You feel concerned over the frequent error reports for this component and you decide to create a report that contains information regarding the amount of errors and the relation between error and time. Use the interface to perform the task/find the requested information. 5. One of your colleagues calls you and complains that he can’t find the POS-information for the IR-camera in the Director with component-ID: RP667700-06. Find the requested information and send it in an e-mail to your colleague. Use the interface to perform the task/find the requested information. 6. Shortly afterwards, you receive another phone call (once again from the same colleague). This time he doesn’t find any test instruction documents for the ’Radar unit’ in the same Director as earlier (RP667700-06). You feel a little bit annoyed and investigate whether or not there is some test instruction documents for the component in question. Use the interface to perform the task/find the requested information.

Appendix I

Summarized prototype evaluation The POC interface has been evaluated three times on different iterations of the design process, two evaluations using paper and finally one computerized version of the interface. A total of thirteen unique participants participated in the evaluation sessions. During the evaluations the participants ranked the importance of some interface characteristics (see question 1 in Appendix F and Appendix J). The results are presented below (starting with most important): 1. Easy to navigate structure (layout) 2. Good, clear and transparent information display 3. Easy access to search field 4. The possibility to use free-text search 5. A clear workflow 6. Generate reports In the first evaluation iteration, three different conceptual interface designs were evaluated with the goal to investigate which interface that best supported the users’ work activities. The second evaluation was done on an in-depth prototype based on the results from the previous iteration. Finally, an evaluation of an interactive high-fidelity prototype was conducted. The interactive interface was based on the results from in-depth paper prototype. A summarized presentation of the results of each evaluation is found below: Workflow prototypes (3 paper based designs on 3 participants): The evaluation of the workflow prototypes showed that the most appropriate design was the horizontal layout. Each prototype was individually evaluated using the questionnaire in Appendix J, the evaluation session closed with a final survey that compared the interfaces (Appendix F). Every participant rated the horizontal layout as most useful and the pop-up layout as least useful. The summarized results for each evaluated part of the interfaces are presented in table I.1. Some comments of each prototype interface are presented below: 79

80

Chapter I. Summarized prototype evaluation

Prototype Horizontal Vertical Pop-up

Layout 7,27 6,80 5,87

Component Structure 7,25 6,92 6,17

End of Life 7,50 7,00 6,17

Search 7,58 6,92 6,67

Reports 6,83 7,17 5,50

Functionality 7,17 7,00 6,00

Total 7,27 6,97 6,06

Table I.1: The scores for each evaluated part for respective prototype version. The maximum score is 8,00 for each column

– Horizontal. The participants seemed to like that the information was divided and structured with a simple tab-control. There were several comments for this structure, among them that it allowed easy navigation and that it reduced the amount of information displayed simultaneously on the screen. One negative comment regarding the interface was that the search and report fields claimed screen real-estate all the time. Another was that there was some confusion regarding which parts that could be expanded and how to do it. – Vertical. The participants seemed to like that the component information could use more screen real-estate for presentation and they liked that the information was easily accessible; one participant commented that this structure provided a good overview of the current component since all information is presented at the same page and it was appreciated that one could directly expand more relevant information regarding the component. One interface feature that the participants lacked was a search result amount counter that represented the number of search hits for a search. One participant highlighted that it is important to see the number of search hits in order to decide whether or not to refine the search. – Pop-up. This layout was rated lowest by all participants. They perceived it to be hard to use and that the hidden information was somewhat confusing. One appreciated function was the search path breadcrumbs, two participants stated that it was really nice to see ’where I came from’. A functionality that got both good and bad reviews was that the search box and the report box could be made visible on-demand. One participant liked the idea of pop-up but stated that it could be used to display clarifying tooltip information instead of the component information. One possible explanation for the low rating could be that it was difficult to simulate the pop-up functionality in a fair manner. One interesting aspect of the workflow prototype evaluation is that the results of the final survey differed from the questionnaire results for each prototype interface. According to the final survey the vertical layout was slightly better than the horizontal layout (the pop-up layout still scored significantly worse than the other two prototypes). One possible explanation for this effect might be that there was some confusion regarding the naming of the prototypes, more specifically, during the evaluations the prototypes were called one, two and three while the survey names them horizontal, vertical and pop-up. Another possible explanation could be a form of bias: the department uses an old system that shares similarities with the vertical prototype, so it might be possible that some participant considered the vertical interface as the most useful since it was more familiar than the horizontal. The results from the individual prototype evaluation were considered as more important than the final survey. This decision was based on the fact that the evaluation questionnaire

81

for each design was a more in-depth evaluation with higher accuracy than the final survey. After a short discussion with the project initiator it was decided that the future interface designs should be based on the horizontal interface. In-depth low-fidelity prototype (1 paper based design on 5 participants): The structure of the in-depth prototype was based on results of the previous evaluations, that is, on the horizontal layout proposal. The participants gave positive comments regarding the layout and structure of the interface. Several participants emphasized that the information was tastefully divided into suitable parts due to the tab control. Two functions that were well received by the participants were the real-time term suggestions and the search path breadcrumbs. Some negative comments regarding the interface were some confusing headings and the display of irrelevant component details in the search result listings. There was also some confusion regarding the report generation: in order to generate a report the users were supposed to click either on the report name in the header or on a small thumbnail in the content detail view. However, the participant didn’t find this intuitive and suggested to add an ordinary button for report generation instead of clicking the thumbnail. Interactive high-fidelity prototype v.0 (1 computerized design on 5 participants): The participants’ positive comments regarding the interactive interface included that it had good presentation of the information, that the interface was well structured as well as easy to use. The overall rating score of the prototype interface was 6,80 out of 8,00. Most of the negative comments regarded graphical design issues, such as small font, lack of colors, unnecessary margins (the evaluated interface had a fixed width of 1024 pixels and was displayed on a widescreen laptop with high-resolution, hence the margins), and somewhat unclear marking of clickable links. Other small issues were misplaced information (such as the EoL bar chart that was originally placed under the Service and History tab) as well as vague formulations of headings and other texts. However, some bigger issues were identified as well; since the prototype is embedded in a web browser some of the participants found it confusing that they couldn’t use the browser’s back- and forward button. Two of the participants commented that they missed smart search functionality (that is, real-time term suggestions. This functionality was disregarded in version 0 of the interactive prototype, however, this functionality was present during earlier evaluations) as well as a search history list that presents a set of previously performed searches. Interactive high-fidelity prototype v.1 (1 computerized design on 0 participants): Based on the feedback collected during the high-fidelity evaluation session, some changes to the prototype were made. Unfortunately, this new version of the prototype was never evaluated due to time constraints. However, most changes were small and tightly coupled to the participants’ feedback. Some changes were: – Indicate clickable part of the GUI as web hyperlinks (blue and underlined) – Increased the font size, the size of GUI components as well as the resolution – Moved information such as the EoL bar chart (from the Service/History tab to the End of Life tab)

82

Chapter I. Summarized prototype evaluation

– Added a search history component that displays the 10 most recent searches for each individual user – Added real-time term suggestion functionality to the search box A more detailed description of the final interface can be found in chapter 5.

Appendix J

GUI evaluation questionnaire 1. Rank the importance of the following interface features according to your opinion (1=Most important, 6= Least important) Easy to navigate structure (layout) Easy access to search field Easy to generate reports The possibility to use free-text search Good, clear and transparent information display A clear workflow How well did you perceive (according to your opinion) that the prototype fulfilled the following statements? Instructions: Mark the most appropriate box in each respective scale with an ’X’. 2. General (Layout) 2.1. The interface was well organized

2.2. It was easy to distinguish the different parts of the interface

83

84

Chapter J. GUI evaluation questionnaire

2.3. The interface was easy to understand

2.4. The interface was easy to work with

2.5. It was easy to understand where to find the desired information

2.6. Suggested improvements regarding layout for the prototype

3. Component structure 3.1. It was easy to get an overview of: Part of

3.2. It was easy to get an overview of: Consists of

85

3.3. It was easy to understand how to navigate the component structure

3.4. It was easy to understand which components one had previously visited

3.5. Suggested improvements regarding component structure for the prototype

4. End-of-life (EoL) 4.1. It was easy to get an overview of the EoL sub-components in a specific component

4.2. It was easy to get an overview of other projects that is affected of the EoL component

4.3. Suggested improvements regarding the handling of EoL in the prototype

86

Chapter J. GUI evaluation questionnaire

5. Searchability 5.1. The search field(s) was easy to find

5.2. It was easy to understand how to use the search functionality

5.3. The search results was presented clearly

5.4. The “search help” was useful ∗

5.5. It was easy to pick the correct component from the search results

5.6. Suggested improvements regarding searching in the prototype

87

6. Reporting 6.1. It was easy to get an overview of the content of each respective report type

6.2. It was easy to understand how to generate a report

6.3. Suggested improvements regarding the handling of reports in the prototype

7. Functionality 7.1. The prototype implemented the most necessary functionality in order for me to do my everyday work

7.2. It was easy to understand how to use the different functionality

7.3. Suggested improvements regarding the functionality of the prototype

88

Chapter J. GUI evaluation questionnaire

8. Other 8.1. List three good things of the prototype 1. 2. 3. 8.2. List three bad things with the prototype 1. 2. 3. 8.3. Other comments/improvement suggestions for future work of the prototype

∗ The question regarding “search help” (Question 5.4) was disregarded in the high-fidelity evaluation. The main goal with the question was to get a rating for the term-suggestion feature. However, due to time-constraints this feature was not implemented before the evaluation sessions.