Gwizdka, J. (2009d). Cognitive load and web search tasks. In Proceedings of the third Workshop on Human-Computer Interaction and Information Retrieval (pp. 54-57). Washington, DC: Catholic University of America.
Cognitive Load and Web Search Tasks Jacek Gwizdka Dept. of Library and Information Studies, School of Communication and Information, Rutgers University New Brunswick, NJ, 08901 USA
[email protected]
ABSTRACT Assessing cognitive load on web search is useful for characterizing search system features, search tasks and task stages with respect to their demands on the searcher’s mental effort. It is also helpful in examining how individual differences among searchers (e.g. cognitive abilities) affect the search process and its outcomes. We discuss assessment of cognitive load from the perspective of primary and secondary task performance. Our discussion is illustrated by results from a controlled web search study (N=48). No relationship was found between objective task difficulty and performance on the secondary task. There was, however, a significant relationship between search task stages and performance on the secondary task.
Categories and Subject Descriptors H.1.2 [Models and Principles]: User/Machine Systems – human information processing H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval – search process
General Terms Measurement, Performance, Experimentation, Human Factors.
Keywords Cognitive load, search task, user behavior.
1. INTRODUCTION Web search behavior is affected by the task, system, and individual searcher characteristics. These factors, either alone or in combination, influence the level of difficulty experienced by a searcher. One kind of difficulty is related to mental, or cognitive, requirements that are imposed by the search system or the task itself. Understanding what contributes to a user’s cognitive load on search tasks is crucial to understanding the search process and to identifying search tasks types and search system features that impose increased levels of load on users. As new user interfaces and interactive features are introduced into the information search systems we need to understand how the new functionality affects user performance and the system usability, usefulness, and acceptance. For example, user relevance feedback is a feature that was reported to be avoided by users due to the heightened cognitive load [1]. In the next section we briefly discuss cognitive load and provide a short overview recent research that used cognitive load in the context of information search. We then highlight our results demonstrating that mental effort varies across search task stages.
2. BACKGROUND The concept of cognitive load has been used in various fields that deal with the human mind interacting with external stimuli (e.g., ergonomics, psychology, learning). In this paper, we define cognitive load can as the mental effort required for a particular person to complete their task using a given system. Hence, at any Copyright is held by the author/owner(s). HCIR Workshop 2009, Washington, D.C., October 23 2009.
point in task performance, cognitive load is relative to the user, the task being completed, and the system employed to accomplish the task. It should be clear that cognitive load is of interest to interactive information retrieval researchers for two reasons. First, it can be used to characterize search interfaces with respect to cognitive cost. Second, it can be used to characterize user tasks and their elements with respect the required mental effort. Both perspectives have a long history in human factors and humancomputer interaction literature. Most recently, the first approach was elaborated by Wilson and schraefel at the last year’s HCIR workshop [16]. Wilson and schraefel proposed Cognitive Load Theory [4] as a tool useful in estimating cognitive costs of information search interfaces and proposed an inspection-based evaluation framework [18]. In other related recent work that exemplifies the first approach, Harper and colleagues established web page ranking according to their perceived visual complexity and linked it with cognitive load [9]. In CLT terminology, the first approach deals mainly with extrinsic load, that is with the complexity imposed by search interface and system. The second approach, deals mainly with intrinsic load, that is with search task demands on user’s cognitive resources. The primary goal of the first approach is to lower the extrinsic load so that user can commit more cognitive resource to the good germane load that facilitates task performance. The primary goal of the second approach is to understand better mental requirements of search tasks. A factor that often mediates the effects the task and the system are the user’s cognitive abilities (e.g., [6]). This paper promotes the second approach, and also considers selected cognitive abilities in addition to task performance factors.
2.1 Measurement of Cognitive Load Methods used to date to assess cognitive load included searcher observation, self-reports (e.g., using questionnaires, think-aloud protocols, and post-search interviews), dual-task techniques [5], [11], and various approaches that employ external devices to collect additional data on users (e.g., eye-tracking, pressuresensitive mouse and other physiological sensors [10]). The two latter groups of techniques have the advantage of enabling realtime, on-task data collection. However, use of external devices can be expensive and impractical. Hence, the promise of dual-task (DT) method that allows for an indirect objective assessment of effort on the primary task. Only few studies employed this method to assess cognitive load in online search tasks [12][5]. The dual-task technique measures directly instantaneous cognitive load at discrete points in time. The discrete values are typically used to calculate averages over time intervals of interest (i.e., during performance of a task or a task stage). The average values reflect the intensity of the load [13], [19]. The intensity is related to the overall load perceived by a person, but is not necessarily the same, as it is often assumed.
We present a study that employed the dual-task method as the technique for assessing cognitive load on web search tasks.
data collected for 48 users contained 288 tasks and 1447 task stages.
3. METHODOLOGY
The two main controlled factors were the objective task difficulty (OBJ_DIFF) and the search system (UI). The additional two independent factors were the levels of working memory (WM) and spatial ability (SA). We assessed intensity of cognitive load within each task stage by calculating the average reaction time (RT) to the secondary task events.
The details of the experimental methodology were reported in [7], [8]. However, the results presented in this paper have not been reported earlier. This section provides only this information that is needed for understanding the main points. Forty-eight subjects participated in a controlled web-based information search. Two cognitive abilities were assessed, working memory and spatial ability. The study search tasks were designed to differ in terms of their difficulty and structure. Two types of search tasks were used: Fact Finding (FF) - find one or more specific pieces of information, and Information Gathering (IG) - collect several pieces of information about a given topic. The tasks were also divided into three categories based on the structure of the underlying information need, 1) Simple (S), satisfied by a single piece of information; 2) Hierarchical (H), satisfied by finding multiple characteristics of a single concept (a depth search); 3) Parallel (P), satisfied by finding multiple concepts that exist at the same level in a conceptual hierarchy (a breadth search) [15]. Based on these characteristics, the tasks were categorized into three levels of “objective” difficulty. FF-S was assigned low difficulty level, FF-P and FF-H middledifficulty level, and IG-H and IG-P high difficulty level. During the course of each study session, participant performed a set of six tasks of differing type and structure. The search tasks were performed on the English Wikipedia by using two search engines with the associated search interfaces: U1 Google, and U2 ALVIS [3]. The order of tasks was partially balanced with respect to the objective task difficulty to obtain all possible combinations of low-medium-high and high-medium-low difficulty within the groups of three tasks. This yielded four task rotations that were repeated for two orders of user interfaces. Thus there were eight task/UI rotations. A secondary task (DT) was introduced to obtain indirect objective measures of user’s cognitive load on the primary search task. A small pop-up window was displayed at a fixed location on a computer screen at random time. The pop-up contained a word with a name of a color. The color of the word’s font either matched or did not match the name of the color. Participants’ were asked to click on the pop-up as soon as they noticed it. The secondary task involved motor action, as well as visuo-spatial and verbal/semantic processing. The modalities of the primary task and the secondary task overlapped. One could have reasonably assumed that higher demands on cognitive resources by the primary search task would be reflected in lower performance on the secondary task.
3.1 Data Collection and the Measures User interaction was recorder by Morae screen cam software from TechSmith and by the secondary task software. The interaction logs that were used in the analysis presented in this paper included time-stamped sequences of visited web pages, keyboard clicks, and mouse clicks. The latter were recorded for the primary and the secondary task. User search process was divided into four main task stages (Figure 1). We used a semi-automatic process to segment user interaction data into task stages. The process involved classifying URLs, and detecting patterns in the keyboard and mouse data. The
4. RESULTS The analysis presented in this paper focuses on the relationship between the independent variables and the performance on the secondary task (RT). The analysis was performed at the task and the task-stage levels.
Figure 1. State diagram of task stages. An analysis of covariance (ANCOVA) performed with the objective task difficulty, task stage and user interface as fixed factors and with the cognitive abilities as covariates revealed that mean reaction time differed significantly between task stages (F(3,862)=6.2, p
