we performed a multi-task experiment in a desktop environment with a word-finding game and the .... a cross inside a pentagon, they needed to find and click the.
Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014
1521
Using Cursor Hints to Supplement a Less Interruptive Desktop Notification Ya-Hsin Hung Purdue University Mina Ostovari Purdue University In this study we designed an assistive notification mechanism that did not interrupt users' primary task so that they could focus on their main activity in a desktop environment. When a notification appeared in the peripheral area of the users' field of view, we gave hints such as the color of cursor or a label to cursor to lead their attention to the notification that was originally outside of users' field of view. To test our mechanism, we performed a multi-task experiment in a desktop environment with a word-finding game and the mouse cursor hints. The result revealed that the color hint as well as label hint coming from the cursor could increase the accuracy of participants’ ability to aware of the notification and decrease their subjective frustration while having little effect on their primary task. This implies that these approaches can be a potential assistive mechanism to current notification systems in a desktop environment.
Introduction
Copyright 2014 Human Factors and Ergonomics Society. DOI 10.1177/1541931214581317
A notification is a mechanism that alerts users about new information while they are focusing on a primary task (Fabian et al., 2004). However, current notification alerts may cause serious interruptions to the primary task which may decrease efficiency and accuracy of the primary activity (Iqbal & Horvitz, 2010). Many interfaces, such as web page advertisements and animated software agents, seem to be ineffective and distracting, and are abandoned or ignored after brief use (McCrickard, Chewar, Somervell, & Ndiwalana, 2003). Notifications are necessary as, without them, there would be many challenges for people to easily keep track of information from variety of sources such as messages and alerts from computer software (Landry, Pierce, & Isbell, 2004). In this study, we introduce a less-interruptive notification mechanism that can work with the existing notification mechanisms in the desktop environment. Our goal is to help people become aware of the notification or alert coming from a software or application with less interruption in their primary tasks. We believe the results of this study can help to increase the effectiveness as well as the accuracy of current notification without seriously interrupting users’ primary task, and also decrease users’ workload such as feelings of frustration and mental demand when they are dealing with notifications in a multi-task environment. The result may also help engineers and designers to develop a more effective and less-interruptive notification system.
Background Human’s mental resources are limited. When a person is receiving notification, this might somewhat affect primary task and decrease work performance (Landry et al., 2004). To solve this problem, previous researchers have come up with several ideas for designing better notification systems. Many studies about notifications focus on utilizing multiple modalities in order to prevent possible interruptions. Assistive equipment or tangible interface are popular solutions for
minimizing the disruption of notification (Bhatia & McCrickard, 2006; Warnock, McGee-Lennon, & Brewster, 2011). Some studies proposed systems that provide information only when sensors have been activated by users’ attempted action such as an eye contact sensor to detect someone’s gaze (Altosaar, Vertegaal, Sohn, & Cheng, 2006; Chen, Epps, & Chen, 2013), while some studies implemented notification systems using human’s sensory that is not occupied by users’ primary task, like utilizing haptic notification for timing awareness during oral presentations (Tam, MacLean, McGrenere, & Kuchenbecker, 2013). Besides using multiple sensory inputs to prevent attentional conflicts, some studies emphasize scheduling the time of notification to avoid interrupting users’ primary task. By monitoring users’ current tasks and analyzing them in terms of the task transition, researchers could use statistical models to realize defer-to-breakpoint policies for managing notifications, and reducing interruption cost like resumption lag, frustration, and errors (Iqbal & Bailey, 2008). Also, there were studies that combined the above two methods and aligned notification scheduling with the perceptual structure of the user tasks which could layer flexible scheduling policies on top of the detection mechanism (Iqbal & Bailey, 2010). Although there are many studies that focus on decreasing interruption caused by notifications, current approaches have some shortcomings. For external devices such as vibration motor or headphones, if neither the environment nor the object itself is equipped with displaying technology, users can not benefit from the augmentation of these designs (Mahler, Hermann, & Weber, 2009). Besides preventing attentional conflicts, giving subtle hints instead of directly informing users is also a good way to give notification with lower interruption (Zhang, Tu, & Vronay, 2005). Based on this concept, user acceptance is an important issue when designing ideal notifications. The urgency of the notification plays an important role on how users accept the notification; if users think a notification is urgent, they tend to tolerate the interruption caused by a high-urgency message (Vastenburg, Keyson, & Ridder, 2008).
Downloaded from pro.sagepub.com by guest on October 28, 2015
Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014
1522
Hypotheses
Primary Task
By applying our assistive notification mechanism, we conducted a multi-task experiment where participants needed to pay attention to a notification while doing a primary task. The definition of primary task here is user’s main activity in the desktop environment such like editing slides in a presentation software or making animation in a 3D design software.
The primary task was a word-finding game as shown in Figure 1. Each of the cards would show an English vocabulary word when a mouse hovered over it. The question for each trial was shown at the left side of the interface, participants needed to find out the words that matched the shapes that were shown to them and click on the cards that contained those words. For example, when the system showed a square inside a heart and a cross inside a pentagon, they needed to find and click the cards which had “square,” “heart,” “cross,” and “pentagon” written among 15 cards. If their choice was correct, one green circle mark would show up, otherwise a red cross mark would appear instead. The time limit for one trial was 15 seconds; if participants finished before the time limit, they needed to wait until the full time was up.
Our hypotheses were: H1 When we apply the assistive mechanism we proposed in this study, compared to the tradition notification system: H1A participants perform better in the primary task. H1B participants perform better in detecting a notification. H2 Participants would feel less demanded after adding our mechanism, which means when dealing with notifications in a multi-task work environment, our mechanism can decrease users’ cognitive loading in six aspects of NASA-TLX.
Method We conducted a multi-task experiment that consisted of a word-finding game as the primary task in which participants would find and click the cards that matched a question shown on the interface, and a secondary task in which participants needed to pay attention to a notification while doing the primary task; they needed to figure out which of the four icons in the screen had sent a notification. We designed the primary task to simulate users' daily activity while our assistive notification mechanism was defined as the secondary task. Since human's mental resources are limited, there might have been some counterbalance between those two tasks. Our goal in this study was to measure the effectiveness and error rate of the primary task, the notification response accuracy in the secondary task, and find out whether our proposed mechanism affected participants’ subjective workload measurement. The definition of effectiveness was the number of correct chosen cards while the error rate was the number of wrong chosen cards in one trial; At the same time , the number of trials that correctly answered where the notification came from was the accuracy of secondary task. We used a between-subject design for our experiment; it had four groups of participants; a control group and three experimental groups. Both primary and secondary tasks were the same in these groups. However, in the experimental groups we applied the mechanism we designed as a hint in the focal area while they were doing the primary task.
Figure1. The interface for the word-finding game. When participants click the right answer, a green circle mark will show up indicates it was correct; if they click on a wrong answer, a red cross mark would show up instead.
Secondary Task For the secondary task, participants needed to find out where the notification came from in the experiment interface and answered it at the end of a trial as a multiple choice question. When playing the word-finding game, a notification showed up from one of the four icons at the bottom of the screen during each trial, it would stay for 3 seconds and disappear, the timing of appearance ranged from the 5th to the 10th seconds of the trial, it would stay with the cursor until the end of the trial. In order to compare whether an assistive notification mechanism worked or not, participants did not receive any hint in the control group, while in experimental groups we gave hints to the participant.
Mechanism The assistive notification mechanism we proposed was when the notification appeared in the peripheral area of the users’ field of view, we gave hints in the focal area to lead users’ Downloaded from pro.sagepub.com by guest on October 28, 2015
Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014
attention to the notification that was originally outside of users’ attention. As shown in Figure 2, in the first experimental group the hint was changing the color of the cursor; the cursor’s color would change from the original black to the same color with the icon that showed the notification; for experimental group 2, the hint was adding a name label close to the cursor, a label that indicated the name of the icon that showed the notification; and for experimental group 3 the hint was both changing the color of the cursor and adding a name label close to the cursor.
1523
Measurement For the primary task, we chose the effectiveness and error rate in the word-finding game as the indicator of participants’ level of focus. For the secondary task, we measured the notification response accuracy by whether the notifications were either correctly acknowledged or ignored. The definition of effectiveness was the participants’ number of correct cards selected while the definition of error rate was the participants’ number of wrong cards selected. To better illustrate the overall picture, we translated the result into percentage by dividing the value of effectiveness by 4 and the value of error rate by 11, which were the maximum for each indicator. On the other hand, the accuracy measurement was the number of multiple-choice questions that participants correctly answered, which was also translated into percentage. For the mental loading, we used 7-Likert NASA TLX to measure participants’ subjective evaluation on trials, it provides criteria including: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.
Procedure
Figure 2. The appearance of mouse cursor changes when a notification shows up, (a)(b)(c)(d) indicates the appearance of the cursor for four different groups. (a) The normal state; (b) When the notification comes, the cursor color changes in the experimental group 1; (c) one label indicates the name of the icon that shows notification will appear next to the cursor in the experimental group 2; (d) one label as well as the color change will show in the experimental group 3.
Participants In this research we recruited participants through a crowdsourcing internet marketplace, Amazon Mechanical Turk. We carried out the experiment with 141 participants, however, in the crowdsourcing labor market, there are unwanted participants or random clickers who are only interested in the payment but pay little attention to tasks (Kim, Yun, & Yi, 2012; Shaw, Horton, & Chen, 2011). In order to exclude those unqualified participants, we ran a pilot test prior to the experiment to check the clarity of the instructions and decided the criteria for filtering unqualified participants. The criteria was that for a single participant, their total number of wrong chosen cards in the experiment should be less than 15, the total number of correct chosen cards should be more than 20 and the accuracy of multiple choice questions should be more than 40%, if any of his/her performance was less than the criteria, that participant would be ruled out. Eventually, we filtered out 21 unqualified participants and had a total of 120 legitimate participants (45 female), which remained 30 for each group. Their average age was 31.1, ranging from 20 to 58.
Participants in this experiment were recruited using Amazon Mechanical Turk. They were shown the announcement and introduction explaining the goal of the study without mentioning the concept of human’s visual focal area. After agreeing to participate, they were directed to an online survey and answered demographic survey questions. There were three demos before the real trials that taught participants how to play the game. After completing three demos, participants were asked to finish 10 trials in a random order containing a word-finding game and answer multiple choice questions about the notifications sent during the game, they would receive different kinds of cursor hints (or no hints) based on which experimental group they were in. When the trial time was up, participants were asked to answer which icon at the bottom had sent a notification and enter a code that was shown on the last screen of the game. Each trial took 15 seconds, the total number of cards they clicked was 40. At the end of experiment, participants were asked to fill out a post-task survey based on NASA-TLX. The whole experiment took approximately 10 minutes for each participant.
Result Effectiveness and Error Rate of Primary Task The result of participants’ effectiveness and error rate in the primary task are illustrated in Figure 3(a) and (b). We treated changing cursor’s color and adding label as the two factors in the analysis. With two-way ANOVA, we found no significant main effects of applying our mechanism (F(3,116)= 1.517, p=0.214, partial η2=0.038) on the effectiveness of participants’
Downloaded from pro.sagepub.com by guest on October 28, 2015
Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014
1524
primary task and no main effect on the error rate (F(3,116)=1.613, p=0.190, partial η2=0.040).
and experimental group 3. Error bars show 95% confidence intervals among all participants in that group.
Both changing cursor color as well as adding a label didn’t affect the effectiveness or the error rate of participants’ primary task significantly. Therefore, H1A was rejected by the result, implying that applying our assistive mechanism can’t significant increase the effectiveness in participants’ primary task and does not decrease the error rate.
Participants had higher accuracy with changing cursor color (Mean=81.2%, SD= 0.212) than without changing cursor color (Mean=67.0%, SD=0.278). Changing cursor color affected the accuracy of participants’ secondary task significantly (F(1,116)= 12.917, p
