Trust in Multimodal Sensory Cueing Automation in a ...

Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015

1017

Trust in Multimodal Sensory Cueing Automation in a Target Detection Task Timothy L. White U.S. Army Research Laboratory, Aberdeen Proving Ground, MD University of Central Florida, Orlando, FL Julia Wright, Joe Mercado, Tracy Sanders, and Peter A. Hancock University of Central Florida, Orlando, FL

Not subject to U.S. copyright restrictions. DOI 10.1177/1541931215591289

The goal of our work was twofold. The first was to examine the effects of dispositional trust on performance in a target detection task. The second was to examine the effects of performance on implicit and explicit trust in cueing modalities in that same target detection task. Fifty-four participants detected targets using four cueing modalities (non-cued, auditory cue alone, tactile cue alone, and combined auditory and tactile cueing). Participants monitored three screens for targets and responded as rapidly and accurately as possible when the presence of a target was perceived. Dispositional trust proved to be a significant predictor of performance for the auditory modality. Performance was a significant predictor of explicit trust in the tactile and combined conditions. Overall, participants reported preferring the tactile and combined cueing modalities for this target detection task. These findings suggest that measures of explicit trust should be employed early in system design to enhance eventual trust and system usability. INTRODUCTION Multi-tasking is now perhaps the embodiment of the modern workplace. Our contemporary world appears to demand that people in almost all professions be able to perform multiple tasks simultaneously. For example, a driver of a military vehicle in threat-laden conditions must maintain an appropriate distance from the vehicle in from of them while simultaneously processing information from multiple sources. Further, they need to scan for sources of threat while also monitoring instruments inside the vehicle itself (Hancock, Mouloua, & Senders, 2008). One way in which people are supported in their multitasking capacities is through the use of automation. Automation refers to the purpose-designed capacity to reduce the need for human intervention. Although partial task automation is an efficient strategy, the operator should be able to recognize when his or her skills are still required. Therefore, through the use of computer-guided automation, different tasks can be cued through different modalities (i.e., visual, auditory, and tactile stimulation) (see Merlo & Hancock, 2011). In this role necessitating periodic input, the human operator becomes a monitor or overseer who is tasked with intervening when and if necessary (Hancock, 2013; Muir, 1994). Automation is used to perform many tasks that are too dangerous, too challenging, or simply too boring for human accomplishment. Further, automation can augment the ability to perform tasks that are beyond the scope of human capability alone. Automation’s applications thus span many areas, which include but are not limited to military operations, manufacturing process, and medical procedures to name but a few. Today, automated devices are even used to assist in surgery (Manzy et al., 2011), and are currently being tested to produce autonomously driven vehicles on the open road (Cottrell & Barton, 2013). According to Lee and See (2004), trust can influence our reliance on automation, and therefore it is important that the degree of influence of trust in automation be empirically

evaluated and determined. Trust can be conceived as the degree of confidence that an individual feels in regards to a relationship (Muir, 1994). More formally, trust represents the reliance by one agent that actions prejudicial to the well-being of that agent will not be undertaken by influential others (Hancock, Billings & Schaefer, 2011). In our present case, trust is assessed through the relationship between the human and the automated cueing device. Trust is important to the integration of automated devices for many reasons. Appropriate trust levels are a major factor in successful interaction between the human user and the automation (Muir, 1987). Trust, as well as self-confidence, informs the user’s allocation strategy – specifically the decision to choose between manual control, a hybrid-sharing control, or purely automated control (Lee & Moray, 1994). In general, the more a user trusts a system, the more willing they are to choose the automated over the manual option. It has been shown that if a machine is not trusted it will not be used to its full potential; if it is used at all (Merritt & Ilgen, 2008). As Muir (1994) noted, “The operator’s choice of automatic or manual control has important consequences for system performance, and therefore it is important to understand and optimize this decision process” (p. 1905). The decisions users make regarding whether or not to use automation can have life-anddeath implications (Parasuraman & Riley, 1997). We have set our present work in a military context where these decisions can have profound effects. For example, if a soldier over relies on an automated target detection system and the automation fails, they may miss an enemy altogether, with potentially fatal consequences. Therefore, the goal of our experiment was to examine dispositional trust and how it mediates performance while considering explicit and implicit trust as functions of task performance. According to Merritt and Ilgen (2008), there are two types of trust. These are dispositional trust and history-based trust. Dispositional trust is a stable construct that describes an individual’s general feelings about an automated aid before any actual encounter. Since dispositional trust is dependent on

Downloaded from pro.sagepub.com at University of Central Florida Libraries on December 11, 2015


memory and a wealth of previous input from social sources such as films, television, books, etc. (Hancock et al., 2011), large individual differences in dispositional trust occur. Conversely, history-based trust is built through specific interactions. History-based trust can be expressed implicitly or explicitly. An individual’s implicit attitudes are based on their previous experience and may be outside of their conscious awareness (Fazio & Olson, 2003). Implicit trust measures use indirect probe methods to evaluate trust at a subconscious level. However, explicit trust measures focus on evaluating trust at a conscious level, typically by allowing individuals to state their perceived trust level. Masters, Miles, D’Souza, and Orr (2004) describe trust as incorporating four dimensions: competence, predictability, reliability, and faith in the system. Of these factors, reliability is one of the most widely cited factors in trust literature (Kelly, 2003; Dzindolet, Peterson, Pomranky, Pierce, & Beck, 2003; Hancock et al., 2011). A previous multi-modal cueing study in which trust was not evaluated revealed a performance decrement in relation to unreliable automation (Mercado, White, Sanders, & Wright, 2012). Although we acknowledge that reliability is a major factor related to trust, as an initial step to examine the relationship between trust and performance with these specific cueing modalities, the present work employs a 100% reliable automation. In cueing automation, individuals are asked to attend to multiple sources of stimuli. It has been demonstrated that divided attention has been successfully managed between spatially separate locations through the use of cueing (Muller, Malinowskim, Gruber, & Hillyard, 2003). While most cueing studies have focused on the visual modality, the use of auditory and tactile modalities has also been explored. One such study using U.S. Army personnel found that visual cues, 3-D audio cues, and tactile cues each significantly reduced the amount of time to engage an enemy combatant (White, Kehring, & Glumm, 2009). Another incorporated tactile stimulation in a driving simulator and reported that response time was 15% faster when participants used the multimodal directions as compared to only visual directions (Van Erp & Van Veen, 2004). Such findings suggest that response time is faster when using tactile cues than when using visual cues while the combination of the two can be even faster (Hancock, Mercado, Merlo, & Van Erp, 2013). Dual coding theory suggests that the presentation of information in multiple sensory systems yields better performance than presentation of information through only a single sensory system. Gellevij, Van Der Meij, DeJong, and Pieters (2002) found that information presented using two different cue modalities resulted in an increase in performance efficiency. Tindall-Ford, Chandler, and Sweller (1997) reported that when participants were subjected to combined audio and visual methods of presentation, learning increased compared to visual-only or audio-only presentation methods. The dual-coding theory has been studied primarily using the visual and auditory modalities. However, the tactile modality is now also being explored. One aspect of this study examined the dual-coding theory through the use of such various multimodal cues (i.e., audio and tactile).

1018

This study examined the dynamics of trust and performance in an automated multi-modal cueing task. In order to examine the relationship between performance and trust in automation across modalities, participants engaged in a multiple target detection task aided by cueing in various modalities. Since performance is affected by both success and errors, in the present experiment it was explained in terms of throughput. The throughput measure yields the number of successes per unit of time, and it allows comparisons to be made across tasks (Thorne, 2006), or as in the present study, across cueing modalities. Based upon the prior formal investigations, we offer the following hypotheses: H1: Dispositional trust in automation will be a reliable predictor of task performance. H2: Performance in a specific cueing modality will be a reliable predictor of implicit trust in that modality. H3: Performance in a specific cueing modality will be a reliable predictor of explicit trust in that modality. METHOD Participants Fifty-four students (18 male, 35 female, 1 unreported gender) from the University of Central Florida (UCF) volunteered to participate in this research for class credit. The age of participants ranged from 18 – 45 years of age (M = 20.7 years, SD = 5.6). Apparatus A custom-built computer, including three monitors, three speakers, and an Engineering Acoustics, Inc. (EAI) tactor belt was used (Figure 1). LabView software was used to synchronize the video stream across monitors, as well as record response times and accuracy measures. Each monitor displayed a specific task and a large ‘acknowledge’ button centered on the lower third of the screen. The left monitor displayed incoming text messages, the center monitor displayed a video of driving along a specified route (as viewed from the driver’s perspective, looking out through the windshield) and the right monitor displayed a ‘Blue Force Tracker’ system, which showed a map of the area with specific targets appearing at various locations on the map.

Figure 1. Experimental task and simulation environment. Shown are three monitors, each with a speaker mounted below, keyboard, and mouse. Also shown are the tactor belt with a battery pack, and reference sheet illustrating visual targets that appear on screens two (route markers) and three (Blue Force Tracker symbols).



The EAI tactor belt was located around the participant’s body and fastened behind with a Velcro closure. The belt is constructed of an elastic cloth material and held eight C2 vibrotactile actuators (tactors). When worn, one tactor is centered above the umbilicus, one centered on the spine and the other six tactors are equally spaced, three on each side. The tactor control unit connects to the tactor belt by a cable and uses Bluetooth technology to communicate with the computer-driven interface. The tactors are acoustic transducers that displace 200-300 Hz sinusoidal vibrations via 7mm dia. contactors and their 17 gm mass is sufficient for activating the skin’s tactile receptors. Tactile messages were single tactile bursts of 500 ms. that occurred in one of the three tactors on the front of the abdomen as the corresponding visual screen was being cued (i.e., left, right, and center). Audio cues consisted of a single auditory tone of 900 Hz for 500 ms. that emanated from a speaker beneath each of the three corresponding monitors. Cues were provided simultaneously with the presentation of targets. Questionnaires Two trust questionnaires were administered to evaluate the participant’s trust in automation. The pre-experiment questionnaire, based on the Human-Computer Trust questionnaire developed by Jian, Bisantz, and Drury (1998), was used to assess dispositional trust in automation. The postexperiment questionnaire, derived from the Trust and Usability Questionnaire by Madsen and Gregor (2000), was used to assess explicit trust in each cueing modality. Design This study was a within-participants design. There were four cueing conditions: no cueing (NC), auditory (A), tactile (T), and combined auditory and tactile (C). Each participant completed one trial in each cueing condition, and the order of conditions was counterbalanced across participants. To evaluate performance, a repeated measures analysis of variance (ANOVA) was conducted, followed by pairwise comparisons. Using simple linear regression, dispositional trust was evaluated as a predictor of performance, and then performance was evaluated as a predictor of explicit trust. A binary logistic regression was used to determine if performance was predictive of implicit trust. Procedure Upon arrival, participants reviewed and signed the informed consent materials, completed the demographic questionnaire and the pre-experiment trust questionnaire. They then were fitted with the tactor belt and seated at the testing station. Participants were shown examples of the different targets to be detected as well as how to respond by clicking the ‘acknowledge’ button on each respective screen. The participants were briefed on their role in monitoring the three screens, what targets they were watching for, the types of cues (no cueing, auditory, tactile, or combined auditory and tactile cueing) they would receive, and then engaged in a 5minute practice trial, which provided both tactile and auditory cueing. After addressing any participant questions, they then began the experimental trials.

1019

During the experimental trials, participants were asked to identify specific targets on each of the three screens. The left screen (screen one) displayed incoming text messages, and the task was to monitor these messages for specific messages from ‘Bulldog 6’. Whenever a message from Bulldog 6 appeared, the participant was to click the Acknowledge button on screen one. The center screen (screen two) displayed a video from the perspective of a vehicle driving along a specific route, and the task was to identify specific route markers, which would appear sporadically. Whenever the route markers would appear, the participant would click the Acknowledge button on screen two. The right screen (screen three) displayed a Blue Force Tracker system, on which symbols would appear on the map. Each time one of these symbols appeared, the participant would click the Acknowledge button on screen three. Every participant completed four trials, one in each cue condition, and each trial lasted approximately five minutes. After the participants had completed each trial, they completed the post-experiment trust questionnaire. Upon completion of all experimental trials, participants were told that they were to complete an additional scored trial in which they were allowed to use their preferred modality. Their selection was used as an inference of implicit trust. After participants had indicated their preferred modality, they were informed that there was no additional trial. Participants were then debriefed and dismissed. RESULTS Performance Performance on the target detection task was measured using the throughput calculation: Repeated measures ANOVA (α = .05) indicate a significant difference in performance (as measured using throughput calculations) between the cueing conditions, Wilks’ λ= .437, F (3, 51) = 21.909, p < .001, np2 = .563. Pairwise comparisons indicate that performance in the noncued condition (MNC = 0.426) was significantly lower than performance in the auditory (MA = 0.680, Cohen’s d = 1.324, p < .001), tactile (MT = 0.675, Cohen’s d = 1.306, p < .001), or combined (MC = 0.678, Cohen’s d = 1.293, p < .001) cueing conditions (Figure 2). There was no significant difference in performance between the cued conditions.

Figure 2. Mean throughput score by cueing condition. Note: Error bars depict standard error.



Dispositional Trust Participants completed the Trust in Automation questionnaire at the beginning of the experimental session to assess their dispositional trust towards automation (Min = 2.90, Max = 7.00, M = 4.87, SD = 0.842). Dispositional trust was evaluated as a predictor of performance in the cueing modalities using simple linear regression, α = .05 (Table 1). Dispositional trust was a significant predictor of performance in the auditory cueing condition, indicating the higher a participant’s dispositional trust in automation the better their performance in the auditory condition. R2 indicates that dispositional trust explained roughly 7.5% of the variance in performance.

1020

non-cued modality for the hypothetical final trial (Figure 3). Of those opting for cued modalities, the majority selected the combined auditory and tactile cueing (N = 27) over either the auditory cueing alone (N = 11) or tactile cueing alone (N = 12) conditions.

Table 1. Regression values for dispositional trust as a predictor of performance.

Cue Condition

Beta

t (53)

p-value

R2

NC

0.11

0.80

0.425

.012

A

0.27

2.06

0.045

.075

T

0.23

1.67

0.100

.051

C

0.10

0.71

0.484

.009

Explicit Trust Participants completed the Trust and Usability questionnaire after each experimental trial to assess their explicit (history-based) trust in each cueing modality. Explicit trust scores were slightly negatively skewed, (Auditory, Min = 2.28, Max = 5.00, M = 4.04, SD = 0.613; Tactile, Min = 2.14, Max = 5.00, M = 3.91, SD = 0.792; Combined, Min = 2.28, Max = 5.00, M = 4.13, SD = 0.684), but deemed acceptable for use in regression. Simple linear regression was used to evaluate performance as a predictor of explicit trust in the cueing modalities, α = .05 (Table 2). Performance was found to be a significant predictor of explicit trust for both tactile and combined cueing conditions, accounting for 10.3% and 7.6% of the variance, respectively. Table 2. Regression values for performance as a predictor of explicit (history-based) trust. Cue Beta t (53) p-value R2 Condition A

0.10

0.75

0.457

.011

T

0.32

2.44

0.018

.103

C

0.28

2.07

0.043

.076

Implicit Trust Participants’ selection of a cue modality for a hypothetical additional trial was used as an inference of implicit trust. Performance in each cueing modality was analyzed as predictive of implicit trust using binary logistic regression, α = .05, and was found not to be a significant predictor of implicit trust in any cueing modality. It is interesting to note that participants preferred cued modalities in general to the noncued modality, with only 4 of the 54 participants choosing the

Figure 3. Participant preferred cueing modality.

DISCUSSION This investigation examined the relationship between trust in automation and task performance in multimodal cueing scenarios. Based on the throughput measure, present findings clearly indicate that cueing automation provides an advantage for performance in a target detection task over a no cueing control. Although this is an anticipated outcome, it does confirm certain practical advantages of cueing (Elliott, Coovert, & Redden, 2009). A direct relationship between dispositional trust and performance in the auditory cueing modality was revealed, partially supporting the hypothesis that dispositional trust would be predictive of performance. One explanation for this finding may lie in the fact, that unlike the auditory modality, the tactile modality is not yet currently employed in an abundance of user interfaces. The vibration feature of a cell phone is probably the most common tactile interface. Therefore, most of the participants in this research may have had relatively minimal, prior experience with a tactile interface. Consequentially, a similar relationship did not exist between dispositional trust and performance in the tactile and the combined cueing modalities. In contrast, findings revealed that performance in the target detection task was predictive of explicit trust in both the tactile and combined cueing conditions, despite the participant’s relative unfamiliarity with the tactile modality. As performance improved, trust increased in both modalities, partially supporting the hypothesis that performance would be predictive of explicit trust. This finding corroborates with previous research that suggest that the tactile modality is a suitable communication means. The synergy and redundancy of the combined cueing modality may play a role in the relationship between performance and explicit trust (Sarter, 2006). It is not clear why performance was not found to be predictive of explicit trust with the auditory cueing. Performance in the target detection task was not predictive of implicit trust, as inferred by the preferred cueing modality, in any of the modalities. Thus, our hypothesis was not supported. Participants clearly preferred cued modalities over the no cueing. Specifically, most participants preferred



the combined cueing modality, which may be due to the redundancy provided by the combined cueing modality (Sarter, 2006). Also, the number of participants that preferred the relatively unfamiliar tactile cueing modality was almost equivalent to the number of participants that preferred the auditory cueing modality. This further highlights the potential of the tactile modality. CONCLUSION System designers generally build a prototype during their early design phases, during which it is a common usability goal to obtain user input in the design. Usability should be evaluated through measures of user effectiveness, efficiency, and satisfaction, allowing designers to understand and define system requirements early. Defining requirements early can guide the designers during later development phases. This study demonstrated the importance of evaluating user trust explicitly to optimize usability in the system. Evaluating explicit (history-based) trust during prototype development would enhance a system creator’s final design. This investigation employed cues that were 100% reliable as an initial step in examining the relationship between performance, user trust, and various cueing modalities, particularly tactile cueing. Because reliability is related to user trust (Hancock et al., 2011; Parasuraman & Riley, 1997) future research should examine the role of imperfect cue reliability as it relates to tactile cueing and subsequent trust in automation. Previous research has shown that trust in automation is reduced in situations where cueing is unreliable (Chen & Terrence, 2009). Empirically identifying acceptable levels of reliability for tactile cueing in automated systems can guide future design standards. Future research should also investigate increasing task complexity and/or target frequency, as these may have important effects on performance, and indirectly affect trust levels in the automation. REFERENCES Chen, J. Y. C., & Terrence, P. I. (2009). Effects of imperfect automation and individual differences on concurrent performance of military and robotics tasks in a simulated multitasking environment. Ergonomics, 52(8), 907-920. Cottrell, N. D., & Barton, B. K. (2013). The role of automation in reducing stress and negative affect while driving. Theoretical Issues in Ergonomics Science, 14(1), 53-68. doi:10.1080/1464536X.2011.573011 Dzindolet, M.T., Peterson, S.A., Pomranky, R.A., Pierce, L.G., & Beck, H.P. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies, 58(6), 697-718. doi:10.1016/S1071-5819(03)00038-7 Elliott, L.R., Coovert, M.D., Redden, E.S. (2009). Overview of meta-analyses investigating vibrotactile versus visual display options. HumanComputer Interaction. Novel Interaction Methods & Techniques, Lecture Notes in Computer Science, 5611, 435-443. doi:10.1007/978-3-642-02577-8_47 Fazio, R. H., & Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54(1), 297-327. Gellevij, M., Van Der Meij, H., De Jong, T., & Pieters, J. (2002). Multimodal versus unimodal instruction in a complex learning context. Journal of Experimental Education, 70(3), 215-239. Hancock, P.A. (2013). In search of vigilance: The problem of iatrogenically created psychological phenomena. American Psychologist, 68(2), 97-109.

1021

Hancock, P.A., Billings, D.R. & Schaefer, K.E. (2011). Can you trust your robot? Ergonomics in Design: The Quarterly of Human Factors Applications, 19(3), 24-29 Hancock, P.A, Billings, D.R., Schaefer, K.E., Chen, J.Y.C., de Visser, E.J., & Parasuraman, R. (2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53, 517-527. Hancock, P.A., Mercado, J., Merlo, J., & van Erp, J.V.F. (2013). Improving target detection in visual search through augmenting multisensory cues. Ergonomics, 56(5), 729-738. Hancock, P.A., Mouloua, M., & Senders, J.W. (2008). On the philosophical foundations of driving and distraction and the distracted driver. In: M.A. Reagan, J.D. Lee and K.L. Young (eds.) Driver Distraction: Theory, Effects and Mitigation. Boca Raton, FL: CRC Press. Jian, J.Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an empirically determined scale of trust in automated system. International Journal of Cognitive Ergonomics, 4(1), 53-71. Kelly, C. (2003). Guidelines for trust in future ATM systems: Principles. (HRS/HSP-005-GUI-03). Brussels, Belgium: European Organization for the Safety of Air Navigation. Lee, J.D. & Moray, N. (1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of HumanComputer Studies, 40(1), 153-184. Lee, J.D., & See, K.A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80. Madsen, M. & Gregor, S. (2000, December). Measuring human-computer trust. In: Proceedings of Eleventh Australasian Conference on Information Systems, Brisbane. Manzey, D., Luz, M., Mueller, S., Dietz, A., Meixensberger, J., & Strauss, G. (2011). Automation in surgery: The impact of navigated-control assistance on performance, workload, situation awareness, and acquisition of surgical skills. Human Factors, 53(6), 584-599. doi:10.1177/0018720811426141 Masters, J.K., Miles, G., D'Souza, D., & Orr, J.P. (2004). Risk propensity, trust, and transaction costs in relational contracting. Journal of Business Strategies, 21(1), 47-67. Mercado, J.E., White, T.L., Sanders, T. & Wright, J. (2012). Effects of CrossModal Sensory Cueing Automation Failure in a Simulated Target Detection Task. Proceedings Autumn Simulation MultiConference, San Diego, CA. Merlo, J. & Hancock, P.A. (2011). Quantification of tactile cueing for enhanced target search capacity. Military Psychology, 23, 137-153. Merritt, S.M., & Ilgen, D.R. (2008). Not all trust is created equal: Dispositional and history-based trust in human-automation interactions. Human Factors, 50(2), 194-210. doi:10.1518/001872008X288574 Muir, B. (1987). Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies, 27(5-6), 527-539. Muir, B. (1994). Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37(11), 1905-1922. Muller, M.M., Malinoswski, P., Gruber, T., & Hillyard, S.A. (2003). Sustained division of the attentional spotlight. Nature, 424(6946), 309-312. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230-253. doi:10.1518/001872097778543886 Thorne, D.R. (2006). Throughput: A simple performance index with desirable characteristics. Behavior Research Methods, 38(4), 569-573. Tindall-Ford, S., Chandler, P., & Sweller, J., (1997). When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 3(4), 257-287. Van Erp, J., & Van Veen, H. (2004). Vibrotactile in-vehicle navigation system. Transportation Research Part F: Traffic Psychology and Behaviour, 7(4-5), 247-256. White, T.L., Kehring, K.L., & Glumm, M.M. (2009). Effects of unimodal and multimodal cues about threat locations on target acquisition and workload. Military Psychology, 21(4), 497-512.
