Development of Touch Based Interaction Mechanisms for - Vanderbilt ...

Development of Touch Based Interaction Mechanisms for the  HuRT Project  Technical Report: HMT-10-01 Sean T. Hayes and Eli R. Hooten January 31, 2010

 

 

1 

Development of Touch Based Interaction Mechanisms for the  HuRT Project  Sean T. Hayes and Eli R. Hooten, Vanderbilt University, Nashville, Tennessee Objective: To develop an interface optimized for touch and multi-touch input for the mobile control of robot teams. Background: Currently, the Human Robot Teaming (HuRT) project uses an interface that was not initially designed to support touch-based interaction. This interface design makes touch input difficult to implement and use. Method: A new interface was designed to incorporate general multi-touch functionality (such as gestures for zooming and panning a map), as well as touch interactions that are more specific to the HuRT project. Ten people completed a within subjects user evaluation. The dependent variables assessed were task completion time, overall scenario completion time, workload, and subjective feedback from user surveys. Results: The post-completion scenario surveys and interface comparison surveys demonstrate that the new interface is significantly preferred and requires a lower workload. An analysis of the automatically logged experiment data shows that the new interface is significantly faster to use as whole for path and area entry. Also, that data suggests that less overall errors are made in the new interface. Conclusion: The new interface is preferred and demonstrates superior performance compared to the current interface.

Introduction  The Current HuRT Interface  The Human Robot Teaming (HuRT) project’s primary goal is to permit a single mobile human operator to interact simultaneously with teams of semi-autonomous robots and humans using high-level instructions (“KSU Hu RT Project,” 2009). Currently, this objective is accomplished through the use of a mobile interface that supports multimodal (i.e., touch, voice, etc.) interaction in a manner allowing a human operator to communicate tasks to robots and humans simultaneously. Mobility of the user is a significant criterion in the HuRT project, since this application is intended to be used in the field, where the user is moving at all times without being encumbered by the interface or its supporting hardware. Therefore, a user must be able to interact with the interface quickly and efficiently. For this reason, user input is accomplished via a tablet PC and headset that facilitate touch and voice input, respectively. A tablet PC is used because a QWERTY keyboard and mouse would greatly hinder mobility in the field and is therefore impractical as a primary input method. At the same time, most small touch screen devices do not have the processing or memory capacity to support the research objectives. Since touch-based input is one of the primary input methods, it is crucial that this input method be as robust as possible. The use of touch should, as often as possible, perform as good as or better than traditional input methods (i.e., using a mouse and keyboard). The purpose of this project is to help achieve this goal. The existing HuRT interface (EI), shown in Figure 1, can be divided into three main widgets. The map in the top left and center, displays the current robots’ location as 2 

well as a the visuall items used for task creaation. The right side of thhe interface contains statuss information n. Three tabs organize thhis informatiion: By Taskk, By Robot,, and Robot Statuus. The bottom widget is for task creaation, whichh displays alll the options necessary too createe one of the three robot task t types: Cross C Dangeer Area, Go to t X, and Reecon.

Figuree 1: The current HuRT interfacee. The interface is i broken into thrree principal secctions: the map, in i the top left annd T Panel in thee bottom center. centeer; the Status Pannel on the right; and the Create Task

The EI cu urrently utilizzes the built-in touch funnctionality of o the Dell Innspiron XT22 tablett PC on whicch the system m is tested (ssee Figure 2). This functtionality trannslates touchh inputt into mouse commands. The interfacce receives these t mouse commands and does not deal with w any tou uch input directly. The onnly gesture that t can conssistently be used u is the Pointt-to-Click geesture. Whenn a user placees one finger on the screeen, it acts ass a mousedownn event. Wheen the fingerr is lifted, a mouse-up m evvent is triggeered. The moouse-up evennt can be b used for seelecting optiions from menus and draagging the map m in order to repositionn it. Noo touch or multi-touch m evvents are inttegrated into the existingg interface.

Figure 2: Toouch input on a Dell XT 2 runniing the HuRT intterface.

3 

The origin nal HuRT innterface impllements a ruudimentary appproach to touch-based t interaaction. For th his reason, a new interfaace was desiggned to be more m accomm modating to touchh input by in ncorporating multi-touch capabilitiess. Multi-toucch interactionns involve the usse of a Multi-Touch Surrface (MTS),, which is tyypically a flatt surface witth proximityy sensoors underneaath that are capable of unnambiguouslly measuringg the positionns of multiiple finger co ontacts simuultaneously (Westerman, ( , 1999). The use of o an MTS alllows an inteerface to resppond to varioous types off touch-basedd interaactions. For example, coommon gestuures include:: sliding a finnger to scrolll, pinching two fingers f togeth her to zoom in, pulling two t fingers apart a to zoom m out, and rootating two fingers to facilitaate on screenn rotation. More M advanceed multi-toucch gestures can c also accouunt for the momentum m annd acceleration used whhen performinng the gestuure. These factorrs can then illicit i differeent responsess from the innterface (i.e.,, faster and slower s zoom ming, scrollin ng, etc.). A summary s of basic b multi-ttouch gesturres can be seeen in Figuree 3.

Figu ure 3: A visual depiction d of the pan p (left) and pinnch (right) multi-touch gestures.

Initial clieent interview ws indicated that incorpoorating such multi-touch gestures m the intterface easieer to use. Com mmon tasks,, such as zoooming, pannning the map, may make and taask assignm ment were connsidered to be b tedious. Therefore, T thhe team decidded to impleement multi--touch functiionality, wheere appropriiate, to makee these tasks easier to accom mplish. m multi-toucch gestures, other functionality has been b implem mented to Aside from makee the HuRT interface i eassier to use. For F example, the EI task creation wass accom mplished thrrough the intterface’s Creeate Task Paanel. This pannel was origginally desiggned to suppo ort mouse annd keyboardd interactionss and was latter extendedd to support voicee and touch interaction. i F this reason, it used common For c widdgets, such as pull down menuus, check box xes, and pussh buttons (seee Figure 4). While thesse widgets arre appropriatte for a keyboard an nd mouse intterface, theirr usability may m not translate effectively to a touchh-only appliccation. Anothher hurdle too touch interraction with the EI was the t fact that the usser had to rig ght click on screen to oppen the Creatte Task Paneel. While thiis is a simplee task to t perform using u a mousse, right clickking is usuallly more involved on a toouch

4 

interface, requiring the user to hold his/her finger in one position on the screen for a period of several seconds under the Windows 7 default settings.

Figure 4: The Create Task Panel. Note the various pull down menus, text fields, check boxes, and buttons used. These components can make touch-based interaction difficult.

Prototype Design  With the aforementioned shortcomings of the current HuRT interface in mind, a prototype was designed that supports multi-touch interaction. This multi-touch interface (MTI) makes use of two popular multi-touch gestures to manipulate the map: pinching to zoom in and out, and using two fingers to pan. These gestures are popular in many other touch interfaces including the Apple iPhone (“Apple - iPhone - Learn about high-tech features like Multi-Touch.,” n.d.), among others. These gestures have also been used in other robot control applications (Micire, Drury, Keyes, & Yanco, 2009). The pan gesture was implemented with acceleration, which allows the user to quickly swipe two fingers across the screen in order to pan the map rapidly in a particular direction. Aside from the multi-touch gestures, a new method of task entry was developed. In the EI, users are required to specify task waypoints on a map. These waypoints are connected programmatically by black connecting lines, as shown in Figure 5. In order to specify the waypoints, the user must touch on the screen at the location where the waypoint is to be placed. While this type of interaction is relatively easy for simple tasks (i.e., specifying a straight-line path that only requires two waypoints, the start of the line and its end), it can become tedious for more complex paths. For example, if the user wishes to specify a path along a winding road, the user has to add a waypoint at every curve in the road.

5 

Figure 5: The current HuRT interface visualizing a task on screen. Note the black connecting lines joining each waypoint (colored yellow) on the screen. Completed tasks are visualized without waypoints.

A new means of task input was designed in order to remedy this difficulty. This new method involves treating the on screen map similar to a canvas and is referred to as the Drawing Mode for the rest of this document. The Drawing Mode allows a user to simply “draw” paths on the screen to task the robot. The user initiates a specific task, such as a Go to X task, and then uses a single finger to draw a path from a start point to the robot’s desired destination. The interface interpolates the drawn line in real time and displays waypoints and connecting lines on the map that correspond to the user’s drawn line (see Figure 6).

6 

Figure 6: A drawn path that has been interpolated into a series of waypoints and connecting lines in the multi-touch interface.

The Drawing Mode can also be used to specify areas on the screen. For example, in the case of the Recon Area and Cross Danger Area tasks, the robot must complete the task based on the area specified by the user. Using the Drawing Mode, a user draws a shape on screen and the interface interpolates the shape as a series of waypoints and connecting lines on the screen (see Figure 7a). This method of shape drawing is particularly beneficial in the event that the user needs to specify a complex area. Once the task is added, the shape fills with a semi-transparent color in order to help the user differentiate line based tasks from those that require an area specification (see Figure 7b).

7 

Figure 7: a) A shape that has been interpolated into a series of waypoints and connecting lines. b) Once the task is added, the inside of the shape is filled with a color to help differentiate it from paths.

In addition to the drawing mode, another mode that utilizes the drawing concept was also added to the interface. This feature, called the Communications (Comms) Mode, was introduced as a means for the user to communicate information to other individuals who may be nearby. For example, the user can draw on the screen to indicate areas of interest, potential routes for the robots to travel, or even handwritten notes (see Figure 8). The user makes these markings directly on the map, which can be hidden, shown, or cleared at any time.

8 

Figure 8: The Comms Mode being displayed with several annotations on the screen. The user can choose to display or hide these annotations at anytime.

An added benefit to the Comms Mode is that it can be active while the user is tasking the robot. This means that a user can reference the communication information while specifying tasks, which prevents the user from having to look away from the screen to consult mission plans external to the interface. Also, since the mission plan is displayed on the interface, the user does not have to rely on his/her memory to recall the tasks specifics. The user is also able to make special notes (such as the hostile locations marked in Figure 8) that can be used to mark areas or findings of interest. Aside from the Drawing Mode and Comms Mode additions, the entire user interface was also redesigned. The Create Task Panel was removed from the lower portion of the screen and replaced with a new widget, the Menu widget (see Figure 9a). The Menu widget incorporates the mini map, buttons to access the Create Task and Comms widgets, as well as a button to minimize the entire panel (see Figure 9b).

9 

Figgure 9: a) The Menu M widget. Thhis panel contains the mini map, buttons to accesss the Create Tassk and Comms widgets, as a well as a buttoon to minimize thhe entire panel. b) The widget inn its minimized state.

The Menu u widget proovides a meaans of centralizing the locations of keey interface featurres. For exam mple, instead of the minni map beingg in the upperr left hand corner of the interfface by defau ult as seen inn Figure 1, itt has been reepositioned in i the lower left corner closee to the interfface’s navigaation buttons. This desiggn provides a single poinnt of refereence from which w all inteerface compoonents are acccessed and manipulatedd. The Menu u widget was designed too take up as little space as a possible on o the screenn. It waas also given a semi-transsparent fill so s that, if neccessary, the user can seee the map behinnd the Menu widget. u widget alsoo eliminates the need forr standard puull-down meenus, which The Menu are more m difficultt to use withh touch interaaction. Ratheer than rely on a pull dow wn menu forr task selection, s thee Create Tassk widget, which w is accessed from thhe Menu widdget, uses three large button ns (see Figurre 10). Thesee buttons proovide the useer easy accesss to a Task Speciification win ndow. Depennding on thee button seleccted by the user, u the Tassk Speciification win ndow will bee displayed with w the propper settings (see ( Figure 11). 1

Figurre 10: The Menu u widget with thhe Create Task menu m displayed. Three, T large butttons oriented vertically replacedd thee pull down mennu for task selecttion in the EI.

10 

Figurre 11: The Task k Specification window w displayedd on screen. Deppending on the button b selected by b the user in thee Create Taask widget, the Task T Specificatioon window will be b displayed withh the proper setttings.

The Task Specificatioon window iss essentiallyy a redesign of o the EI’s Create C Task panell. This windo ow is draggaable and launnches above the Menu widget w when the user clickss one of the three task buuttons in thee Create Taskk widget. Thhis is an alterrnative to thee Creatte Task paneel on the EI, which is dissplayed below w the map (ssee Figure 1). In the EI, the siize of the on n-screen mapp is comprom mised to dispplay the task specificatioon panel. Whenn the panel is not displayyed, the mapp does not reesize to fill thhe empty spaace. Thereefore, the Crreate Task paanel always compromisees the size off the map, reegardless of whethher or not th he panel is acctually in usee. We placedd the Task Specification panel withinn a winndow thereby y allowing thhe map to occcupy as muuch screen sppace as possiible at all timess. Since the MTI makes use of multti-touch gestuures for pannning and zoooming the map, the zoom to oolbar seen above a the maap in the existing interfaace (see Figuure 1) was wed the map’’s viewable area a to be inncreased evenn further in removed. This removal allow M the MTI. The MTI also removees the Task Status S panel located on thhe right sidee of the EI (see Figure F 1). However, during testing, this t panel was also remooved in the EI, E since it was not n used in either e interfaace during usser testing. Iff the MTI were to eventuually utilize the Task T Satus paanel, it is susspected that this panel would w be incoorporated intto the MTI inn a metthod similar to the Task Specificatioon panel. The remo oval of the Crreate Task panel, p as welll as the otheer modifications resultedd in an interface thaat shows a great g deal moore of the onn-screen mapp than the EII. In the worsst w is open andd the Menu widget w is dissplayed in case, (i.e., the Taask Specificaation window t MTI con ntains 19% more m viewabble map area than the EI. In the best case, c (i.e., full) the the Menu M widget is minimizeed and the Taask Specializzation windoow is not oppen) the MTII displaays a viewab ble area of thhe map that is i 42.9% greeater than in the EI. A side-by-side compparison of the two interfaaces can be seen s in Figurre 12 below.

11 

Figure 12: A viisual comparisoon of the EI (top p) and the MTII (bottom). 

Metthod  Once the MTI was coompleted, a user u evaluatiion was designed in ordeer to comparre the ussability of th he MTI withh the EI. The experiment was a within subjects deesign, comparisonss between thhe two interfa allow wing for quallitative and quantitative q faces from each participant. Each interfaace was testeed under Winndows 7. Thhe hardware used was thee Dell Inspiron I XT T 2 tablet PC, which nativvely supportts multi-toucch interactionn. Each

12 

participant used both interfaces individually and completed survey information for that interface (i.e., evaluation survey and NASA TLX) before moving on to the next interface. Interface presentation order was randomized to each participant. Each interface took approximately twenty minutes to evaluate. A complete user evaluation took between forty and sixty minutes. The hypothesis to be tested was that the new touch-based interactions and interface design modifications result in improved interface usability over the existing interface. It is expected that these improvements will increase the interface’s ease of use and accessibility during mobile operations in the field. Also of interest was the utility of the MTI’s new multi-touch gesture integration and drawing capability. In order to confirm the validity of the hypothesis, the objectives of this evaluation were defined as follows: 1. To identify and compare the time required for a participant to input tasks  using the interface’s current methods versus the newly developed methods.   2. To subjectively determine the participant’s level of satisfaction with each  system.  3. To subjectively determine with which interface the participant feels more  productive.   For the study, the following independent variables were identified:   • Participant computer knowledge and skills.  • Participant handedness – right and left.  • Participant gender – male and female.  • Each interface – the current HuRT interface and the MTI.  The following dependent variables were also identified:   • Time required to complete each task (Go‐To, Cross Danger Area, Recon  Area).  • Overall scenario completion time.  • Subjective workload.  • Post‐trial subjective questionnaire results.  • Task entry errors.   

Evaluation  The evaluation consisted of introducing participants to each interface, a brief training session with each interface, and the completion of three evaluation scenarios per interface. Participants were also required to complete demographic information, and evaluation surveys for both interfaces. During the completion of the scenario, data was captured in the form of time to complete assigned tasks. A timestamp was stored by each interface every time a participant started a task, finished a task, cancelled a task, and navigated the map. This timing data was used to objectively evaluate each interface.

13 

The three evaluation scenarios were designed in order to evaluate the participant’s ability to task a robot in each interface. These scenarios were summarized for the participant in the form of an image that displayed the map with tasks drawn on it. These task sets are provided in Figure 13. These task sets were developed to test each interface’s ability to create tasks that require both simple and complex shapes and paths. For example, Task Set I consisted of a straight line Go to X task, a rectangular Cross Danger Area task, and a small circular Recon task. Task Set II introduces multiple Go to X tasks, with path specifications that are more complex than the path seen in Task Set I. Task Set II also requires that the participant specify a complex area for the Recon task. Task Set II requires the participant to specify two Go to X tasks with paths that follow roads on the map. Each of these task sets provides a means of evaluating the functionality of the Drawing Mode for creating both simple and complex paths and areas.

14 

(I.)

(II.)

(III.)

Figure 13: The three task sets used in the user evaluation. Task Set I, II, and III are shown in ascending order.

15 

The administered surveys include a post-scenario completion questionnaire for each interface, a survey comparison questionnaire, and the NASA Task Load Index (TLX). The NASA TLX was used to determine subjective workload and the postscenario completion questionnaire was used to determine qualitative and quantitative information from each participant. The interface comparison survey was administered at the end of the participant evaluation and consisted of questions that required the participant to choose their preferred interface for each question.

Test Environment  Since the HuRT project is intended for mobile use, data was collected wherever a willing test participant was found. Even though this was the case, each test location possessed the following characteristics: • • • • •

Be relatively quiet so that the participant is not distracted by any auditory  stimulus during the evaluation.  Be somewhat isolated so no visual distractions can occur during evaluation.  Have access to a power source such that no power failures (i.e., a dead  battery) occur during testing.  Provide a comfortable area for the participant to interact with each interface.  Provide an Internet connection for entering survey data. 

The tablet PC was positioned on a table or desk. The participant was allowed to adjust the position of the computer in any way he/she felt was most comfortable. Many participants angled the tablet PC so it was more perpendicular to their line of sight. One participant placed the tablet PC on his lap. A few left the tablet PC flat on the table or desk.

Participants  Ten participants completed this evaluation. No participant was compensated for his/her time and each participant met the following criteria: • • • •

Have a working knowledge of computers, including, but not limited to, basic  knowledge of the Microsoft Windows operating system, types of input  (keyboard, mouse, etc.) devices, and computer operation.  Not possess any handicap that would prevent them from serving in the  United States military.   Normal or corrected‐to‐normal color vision.  Willingness and enthusiasm to freely give opinions about each interface  being evaluated.  

For the evaluation, participants varied in age from eighteen to fifty. These participants were given a demographic information survey in order to determine their education level, computer skill level, and familiarity with touch interfaces. Of the ten participants surveyed, 50% were male and 50% were female. 60% of the participants were between the ages of 19 and 25, and 70% of participants were between the ages of 18 and 35, which is the appropriate age range to register for military service. Education level varied 16 

significantly among participants, with 30% of participants having a high school diploma, 40% having a bachelor’s degree, 20% having a master’s degree, and 10% possessing an associate’s degree. Touch device usage varied between participants. 30% reported using a touch device (i.e., a tablet PC, touch enabled phone or media player, etc.) for more than twenty-one hours per week. 20% of participants reported casual (11 to 20 hours per week) touch device usage. The remaining 50% of participants reported little to no touch device usage, 0 to 10 hours per week.

Data Collection Methods  Data collection was facilitated through both the qualitative surveys issued to each participant and the automatic system logging of timing data obtained with each participant’s use of both interfaces. These collection methods resulted in the following types of data being collected:   • Time for completing each task.  • Time for completing the entire scenario.  • Time for communicating information using the interface.  • Qualitative information obtained from the post‐scenario completion  questionnaire.  • Quantitative information from the post‐scenario completion questionnaire. 

Results  This section first covers some of the objective discoveries revealed from parsing the timing data objectively obtained from the evaluation. Afterward, subjective results obtained through the post-scenario completion questionnaire, demographic survey, and other feedback is discussed. A data logger was developed to track participant actions from within both interfaces. This logger counted the amount of times a participant performed a particular action, as well as measuring the time it took to complete a series of actions (e.g., enter a task, draw a path, etc.). After collection, the data was analyzed in order to determine any significant trends. 

Scenario Completion Time  The total time participants spent performing each scenario was extracted from the logged data. Extraneous time before and after interacting with each interface was removed from the measurement (e.g., after the interface was started, but the participant had not started working on a task, and after the participant completed entering the final task, but had not exited the interface). The values used in this analysis were computed from the time between when the participant first selected a task until the time the final task was added. This data was then compared between interfaces and is summarized in Figure 14. The mean time to complete the scenario with the MTI and the EI was 3 minutes and 45 seconds (Std. Div. = 73) and with the EI it was 5 minutes and 33 (Std. Div. = 141) seconds. Subtracting the two mean times results in a 1 minute and 47 seconds mean completion time difference between the two interfaces.

17 

There was only one instance where a participant required more time to perform the scenario in the MTI than in the EI. In this instance, the participant took one second longer to complete the scenario using the MTI. In all other cases, the participants performed the trials faster using the MTI. A Student’s t-test was used to test the null hypothesis that it takes less time to complete the scenario in the EI than with the MTI (i.e., μΔ ≥ 0, where Δi = Bi - Ai and μΔ is the mean of the deltas for the population). The MTI was found to provide significantly faster scenario completion times, t(9) = -2.93, p < 0.0084. Therefore, this hypothesis can be rejected at a 0.01 significance level.

Scenario Completion Time Time in munutes and seconds

11:31 9:38

10:05 8:38 7:12

5:33 5:10

5:46 4:19

3:45

3:30

2:53

EI MTI

1:39

1:26 0:00 Minimum

Maximum

Mean (x̄)

Figure 14: The comparison of the mean, minimum and maximum times to complete the scenario

Task Specification Time  The time to successfully add a task was determined by capturing the complete time from selecting that task type option until the task was added, including when the participant cleared the task specification and began re-inputting data. The average time was calculated for each task type and for all the tasks combined. The number of tasks successfully specified in each scenario differs slightly between the interfaces, because some participants added extra tasks to “finish” the previous task. For example, one participant performed two Go to X tasks instead of one in order to get the robot to the desired location. In another instance, a participant added a Recon task when instructed to enter a Cross Danger area task. The number of tasks completed is presented in Table 1. Table 1: The total number of tasks created in both interfaces

  EI  MTI 

Go To X 49 51

Cross DA 29 29

Recon 20 21

Total  98  101 

Figure 15 provides the average times to specify each task. For each task type, the average task specification time was less in the MTI than with the EI. Also, for all tasks

18 

the average time to specify a task was ~5 seconds faster in the MTI when compared to the EI.

Average Task Specification Time 35

Time in Seconds

30 25 20 EI 15

MTI

10 5 0 Go To X

Cross DA

Recon

All

Figure 15: The average time to specific tasks

In order to perform a t-test, the inconsistent data was removed. This data removal was accomplished by counting the number of each task type each participant performed. If the participant performed the incorrect number of tasks for a specific type, those task types were thrown out. The reduction left 35 Go to X task pairs, 24 Cross Danger Area task pairs, and 18 Recon task pairs, totaling 77 task pairs. The average task specification times for this data are presented in Figure 16.

Corrected Average Task Specification  Time 35

Time in Seconds

30 25 20 EI

15

MTI

10 5 0 Go To X

Cross DA

Recon

All

Figure 16: The average time to specify tasks after inconsistent data was discarded.

19 

The t-tests were performed with the null hypotheses that the participants using the EI perform each task type faster than when using the MTI. For each task type, the null hypothesis can be rejected for each task for the overall task time comparison. The test statistic for the Go to X task is t(34) = -3.90, p < 0.002. The test statistic for the Cross Danger Area task is t(23) = -12.99, p < 0.001. The test statistic for the Recon task is t(17) = -4.55, p < 0.001. Finally, for a test statistic comparing all task pairs results in a test statistic of t(77) = -6.48, p < 0.001. These results proved to be the most significant of our user evaluations.

Path and Area Specification Time  We also investigated the time required for the participant to create paths and areas within each interface. This is of particular interest, because the methods of entry are fundamentally different. The EI allows a user to enter a single point at a time by tapping on the desired location. These points are connected by lines to form a path or area. The MTI allows a user to draw any shape. The points are generated based on the drawn shape. From the log file data, we extrapolated the time required to enter each path or area. We were unable to accurately distinguish between paths and areas that were accepted and those that were discarded; therefore, this data included entries that the participant discarded before a task was specified. Statistics were not performed on this data because it included paths and areas that were discarded. Figure 17 shows the average time to specify a path or an area. The Go to X task required the participant was required to enter simple and complex paths. The Cross Danger Area task required the participant to enter simple rectangles of varying sizes. The Recon task required the participant to enter complex shapes that rarely had straight edges. While the participants were instructed to enter the shapes as closely as possible to what was specified for the task, we observed that the participants were more careful to match the shape of areas than the paths. If this observation is accurate, then considering it with the fact that the most significant difference between the means was for the Recon task suggests that the performance increase in the MTI is more substantial for the more complex shapes. A follow up evaluation should be conducted to verify this supposition. Such and evaluation should include explaining to the participants the importance of the robots staying close to the desired path.

20 

Path and Area Entry Times 14

12

Time in seconds

10

8 EI 6

MTI

4

2

0 Go To X

Cross DA

Recon

All

Figure 17: The average time to create paths and areas in both interfaces.

Participant Errors  Three types of task entry error markers were tracked within each interface. Error markers do not equate to how many errors occurred, rather they track when a participant re-entered a portion of the task specification because of a mistake. As a result, if a participant makes two errors before noticing the mistake and then goes back, only one error marker is recorded. The error markers are: 1. Unfinished: A task specification is started, but the participant switches to entering another task instead of adding the current task, which excludes tasks that were canceled before a new task was started. 2. Cleared: A task specification is started and then reset so that the participant may begin again from scratch. 3. Canceled: A task specification is started, but then canceled because the participant did not wish to add a task. Table 2 shows the average number of error markers for each of these three categories per task and interface. The difference is large for the Cross Danger Area and Recon tasks. This difference may be an indication that drawing areas is less error prone than entering one point at a time. Unfinished tasks were particularly high with the EI, 16.5 times higher than with the MTI. Based on observing the evaluations, we believe the difference between the interfaces regarding unfinished error markers is largely due to the difficulty to select the correct task type from the drop down menu in the EI. The average participant cleared a task less often in the MTI than with the EI.

21 

Table 2: The average number of task entry errors per participant where x̄ is the sample mean and s is the sample standard deviation.

EI 

MTI  Cross DA  Recon 

     

Go To X  Cross DA

Recon

x̄ 

s 

x̄ 

s 

x̄ 

s 

x̄ 

s 

x̄ 

s 

x̄ 

s 

x̄ 

s 

x̄ 

Unfinished  Cleared  Canceled  All 

0.2  0.1  0.0  0.3 

0.4  1.0  0.0  0.6 

1.2  0.6  0.0  1.8 

0.4  0.0  0.0  0.6 

1.9 1.1 0.0 3.0

1.0 1.4 0.0 1.2

5.1 4.2 0.0 9.3

1.0 1.0 0.0 0.9

0.1 0.7 0.2 1.0

0.3 0.7 0.6 0.6

0.1  0.1  0.0  0.2 

0.3  0.0  0.0  0.2 

0.0  0.3  0.2  0.5 

0.0  0.7  0.6  0.5 

0.2 1.1 0.4 1.7

All

Go To X

All s  0.3 0.6 0.5 0.5

When participants were using the EI, there were no canceled task specifications. We believe this result is due to the fact that canceling a task in the EI removes the option to select a task, but does not free up any map space. In other words, there is no perceived benefit to the participant by cancelling, and hence, removing, the Create Task panel. Also, opting to cancel instead of clearing a task in the EI adds an additional step of reopening the Create Task panel the next time a participant wishes to add a task. Every participant experienced this issue during the training. However, canceling a task in the MTI has a perceived benefit, because it frees up map space without removing the task selection options, which are located in the Create Task portion of the Menu widget. This behavior makes the cancel option more useful after a mistake in the MTI than with the EI.

Post­Trial Survey Results  The post-trial completion surveys were compared to determine if participants preferred particular elements of each interface. As part of these surveys, the participants rated six elements of the interface: 1. Completing tasks in general. 2. Specifying Go to X tasks. 3. Specifying Cross Danger Area tasks. 4. Specifying Recon tasks. 5. The ability to pan the map. 6. The ability to zoom the map. These elements were rated on a Likert scale of one (extremely difficult) to nine (extremely easy). Note that these surveys were completed directly after performing a scenario. Therefore, half of the surveys were completed without knowledge of the other interface. Table 3 provides the mean and standard deviation survey results for each interface element.

22 

Table 3: The post-trial survey scores for each question number

Question # 1 2 3 4 5 6

EI Mean 5.8 4.8 4.9 4.9 6.2 5.3

MTI Std. Dev. 1.476 2.098 2.079 2.079 1.398 2.214

Mean 7.2 7.8 7.6 7.5 6.9 7.2

Std. Dev. 1.033 0.919 0.843 0.972 1.663 1.476

Using the survey data, the following null hypotheses were developed from the element list above: • H01: Completing tasks in general using the EI will be preferred by users over completing tasks in general using the MTI. • H02: Specifying Go to X tasks using the EI will be preferred by users over specifying Go to X tasks using the MTI. • H03: Specifying Cross Danger Area tasks using the EI will be preferred by users over specifying Cross Danger Area tasks using the MTI. • H04: Specifying Recon tasks using the EI will be preferred by users over specifying Recon tasks using the MTI. • H05: Panning using the EI will be preferred by users over panning using the MTI. • H06: Zooming using the EI will be preferred by users over zooming using the MTI. The nature of a within subjects experiment means that this survey data is not independent. Therefore, it is appropriate to analyze the data in pairs (i.e., the delta of each subject between each of the interfaces). To test the above hypotheses using a Student’s t  , where test, the values were calculated using the change in value pairs (i.e., ∆ i is a pair of values for one question on the surveys for the EI and the MTI). The test statistic is based on the null hypothesis that for a particular question the EI will score better than the MTI out of the entire population of users (i.e., ∆ 0).

23 

Table 4 provides the results of these hypothesis tests. The null hypotheses for H01, H02, H03, H04 can be rejected based on the significant results presented in

24 

Table 4. This result strongly suggests that specification and completion for all the tasks is significantly easier with the MTI. The null hypothesis for H06 can be rejected, suggesting that the multi-touch pinch gesture in the MTI was significantly easier to use for zooming, than the toolbar buttons provided in the EI. While most participants preferred panning in the MTI, the survey results of H05 are not statistically significant. This result is likely due to the fact that panning was very similar in both interfaces. The MTI provided the additional option to pan with two fingers as well as one, which was only necessary when inputting paths and areas during task creation.

25 

Table 4: The evaluation of 6 hypothesis tests based on the post-completion scenario surveys. The deltas, Δ, were calculated by subtracting the score for the MTI from the EI (i.e, Δi = Ei – Mi).

Mean of Δs (x̄) Std. Dev. Std. Error Test Stat. (t) Sig. Level (p)

H01

H02

H03

H04

H05

H06

-1.4 0.80 0.25 -5.53 0.0003

-3 2.26 0.72 -4.20 0.0012

-2.7 2.06 0.65 -4.15 0.0012

-2.6 2.01 0.64 -4.09 0.0014

-0.7 2.21 0.70 -1.00 0.17

-1.9 2.73 0.86 -2.20 0.028

Effect of Previous Touch Experience  The participant survey data also provided interesting data comparison between participants with and without significant experience using touch enabled devices (i.e., tablet PCs, media devices, the iPhone, etc.). Significant experience was defined as using a touch enabled device for more than ten hours per week. Of the ten participants, 50% met this criterion. Participant responses from experienced and inexperienced touch device participants were averaged for each of the six questions from the post-scenario completion questionnaire. The means and standard deviations of the post-trial questionnaire responses to each of these questions were calculated for both sets of participants. This data was then compared by taking the difference between the mean and standard deviation of each data set for each question. The resulting data is shown in Figure 18 below.

26 

Particpants With and Without Touch Experience 9 8

Mean Survey Rating

7 6 5 EI with Touch Experience

4

EI with no Touch Experience

3

MTI with Touch Experience

2

MTI with no Touch Experience

1 0

Figure 18: Survey responses of participants with and without touch experience

Figure 18 provides some interesting findings. For example, touch participants, on average, rated the EI lower in every category for the ability to zoom. In the case of the pan ability and zoom ability for EI, touch experienced participants were especially critical; ranking the pan ability 1.60 and zoom ability 2.20 lower than participants without touch device experience. Equally as interesting is the fact that non-touch familiar participants tended to rate the MTI’s ability to pan and zoom lower than those that did have touch device experience.   While the sample size is too small to serve as concrete proof of any significant finding, this data does lead to some interesting conclusions. For example, one can hypothesize that those participants with touch experience were more critical when evaluating the EI because they had some pre-conceived idea of how touch-enabled interfaces are supposed to function. Many participants reported frustration when using the EI. This frustration stemmed from the fact that the interface was difficult to interact with using touch. For those participants that possessed more experience with when touch devices, this frustration appears to be exacerbated according to the more critical evaluations of the EI. The fact that these same participants generally rated the pan and zoom capabilities of the MTI higher than those without touch-device experience provides support of this claim. Panning and zooming in the MTI was facilitated through using two very common multi-touch gestures. Participants with touch-device experience, especially multi-touch 27 

experience, would instantly be familiar with these gestures and know how to use them when presented with them in the MTI. This familiarity may have contributed to the participants’ use of touch devices rating this functionality higher in the MTI. If this is the case, it indicates that a bias exists for the two-finger pan gesture and pinch gesture among touch device users.

Post­Evaluation Survey Results  The post-evaluation comparison survey asked the participants to select between the two interfaces that they preferred for a particular aspect and as a whole. These comparisons included: 1. Preferred interface 2. Better touch input 3. More natural touch input 4. Better perceived productivity 5. Easier waypoint entry 6. More aesthetically appealing 7. Easier to specify tasks Table 5 provides an overview of these results. Participants unanimously preferred the MTI overall (1) and specifically for providing more natural touch input (2, 3 and 5), better perceived productivity (4), its aesthetic appeal (6), and providing easier task specification (5 and 7). Interestingly, one participant stated that the EI provided better touch input, but the MTI had more natural touch input. The participant clarified that the response was due to some sluggishness experienced within the interface when making very quick zoom and pan gestures in the MTI. This sluggishness is attributed to the hardware limitations of the tablet PC used to conduct the evaluations. It is expected that with better hardware, the issue will be resolved. Two of the ten participants felt that entering waypoints using the EI was easier. This result reveals an important consideration not present in the MTI. Currently users are unable to create a single waypoint by tapping on the screen in the MTI. Instead a user must drag his/her finger slightly for a single waypoint to be created. Further investigation will determine whether or not single tap waypoint addition needs to be added to the MTI in the future. Table 5: The results of participants' preferences between tested interfaces

Comparison Number Total Responses Participants Preferring B

1 10 100%

2 10 90%

3 10 100%

4 10 100%

5 10 80%

6 10 100%

7 10 100%

Subjective Workload  The NASA Task Load Index (TLX) was used to measure subjective workload. During the evaluation, NASA TLX information from three participants was lost due to problems with the third-party NASA TLX data collection software. The NASA TLX provides an overall score as well as subcategory scores. The mean and standard deviation results for the subcategories and the overall score are presented Figure 19. A lower score implies a better result, such as lower frustrations levels. The reduced data set results suggest that the overall workload is significantly less with the MTI.

28 

Another factor to consider is the meaning of the various subcategories. The NASA TLX is designed to be used on a large range of interfaces. Therefore, not all of the subcategories are applicable to our evaluation. For example, there was little physical demand on the participants, as they were required to perform very small finger and hand movements. Likewise, the scenario placed no temporal requirements on the participants. Participants were not told that they were being timed and were given no time restriction with which to perform the scenario.

Task Load Index

60

55.4

50

41.1

40 EI

30

23.4

MTI

19.2

17.7

20 7.9 7.1

10 2.7 3.8

3.9

Mental

Physical

4.1 1.6

3.3

2.0

0 Temprol

Performance

Effort

Frustration Overall Score

Figure 19: The NASA TLX results for each interface by factor and overall result

The NASA TLX data was analyzed using a t-test in the same manner as the postscenario completion survey. This provides a subjective analysis to aid in determining the interface that requires lower workload. Therefore, the null hypothesis for a particular question stated that the EI would score better (lower) out of the entire population of users (i.e., μΔ < 0, where μΔ is the mean of the deltas for the population). Table 6 shows the results of the statistical comparison between the two interfaces. The overall NASA TLX result can be used to reject the null hypothesis with a significance level less than 0.05, suggesting the MTI requires less workload. Of particular significance was the frustration component result. The null hypothesis for frustration can be rejected at significance levels less than 0.005. This feedback is consistent with the verbal feedback the in which participants frequently made unprompted comments while performing tasks on the EI stating their frustration with the interface. The remaining components were not statistically significant, though they seem to be in favor of the MTI.

29 

Table 6: The evaluation of NASA TLX comparison hypothesis tests

  Test Stat. (t)  Sig. Level (p) 

Mental  1.46  0.10 

Physical ‐0.84  0.22 

Temp. ‐0.44 0.34

Perform. 1.28  0.12 

Effort ‐0.24 0.41

Frust.  ‐4.79  0.0015 

Overall 1.46 0.011

Communication Layer Test Results  During the MTI scenario the participants were given a series of verbal commands using the Communications widget as a visual aid and reminder. After the scenario, the participants were asked to repeat the instructions back to the proctor. The intent was to determine if the participants would utilize the stored information in the Communications widget to help recall the instructions. 70% of the particpants opened the Communication widget and refered to it as they repeated the instructions. All participants repeated the instructions correctly. Since the Communications widget does not exist in the EI, no direct comparison can be made using the Communications widget that supports the original hypothesis. However, these results do indicate that the Communications widget may be a useful feature. More investigation into this issue is required before a definitive claim concerning the widget’s usefulness can be made.

Discussion  The purpose of this user evaluation was to determine the validity of the major hypothesis that the touch additions and interface modifications to the HuRT interface increase the overall usability of the system. To determine the validity of the major hypothesis the following questions are addressed: 1. Do users perform the scenario faster when using the MTI versus the EI? 2. Do users make fewer errors when using the MTI versus EI? 3. Which interface do users qualitatively prefer? 4. Do the multi-touch additions to the MTI prove to be helpful to the user? 5. Does using the MTI result in less overall workload than using the EI? Each of these questions requires analysis of the data gathered from the administered surveys as well as the quantitative timing data gathered for each interface. The timing data shown in Figure 15 and Figure 17 indicate that the MTI allowed participants to perform the scenario tasks faster. From Figure 15, on average, a 25% decrease in the completion time of any individual task was shown to occur with the MTI. The results shown in Figure 17 also indicate that drawing a task was faster than specifying individual waypoints. Both of these findings support the claim that the MTI allows a user to perform tasks faster than with the EI. Figure 14 also supports this claim by showing that, on average, participants completed the overall scenario 1 minute and 48 seconds faster using the MTI. While the timing data seems to conclusively support the hypothesis, it was difficult to determine if a user is less likely to make errors when using the MTI. The Data presented in Table 2 seems to support this claim by showing that participants made fewer errors using the MTI; however, the data can be misleading. For example, there is a large disparity between the number of unfinished tasks reported in the MTI versus the EI. From observation, it is believed that these unfinished tasks were mainly due to difficulties selecting the appropriate task in the EI, rather than problems stemming from actual task 30 

entry. The same is true for cancelling a task. Since there is no inherent benefit to cancelling a task versus clearing it with the EI (i.e., no more of the map is revealed, no additional information is shown on the screen, etc.), participants often cleared a task that was incorrectly entered rather than cancelling it. In the MTI, the opposite was true. Cancelling a task in the MTI causes more of the map to be visible, so if the participant desired more screen space for the map, he or she may have opted to cancel a task rather than move the Task Specification window to another location on the screen. Factors such as these make the reported error data ambiguous. While the reported data can provide some indication as to whether or not fewer errors are committed while using the MTI, without knowing the participants’ intent or the state of the interface at the time an error was committed, it is difficult to determine what exactly constitutes an error. For this reason, more development is required to improve the methods of error tracking in each interface before a conclusive statement concerning error rates for either interface can be made. Regardless of mistakes, however, participants still accomplished task creation and the overall scenario faster using the MTI, which leads the authors to believe that fewer true errors occur while using the MTI interface. Qualitatively, participants prefer the MTI. This preference can be seen by examining the results shown in Table 5. The data in Table 5 reveals that participants almost unanimously prefer the MTI to the EI in every category. While discrepancies exist in the Better Touch Input and Easier Waypoint Entry categories, this fact is overshadowed by the 100% overall preference rate for the MTI. With this in mind, along with the fact that in nearly every other category participants favor the MTI 100% over the EI; the MTI is undoubtedly preferred among the participants. To support the main hypothesis, it is important to determine whether or not participants found the multi-touch additions to the interface helpful. Qualitatively this appears to be the case, as 90% and 100% of participants found the MTI to possess better touch input and more natural touch input respectively. Data specifically showing the importance of the multi-touch gestures can be seen in

31 

Table 4. This table shows the participants’ quantitative scores for each interface from the post-scenario completion survey. Participants indicated that the MTI provided better zooming functionality than the EI. It was determined that panning was not statistically significant, which could be due to the similarities between panning in each interface. Also, the data shows that the two-finger panning gesture is not a deterrent to the MTI’s usability. From the NASA TLX data shown in Table 6, it can be seen that frustration levels are much higher when using the EI. This table also shows that overall subjective workload is lower when using the MTI. However, these results may be skewed by the fact that three of the ten NASA TLX data sets were lost due to a software malfunction. Without these three data sets, it is hard to determine whether or not the represented set is an accurate representation of the actual frustration and workload levels. It has been determined that the MTI does result in less overall workload at a significance level of 0.05.

Conclusion  Based on the results of the users study, a strong argument can be made that supports the hypothesis that the overall usability MTI is an improvement over the EI. Timing data revealed that participants perform individual tasks and full scenarios faster using the MTI. It is also evident from the qualitative survey data that participants prefer the MTI to the EI. It has also been shown that the multi-touch additions to the MTI prove to be, in the case of the zoom gesture, helpful to the participants. The pan gesture appears, at the very least, to not hinder the participants, but determining if this gesture was actually beneficial proved to be inconclusive. From the gathered data, it was difficult to determine if participants committed fewer errors in the MTI versus the EI. In order to determine if this is the case, errortracking methods will require improvement within each interface. Also, due to an unavoidable loss of data, the NASA TLX results proved to be somewhat inconclusive in determining whether or not workload was reduced using the MTI. On the whole, the addition of multi-touch functionality and interface modifications may have improved the overall system. This belief is supported by the faster performance in the MTI, the perceived benefit of the multi-touch gestures, and the fact that participants seem to prefer the MTI over the EI. Based on these facts, it is believed that with further work it will also be shown that participants commit less errors   and experience fewer workload when using the MTI.

32 

References  Apple - iPhone - Learn about high-tech features like Multi-Touch. (n.d.). . Retrieved November 23, 2009, from http://www.apple.com/iphone/iphone-3gs/hightechnology.html Human Machine Teaming Lab | ResearchProjects / KSU Hu RT Project. (2009, August 29). Human Machine Teaming Lab. Wiki, . Retrieved September 3, 2009, from http://eecs.vanderbilt.edu/research/hmtl/wiki/pmwiki.php?n=ResearchProjects.KS UHuRTProject Micire, M., Drury, J. L., Keyes, B., & Yanco, H. A. (2009). Multi-touch interaction for robot control. In Proceedings of the 13th international conference on Intelligent user interfaces (pp. 425-428). Sanibel Island, Florida, USA: ACM. doi: 10.1145/1502650.1502712 Westerman, W. (1999). Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface. University of Delaware. Retrieved from Google Scholar.

33