Stereoscopic 3D displays and human performance

Displays 35 (2014) 18–26

Contents lists available at ScienceDirect

Displays journal homepage: www.elsevier.com/locate/displa

Review

Stereoscopic 3D displays and human performance: A comprehensive review John P. McIntire ⇑, Paul R. Havig, Eric E. Geiselman 711th Human Performance Wing/RHCV, US Air Force Research Laboratory, Dayton, OH, USA

a r t i c l e

i n f o

Article history: Received 4 December 2012 Received in revised form 7 August 2013 Accepted 28 October 2013 Available online 7 November 2013 Keywords: Stereopsis Three-dimensional display Human factors Depth perception Binocular vision S3D

a b s t r a c t To answer the question: ‘‘what is 3D good for?’’ we reviewed the body of literature concerning the performance implications of stereoscopic 3D (S3D) displays versus non-stereo (2D or monoscopic) displays. We summarized results of over 160 publications describing over 180 experiments spanning 51 years of research in various fields including human factors psychology/engineering, human–computer interaction, vision science, visualization, and medicine. Publications were included if they described at least one task with a performance-based experimental evaluation of an S3D display versus a non-stereo display under comparable viewing conditions. We classified each study according to the experimental task(s) of primary interest: (a) judgments of positions and/or distances; (b) finding, identifying, or classifying objects; (c) spatial manipulations of real or virtual objects; (d) navigation; (e) spatial understanding, memory, or recall and (f) learning, training, or planning. We found that S3D display viewing improved performance over traditional non-stereo (2D) displays in 60% of the reported experiments. In 15% of the experiments, S3D either showed a marginal benefit or the results were mixed or unclear. In 25% of experiments, S3D displays offered no benefit over non-stereo 2D viewing (and in some rare cases, harmed performance). From this review, stereoscopic 3D displays were found to be most useful for tasks involving the manipulation of objects and for finding/identifying/classifying objects or imagery. We examine instances where S3D did not support superior task performance. We discuss the implications of our findings with regard to various fields of research concerning stereoscopic displays within the context of the investigated tasks. Published by Elsevier B.V.

Contents 1. 2. 3.

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. Overall results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. By performance task-type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1. Judgments of position and/or distances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2. Finding/identifying/classifying objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3. Spatial manipulations of real or virtual objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4. Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5. Spatial understanding/memory/recall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.6. Learning/training/planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. Human factors/human–computer Interaction vs. medical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. Discussion, conclusions, and recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A. Supplementary material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

⇑ Corresponding author. Tel.: +1 937 255 0589. E-mail address: [email protected] (J.P. McIntire). 0141-9382/$ - see front matter Published by Elsevier B.V. http://dx.doi.org/10.1016/j.displa.2013.10.004

19 20 21 21 21 21 21 22 22 22 22 23 23 25 25

J.P. McIntire et al. / Displays 35 (2014) 18–26

1. Introduction Visual depth perception involves the mental combination of various monocular and binocular visual cues to determine the 3D shapes of objects in the world, their spatial arrangements relative to each other, and their locations relative to oneself. Monocular cues, also known as pictorial or perspective cues, are those that require only one eye to perceive depth, and are easily seen even in two-dimensional representations such as paintings, photographs, or videos. Examples of monocular depth cues include relative size, occlusion, shading, spatial frequency of textures, motion parallax, etc. (e.g., [9]). Unfortunately, if any of these monocular cues to depth are degraded, ambiguous, or are altogether absent in imagery when shown via traditional two-dimensional display surfaces, then depth perception and performance may suffer. This issue may partly explain why there is such a growing interest in display technologies that present not just monocular but also binocular cues to object depth (via binocular parallax/disparity cues), so that a potentially more accurate, natural, and robust depth percept can be provided to viewers. Stereoscopic 3D displays (S3D) are a technology in which high resolution color imagery can allow both monocular and binocular depth cues to coexist in a single display system, with relatively little extra cost in terms of software, hardware, or imagery requirements. S3D displays are systems in which two slightly different views of a scene are provided to a viewer, one image for each eye (the combination of these ‘‘half-images’’ is usually called a ‘‘stereo pair’’). This allows the viewer’s binocular visual system to extract depth information in a scene using this disparate information (stereopsis). A variety of clever display engineering methods have been invented so that each eye receives one and only one image from a stereo pair, including mirrors, optics, eye-glasses with shutters (active), polarizing filters (passive), color filters (anaglyph), and lenticular screens and parallax barriers (autostereoscopic displays). Although S3D display technology has recently garnered wide interest and application in entertainment, medicine, industrial design, education, research, and in the military, this technology push into widespread use has occurred mostly in the absence of considering the task performance implications of these new displays, or when their use might be effectively neutral or even negative. Our main research interest in S3D displays is specifically in terms of performance. There are many tasks, which may span many work domains, that could conceivably benefit from S3D technology. For example, spatial understanding of a complex scene (e.g., surgery, imagery analysis, or route planning), tasks that require manipulation of virtual objects (e.g., CAD drawing or teleoperation of a robotic arm), or tasks involving navigation (e.g., teleoperation of a robotic vehicle) are all spatial tasks that seem logical candidates for S3D technology, among many others. Visual display researchers, designers, and users could benefit tremendously from a clear understanding of where and when S3D technology actually increases user performance, as opposed to only increasing user interest due to the novelty of the technology, or simply increasing the subjective impression of realism devoid of subsequent performance gains (naïve realism; see [64]). In a previous, more limited review of the human factors literature [40], we examined this question as applied to S3D displays: ‘‘What is 3D good for?’’ Theoretically, given that primates have largely-overlapping visual fields affording stereoscopic vision, whereas much of the animal kingdom does not, there must have been notable advantages in terms of survival for this capability to have evolved. We therefore expected to find some clear performance advantages for stereoscopic displays. In particular, for tasks that are close-up (in the near-field, within a few meters of a viewer’s personal space) where binocular disparity cues are relatively

19

larger, and for tasks that are difficult due to poor-quality or absent monocular depth cues. Indeed, in the human factors literature, we previously confirmed S3D displays were most helpful for depth-related near tasks like spatial manipulations of objects, especially if these were difficult/complex tasks. For example, think of the difficulty and complexity (and the precision of spatial vision needed) for threading a needle, conducting surgery, explosive ordinance disposal, understanding a complex twisted wire shape, spotting objects in camouflage, etc. It becomes apparent why the additional provision of any depth cues, especially binocular depth cues, can aid performance. Two of the three most important or ‘‘heavilyweighted’’ depth cues, at least for near-field vision in everyday life, are (self) motion parallax and binocular parallax (e.g., [9]), both of which are incidentally missing from the vast majority of modern display devices. An excellent discussion of perceptual depth cue combination, and its measurement and modeling, is available in Landy et al. [36]. In 58% of the studies in our previous review, S3D showed a definite benefit over non-stereo displays (2D), which we found to be a surprisingly large number based on the often conflicting research results spread throughout various domains. S3D seemed to promote increased spatial understanding of complex/ambiguous scenes: 77% of spatial appreciation studies demonstrated a benefit of S3D use. For spatial manipulations of real or virtual objects, 67% of studies showed a definite benefit of S3D. Stereoscopic 3D displays also seemed beneficial for tasks involving judging distances, discerning relative positions, finding and identifying objects, and navigation – all of which showed a benefit for S3D in approximately 50% of the studies involving these tasks. Alternatively, we also noted some important instances where S3D failed to support better performance. S3D helped little or not at all for tasks that were simple or well-learned, or, understandably, for tasks that did not rely heavily on depth information. For example, experts generally receive less benefit from S3D than novices, and increasing the difficulty of a task can sometimes reveal a benefit of S3D where there was not one before; both are issues to which we will return in later sections. S3D also helped little or not at all where other (monocular) depth cues were strong, or for tasks in which depth information was far away or otherwise lay outside the effective viewing volume (i.e., binocular disparity cues were small or absent). These are reasonable explanations for why 28% of the 71 human factors studies reviewed found no benefit of S3D over traditional non-stereo 2D displays for various performance tasks. We have now completed a more comprehensive review of all research fields that have experimentally investigated non-stereo versus S3D displays in order to determine those tasks best supported by S3D technology. Here, we focus mainly on the performance implications for users. There already exist several comprehensive experiments and reviews comparing performance with ‘‘2D versus 3D’’ displays (e.g., [3,21,75,65]; see reviews by [46,12]), but nearly all these works utilized flat 2D displays to show images of 3D perspective geometry (sometimes called ‘‘2½D’’ or ‘‘perspective 3D’’). Essentially, these previous reviews compared non-stereo displays of 2D data versus non-stereo displays of 3D data, but in neither case was the binocular visual system required to properly appreciate the stimuli (all depth cues were monocular). Instead, we focused our review on studies in which binocular vision was explicitly required, i.e., for viewing stereoscopic stimuli. In other words, we compare stereoscopic 3D displays of 3D data to traditional non-stereo (2D) displays of 3D data. Due to possible confusion these terms can cause, we will use the terms ‘‘stereoscopic 3D’’ or ‘‘S3D’’ to refer to the stereo display hardware used in the reviewed studies, as this terminology is becoming the dominant preference within the research community (e.g., see recent issues of the Proceedings of the IS&T/SPIE

20

J.P. McIntire et al. / Displays 35 (2014) 18–26

Stereoscopic Displays and Applications annual conference). Alternatively, ‘‘non-stereo displays’’ or sometimes ‘‘2D’’ will be used to refer to the non-stereo two-dimensional display hardware on which only monocular/pictorial/perspective cues to depth were visible in the baseline comparison cases. There is a vast literature on the topic of S3D displays and their implications for perception, cognition, and experience of quality and comfort (e.g., [18,53,51,52]; for the important topic of viewer discomfort with S3D displays, the interested reader can see: [71,58,55,56,25,28,74,54,2,26,35,44,67,68]). Additionally, there is an experimental psychology body of literature comparing monocular vision (using only one-eye) versus binocular vision (using both eyes) during real-world tasks, which involve no displays whatsoever and serves to demonstrate the importance of binocular cues for visual tasks in everyday life, and which are strongly suggestive of possible areas of benefit for similar tasks conducted via S3D displays. For instance, studies of exteroceptive, visuomotor task performance show the benefits of binocular vision for real-world reaching, grasping, and manual control (prehension) tasks (e.g., [62,73,5,39,42,43,29,49]). And binocular vision is also beneficial for ground navigation/locomotion tasks—where it has been demonstrated that walking over/around obstacles or over uneven terrain is hampered with strictly monocular viewing (e.g., [22,50]). Despite the apparent benefits of stereoscopic vision for humans in the real-world, the benefits provided by S3D display sources are less obvious. This might be partly due to the fact that what matters most to the viewer’s visual system in terms of binocular depth cues is the magnitude of binocular disparity (the difference in visual angle between the two eyes of a real or virtual object and the viewer’s point of fixation), as opposed to only considering the camera separations, the viewing distances, the on-screen half-image separations of stereo pairs, or other isolated metrics that contribute only indirectly to the perception of depth on S3D systems. The binocular disparity that a viewer ultimately experiences via an S3D display is a function of various features in both the camera/scene space (camera separation, distances between camera and real or virtual objects in a scene, camera fields-of-view, camera alignment, etc.) and the viewer/display space (viewer eye separation, viewer distance to display surface, size and resolution of the displays, etc.). These points are explained in detail by Jones et al. [30], Woods et al. [79], Williams and Parrish [78], Ware et al. [72], Grinberg et al. [20], Kim et al. [32], among others. Unfortunately, binocular disparity magnitudes have been relatively poorly documented in comparative performance studies using S3D (more on this later), which we believe has contributed substantially to confusion about the extent to which S3D displays may improve performance. Another reason that the benefits of S3D displays may be less obvious is the lack of modern representative summaries on the relationship between S3D displays and performance (reviews of performance with stereoscopic displays in the medical domain are given by Hofmeister et al. [24], Getty and Green [19], van Beurden et al. [69], Held and Hui [23]. Some limited reviews on this topic in the human factors literature include factors affecting human performance in teleoperated robotics by Chen et al. [7], a brief review of several S3D studies in Naikar [46], and a brief review in Boff [4]. We believe a comprehensive review of experimental studies comparing non-stereo (2D) to S3D displays is warranted at this time. This is especially true given the present widespread interest in S3D by both the research community as well as the general public.

2. Methods A total of 162 publications describing 184 experiments were included in this review. Some publications contained multiple experiments. Inclusion criteria required that each experiment include at

least one objective task performance measure comparing non-stereo (2D) displays versus S3D displays. According to our definition, an objective measure of performance would be a measure such as reaction time (i.e., time to initially respond to a stimulus), task completion time, spatial placement accuracy, percent correct, or some similar performance metric. In order to afford meaningful comparisons, it was also required that the non-stereo and S3D conditions have closely comparable experimental conditions on some depth-related spatial task(s). Identification processes included standard literature review methods, extensive online searching, and relevant article citation tracking. We searched for relevant terms including ‘‘stereoscopic displays’’ and/or ‘‘3D’’ and/or ‘‘performance,’’ including databases of Google Scholar, SPIE Digital Library, IEEE Explore; and journals including Displays, Human Factors, etc. A major portion of articles found within the human factors/HCI-domain came from IEEE and SPIE conference proceedings. In the medical domain, a large number came from the journal Surgical Endoscopy. Further detailed information is available in Supplementary data table. About 5% of the targeted publications were not discoverable in either electronic or print format. Therefore, these works had to be excluded from our review. The 162 studies were divided among the three authors and read in detail to extract pertinent information. Subsequently, each of the 184 identified experiments were classified into one of six different categories depending upon what type of performance was the primary interest being investigated: (1) judgments of position and/or distances, (2) finding/identifying/classifying objects, (3) real/virtual spatial manipulations of objects, (4) navigation, (5) spatial understanding/memory/recall, and (6) learning/training/ planning. Of course, it may be argued that this classification taxonomy is somewhat artificially contrived–a task such as navigation also requires judgments of distance and relative object positioning. Likewise, manipulating objects generally requires good spatial understanding of the scene. However, these classifications were evident from the experimental tasks chosen and described by investigators in the reviewed works. These distinctions, while perhaps not perfect, are potentially useful and warrant consideration. Again, it should be noted that some publications contained multiple experiments. These were only considered independent if the experiments fell in different task categories. We also classified the 184 experiments into one of three categories of outcome, dependent on their primary findings:  ‘‘S3D is better’’ – performance with S3D was found to be superior to non-stereo (2D) for the main performance measurements of interest, with conventional statistical significance. For instance, many studies simply focused on accuracy as the most important measure. Since, in the comparable real-world applications (such as in surgery or bomb disposal), accuracy is often the primary measure of interest. There were also many studies that measured both response times and accuracy, and judged both as important, but showed a statistically reliable benefit of S3D on only one measure. When no measure was deemed as the primary measure of interest, we considered such a finding to demonstrate superior overall performance of S3D over non-stereo, since participants did not simply trade-off speed for accuracy (or vice versa) in the S3D condition and ‘‘overall’’ performance was improved.  ‘‘Mixed’’ – many papers showed ‘‘mixed’’ results in which S3D performance was better only on a minority of measures, and/ or on measures that were not of primary interest to the task. For instance, often response times or accuracy were portrayed as most important but were not found to be significantly different across the displays, while secondary peripheral measures like total distance moved, or strength of a knot tied by a telemanipulator, may have been statistically different.

21

J.P. McIntire et al. / Displays 35 (2014) 18–26

 ‘‘NS = S3D’’ – S3D performance was found to be not statistically different than performance with non-stereo (2D) displays on all measures of primary interest, or nearly-all measures in the cases where a large number of measures were collected within a single experiment. In some rare instances (as will be discussed), S3D actually proved detrimental to performance, but such instances were still classified into this category because there were few, and we would generally not expect this effect. We admit this categorization scheme is a bit more arbitrary than ideal, but the large variety of tasks that were investigated and the various ways in which ‘‘performance’’ was conceptualized, measured, and analyzed makes perfect categorization impossible. A few studies had dozens of different performance metrics, some more important than others, in which conflicting results were found, and thus were difficult to classify. Despite this occasional difficulty, our overall objective was to provide the clearest idea possible of what topics have been studied comparing non-stereo (2D) to S3D, what the previous research has generally found, and where to go for more information. Additionally, to aid researchers across different domains, we have also coded each study as relating primarily to medical research versus general human factors/human–computer-interaction (HF/HCI) research. Our full dataset for this analysis is published as a Microsoft Excel table, available as supplementary data (see Appendix A). Alternatively, the dataset will be available upon request of the first author at: [email protected].

3. Results 3.1. Overall results The primary findings are presented in Tables 1 and 2. The most striking finding is that of the 184 experiments, 60% showed a definite performance benefit for the use of S3D displays over non-stereo (2D) displays, while 15% showed a possible though unclear or mixed benefit of S3D. Only 25% of the studies clearly showed no benefit of S3D (and in some rare instances, S3D was worse than non-stereo). These results are an interesting outcome given that many authors cite the scattered and conflicting experimental outcomes in the literature as being mostly mixed, and as a primary reason for their performing an experimental investigation of nonstereo versus stereo 3D.

3.2. By performance task-type Across the 184 experiments, almost half (45%) measured performance by asking participants to spatially manipulate real or virtual objects via manual computer interaction. No other task category has been as well-studied. Two areas where experimental S3D studies are under represented include navigation (accounting for only 7% of experiments) and learning, training, and/or planning (accounting for only 6% of experiments). These are possible fruitful areas for future work. Stereoscopic 3D displays were most beneficial for spatial manipulation tasks and tasks requiring finding, identifying, and/or classifying objects in imagery. The S3D displays were least beneficial for learning, training, and/or planning tasks, and navigation tasks. Next, we discuss results within each task category. This discussion is focused on instances where S3D did not appear to aid performance. We focused our discussion on failures of S3D for pedagogical and research purposes, in order to highlight variables, issues, and concerns of when S3D displays do not provide a performance benefit, even if one might otherwise be expected. 3.2.1. Judgments of position and/or distances Of the 28 experiments testing position judgments and/or distances of displayed objects, 57% showed a clear S3D benefit, 14% showed mixed results, and 29% showed no difference between non-stereo and S3D. Obviously, S3D did not always provide a clear benefit over non-stereo for judgments of positions. In a series of experiments using an airspace disambiguation task, Reising and Mazur [59] found S3D to be beneficial only when other monocular depth cues were absent. For similar reasons, Ntuen et al. [48] found no benefit of S3D for a virtual object depth judgment experiment. Willemsen et al. [77] found distance judgments in a virtual environment to be comparable over the non-stereo and S3D conditions. Again, it seems that the presence of other (monocular) depth cues provided sufficient performance during non-stereo conditions. This resulted in many instances of mixed or absent benefits of stereoscopic 3D. 3.2.2. Finding/identifying/classifying objects Of the 26 experiments testing visual search, identification, or visual classification of displayed objects or imagery, 65% showed a clear benefit of S3D, only 8% showed mixed results, and 27% showed no difference. Thus, there were only 34% of these experiments where S3D showed a mixed benefit, no benefit, or a

Table 1 Overall summary results (frequencies of experimental results across task categories). Tasks/results

Judgments of position and/or distances

Finding/ identifying/ classifying objects

Real/virtual spatial manipulations of objects

Navigation

Spatial understanding, memory, recall

Learning/ training/ planning

Totals

Percentages (%)

S3D is better Mixed NS = S3D Totals

16 4 8 28

17 2 7 26

55 12 15 82

5 0 7 12

13 6 6 25

4 4 3 11

110 28 46 184

60 15 25 100

Table 2 Overall summary results (percentages of experimental results across task categories). Tasks/results

Judgments of position and/or distances (%)

Finding/ identifying/ classifying objects (%)

Real/virtual spatial manipulations of objects (%)

Navigation (%)

Spatial understanding, memory, recall (%)

Learning/training/ planning (%)

S3D is better Mixed NS = S3D Totals

57 14 29 100

65 8 27 100

67 15 18 100

42 0 58 100

52 24 24 100

36 36 27 100

22

J.P. McIntire et al. / Displays 35 (2014) 18–26

detriment. A basic psychophysical study of S3D motion perception by McKee et al. [41] suggested that S3D was useful for the detection of static targets within clutter (by eliciting perceptual ‘popout’ of the target) but was little help for the detection of straight-moving targets among random-motion distractors. The relatively slow temporal response of the stereo perception system was suggested as a likely cause in this case. Peinsipp-Byma et al. [57] tested non-stereo versus S3D for a variety of image analysis tasks and found that, while S3D seemed to improve mean detection, recognition, and classification times, these differences were not statistically reliable. Drasic and Grodski [15] also found that non-stereo (2D) was comparable to S3D on a teleoperation IED detection task (probably because sufficient monocular cues were present). Zeidner et al. [80] tested military image analysts in a between-subjects design, and found no difference between the nonstereo and the S3D groups for visual search and identification tasks (in terms of the quality of the information provided by analysts or the confidence assigned to each response). In terms of the number of objects reported in the images, the non-stereo group tended to identify more objects than the S3D group, though it is not clear if this trend was statistically reliable. Steiner and Dotson [66] found the S3D condition to be worse than the non-stereo condition for a visual search task involving the portrayal of tactical aviation information. This was true even though participants reported a preference for the S3D display format. 3.2.3. Spatial manipulations of real or virtual objects Of the large number of experiments (over 80) where spatial manipulations with S3D were investigated, 67% indicated a clear positive performance benefit for S3D over non-stereo. Only 15% showed mixed results, and 18% found no benefit for the use of S3D displays. This is a well-studied task category with remarkably consistent results suggesting that stereo 3D displays can be very beneficial. However, not all manipulation experiments showed a definite and clear benefit for using S3D over non-stereo. Kim et al. [33] found S3D to generally improve telemanipulator performance, but also found that providing clear or enhanced monocular depth cues could result in comparably enhanced performance. On a teleoperation tapping task, Draper et al. [13] showed that S3D was slightly better than non-stereo only under the most difficult task conditions. During less challenging conditions, S3D was not beneficial. With similar ‘‘task load’’ explanations, several other studies found comparable results: S3D provided little or no benefit over non-stereo (2D) displays. 3.2.4. Navigation Of the 12 experiments involving navigation with S3D displays, five (42%) found a clear positive benefit for the use of stereo. Seven experiments (58%) suggested no benefit of S3D over non-stereo. There were no mixed results for this task category. On two simulated remotely piloted aircraft flight tasks, de Vries and Padmos [10,11] found that S3D offered no benefit for steering accuracy or for other flight performance metrics (e.g., speed error, routematching, etc.). The authors speculated that the useful information for completing the task (virtual distance of far objects or waypoints) was simply outside the effective viewing volume of the S3D display. Citing similar reasons, Singer et al. [63] found no beneficial effect of S3D displays for a virtual room/gate ground navigation task. Also, Reising and Mazur [59] found no benefit of S3D during a pathway-in-the-sky virtual aircraft navigation task. Thus, we see that S3D can occasionally be beneficial for navigation tasks, and sometimes not. There may be a dependency on the relationship between the task requirements and the display configurations. While this task area seems understudied, the few experimental results do tentatively suggest that S3D displays may not be especially helpful for this task category. Perhaps the

simplest explanation is that navigation tasks explicitly require self-motion through some environment, which then typically provides ample motion parallax depth cues–thereby diluting the benefits provided by binocular parallax cues via S3D displays. 3.2.5. Spatial understanding/memory/recall Of the 25 experiments which investigated spatial understanding, memory, and/or recall, S3D displays proved to be beneficial in over half of the experiments (52%). Mixed results were found for 24% of these studies and null results for another 24%. This category typically measured performance for tasks like understanding complex spatial figures or objects. Lee et al. [37] found that stereo helped on one data interpretation task (improved accuracy and response times in reading 3D scatterplot data), but S3D did not help on another task (understanding 3D block diagrams of semi-discrete data, in comparison to a familiar tabular presentation of the same data). In this case, it is possible that users’ familiarity with reading the data in table form suppressed any benefit that might have been provided by the S3D presentation of the block data. Brown and Gallimore [6] tested participants on 3D-CAD object understanding tasks and showed that S3D was sometimes beneficial–but only when other monocular depth cues were degraded or absent. The one study that found no benefit of S3D for spatial understanding was performed by these same researchers [17]. Previously they found that during discrimination between two 3D CAD models, disparity was apparently not needed for performing the task at high-performance levels because adequate monocular cues were present. 3.2.6. Learning/training/planning Of the 11 experiments in this task category, only 36% showed beneficial effects of utilizing S3D displays. Similarly, 36% showed mixed results and 27% showed no benefit of S3D over non-stereo. Drasic [14] found that during a teleoperation bomb disposal task, the benefit of S3D decayed over time as participants learned how to complete the task effectively using only monocular cues while also gaining familiarity in controlling the telemanipulator. When the task difficulty was increased, S3D continued to aid performance and learning rate. Neubauer et al. [47] found no positive effect of S3D on training outcome. They studied mental rotation ability both across time (to assess training) and dimensionality (non-stereo versus 3D display). Here, S3D improved reaction times, and performance improved over time (training effect). But there was no interaction between training and S3D (S3D helped, but not differentially across time). The dearth of data in this area calls for further research. However, the results to date suggest little benefit regarding the use of S3D displays for this task category. It may be helpful to point out how different researchers operationalized the terms ‘‘learning,’’ ‘‘training,’’ and ‘‘planning’’ within their studies. Most studies which involved S3D ‘‘learning’’ assessed a performance change over time (via time-series, or comparing first versus last trials, or before/after sets of trials). Many studies used the terms ‘‘learning’’ and ‘‘training’’ interchangeably. Planning (particularly tasks such as surgical planning) is similar to learning or training in the sense that it is preparation (mental and sometimes physical) for what is often a stressful, one-time, high-risk opportunity for which the outcome is critical. Learning, training, and planning are all similar in this regard (preparation for a performance outcome, which is what this review was focused on). As an example for how ‘‘planning’’ was operationally defined in the literature. Litynski et al. [38] studied group planning for a tactical mission, and measured the time spent on evaluation and the number of alternative plans generated. Formal ‘‘training’’ and training transfer were studied somewhat differently. Mourant and Parsi [45] studied the effect of different training regimens (non-stereo, S3D, and real-world environments)

23

J.P. McIntire et al. / Displays 35 (2014) 18–26

on a post-training real-world performance task (i.e., ‘‘training transfer’’). Somewhat differently, Neubauer et al. [47] assessed training by giving a pre-test, then an extended (two week) training regimen for mental rotation tasks, followed by administering a post-test. Using a test-retest learning evaluation paradigm, Votanopoulos et al. [70] compared performance on a test-retest evaluation in which one group received non-stereo first, then either a non-stereo or S3D test; while another group received S3D first, then either a non-stereo or S3D test. Despite the wide variety of ways in which researchers defined and tested training, learning, and planning, we categorized them together due to their conceptual and operational similarities. 3.3. Human factors/human–computer Interaction vs. medical experiments Of the 184 experiments, 128 (or 70%) were classified into the general human factors/human–computer-interaction (HF/HCI) research category. The other 56 experiments (30%) were classified as being primarily medically-focused research. Interestingly, despite the smaller number of medically-related stereo 3D efforts, this literature appears to more strongly support the use of S3D displays. In the medical literature, 70% of the experiments suggest that S3D is clearly superior to non-stereo (2D). Only 7% show mixed results and 23% show null results. Contrast these findings with the HF/HCI literature, in which 55% of experiments showed that S3D was clearly helpful, 19% indicated mixed results, and 26% showed null results. The HF/HCI literature strongly supports the use of S3D displays for spatial manipulation tasks (69% of experiments found S3D to be more beneficial than non-stereo), but fails to support S3D for learning, training, or planning tasks. For learning, training, or planning, only one study showed positive results for S3D, while four studies were mixed and another two studies showed no effect. However, across the remaining task categories, S3D displays were relatively well-studied, with each task category (except navigation) having at least 12 experiments. See results presented in Table 3. The medical literature also strongly supports the use of S3D for spatial manipulation tasks such as teleoperation surgical procedures (65% of experiments in the spatial manipulation category show S3D is clearly better than non-stereo). This task category was the most thoroughly represented in the medical literature (61% of the experiments). Navigation via S3D has not been studied by the medical community. Only finding/identifying/classifying objects as a task category had more than four publications testing

S3D. The medical community is obviously interested in spatial manipulations (for surgical procedures) and for finding/identifying/classifying imagery (e.g., for radiological diagnoses) and understandably less interested in other task categories such as navigation or positional judgments. However, they might find the learning/training/planning and spatial understanding categories to be of future research interest for instructional purposes, surgical planning, and viewing/understanding complex imagery. See results in Table 4. 4. Discussion, conclusions, and recommendations In summary, stereoscopic 3D displays seem to be most beneficial for depth-related tasks performed in the near-field (tasks in close spatial proximity to the viewer, such as spatial manipulations of objects). This is perhaps not surprising, given that in real-world viewing situations, the closer an object is to a viewer, the larger its potential binocular disparity cues, leading to a larger relative benefit of binocular cues over monocular cues when perceiving depth (e.g., [9]). S3D displays were also shown to be especially helpful for difficult/complex tasks and for tasks that are not well-learned; for example, novices often benefit more from S3D than experts at a particular task, and more difficult tasks can sometimes reveal benefits of S3D which did not appear under easier task conditions. In 60% of the 184 experiments reviewed, S3D showed a clear and definite performance benefit over non-stereo (2D) viewing. A mixed or unclear benefit was found in 15% of the experiments and no benefit was indicated in only 25% of the experiments. Given our review data, S3D seems to be useful especially for spatial manipulation of real or virtual objects (67% of manipulation studies showed a clear benefit with S3D), and for finding, identifying, and classifying objects or imagery (65% of experiments in this task category showed a clear benefit with S3D). Stereoscopic 3D displays were also beneficial for tasks involving judgments about object positions or distances, and for tasks involving spatial understanding, memory, or recall (both categories showed a clear benefit for S3D in about 50% of the studies involving these tasks). Despite the instances where S3D enhanced performance, there were notable cases where S3D was of little or no help. In particular, S3D seemed to fail for tasks that were simple or well-learned and for tasks that did not rely heavily on depth information. Also, tasks where other depth cues were strong, or tasks where depth information lay outside the effective viewing volume of the display (resulting in small or absent binocular disparity cues) did not seem to benefit from the use of S3D. We believe that these are all reason-

Table 3 HF/HCI summary results (frequencies of experimental results across task categories). Tasks/ results

Judgments of position and/or distances

Finding/ identifying/ classifying objects

Real/virtual spatial manipulations of objects

Navigation

Spatial understanding, memory, recall

Learning/ training/ planning

Totals

Percentages (%)

S3D is better Mixed NS = S3D Totals

12 4 8 24

9 2 5 16

33 10 5 48

5 0 7 12

11 4 6 21

1 4 2 6

71 24 33 128

55 19 26 100

Table 4 Medical research summary results (frequencies of experimental results across task categories). Tasks/results

Judgments of position and/or distances

Finding/ identifying/ classifying objects

Real/virtual spatial manipulations of objects

Navigation

Spatial understanding, memory, recall

Learning/training/ planning

Totals

Percentages (%)

S3D is better Mixed NS = S3D Totals

4 0 0 4

8 0 2 10

22 2 10 34

0 0 0 0

2 2 0 4

3 0 1 4

39 4 13 56

70 7 23 100

24

J.P. McIntire et al. / Displays 35 (2014) 18–26

able explanations why 25% of the 184 experiments found no benefit for S3D over non-stereo (2D) on various performance tasks. We found that S3D displays were especially helpful for difficult, complex, or unfamiliar depth-related tasks, or for tasks where monocular cues were degraded or absent (these conclusions were consistent with Naikar [46] limited review, as well). Only 13 medical studies were reviewed for objective performance measures at the time by Hofmeister et al. [24], suggesting little work in the area at that time. In comparison to our review, there are now at least 56 experiments in 53 different medical literature publications which evaluated non-stereo (2D) versus S3D. These studies primarily involved telesurgical/robotic and imagery analysis/classification tasks for remote surgery and image assessment applications. In many of the discussed examples where S3D did not help performance, task difficulty and/or complexity plays a factor in the effectiveness of 3D cues: generally, the more difficult the spatial task, the more benefit is derived from having more/better-quality depth cues. This also suggests a possible performance difference between novice and experienced participants, which is generally confirmed in the literature. For instance, Kulshreshth et al. [34] showed that novice video gamers are different than experts in terms of performance and in their derived benefit from S3D. Interestingly, they found that the patterns of S3D benefit seemed to switch depending on the game (or task) type, and according to which monocular cues were available in the scene. Votanopoulos et al. [70] assessed training in novices versus experts, and found that only novices seemed to benefit from S3D training, with the assumption that experts seemed to have already adapted to working with a non-stereo (2D) surgical environment and utilizing the available monocular cues to depth. In fact, in the medical domain, experimental testing is often done using novice versus experienced participant groups (further discussions of this topic can be found in van Beurden et al. [69] and Held and Hui [23]). There are some obvious critiques and criticisms of this work and our subsequent interpretations and conclusions. One might include the possibility that a disproportionate number of S3D display studies that fail to find a positive effect for S3D (e.g., null results) might fail to be published, and are instead thrown into a nearby file drawer never to be seen again (the so-called ‘‘file-drawer’’ effect). Similarly, it is possible that there exists inadvertent experimenter bias in the design of these tests where experimenters may be more likely to make the tasks particularly difficult (perhaps unrealistically difficult) so that a desired S3D effect is found. This would result in an unrepresentative sampling of task difficulty across the literature. We acknowledge these possible biases and can offer no direct remedy for them, except to warn future researchers of these concerns. However, we should point out that file-drawer effects can be partially-alleviated via appropriate statistical corrections. Such corrections are typically more appropriate for metaanalyses with standardized measures (e.g., see [60]), so we did not attempt any file-drawer correction in this review, due to the large variety of operational definitions for ‘‘performance’’ in the experimental comparisons between non-stereo and S3D displays. In conducting this review, we noticed inconsistencies across studies regarding methodologies, descriptions of experimental apparatus, display set-ups, etc. To demonstrate these inconsistencies, we randomly selected about 50 of our reviewed studies and looked to see whether binocular disparity magnitudes were reported (or whether they were calculable based on the reported experimental design description), the display type used, viewing distance to the display surface (or to virtual objects of interest), the type of experimental study design, the primary performance measures, and other incidental information. Unfortunately, only 3 of the 50 studies reported the disparity values in terms of angle, and only 5 more provided enough information that such disparities could be reasonably estimated. Only a few made mention of the

monocular cues visible in their studies or tried to control for them. And only 10 studies specifically mentioned the participants’ viewing distance to the display (although several da Vinci robotic surgery studies and HMD studies complicate this question of what is meant by viewing distance to display surface since optics are used; both the viewer/display space and the camera/scene space are important to characterize for S3D research). In some studies, we could not even determine with certainty which stereo display type was used (shutter, polarized, anaglyph, etc.)! Future S3D display researchers may benefit from considering some of the more common issues involved with conducting experimental research using stereoscopic 3D displays, as discussed by Hsu et al. [27]. Researchers may also benefit from considering the following suggestions in conducting and sharing their work: (1) Clearly report the magnitudes of binocular disparity (preferably in units of visual angle) that are induced or experimentally manipulated in the 3D conditions. Estimate a range if possible. If raw disparity values are not provided, they should at least be easily calculable from given values (e.g., viewer distance to display, viewer distance to virtual objects or regions of importance, on-screen measured disparities, assumed or measured inter-pupillary distances, camera configurations, etc.). See the references cited in the Introduction for instruction on calculating these values/ranges; also see Hsu et al. [27] for an excellent discussion and explanation. Reporting disparity values affords more efficient and accurate cross-study comparisons. Likewise, this knowledge should give a better understanding of what disparity magnitudes might be necessary for optimal performance on particular task categories. (2) Clarify which monocular depth cues are visible within the experimental tasks. Likewise, give some appreciation for the extent to which these cues are present and/or visible. Again, this will allow better cross-study comparisons and should help the research community understand when and under what conditions S3D displays are likely to be beneficial, and when they might be unlikely to help due to the presence of monocular depth cues. (3) Care should be taken to describe the particular experimental display hardware utilized as well as the types of S3D displays that were tested. Information specifying technical features of the displays should be included (i.e. resolution, refresh rate, brightness, contrast, method of stereoscopy, extent of visible cross-talk, etc.). (4) When possible, binocular vision and depth perception capabilities of viewers should be thoroughly examined and reported. And, where appropriate, depth perception sensitivity thresholds should be used as criteria for study participation (clinically normal stereoscopic acuity is usually considered to be on the order of 30–40 arc sec or better; [16]. Some popular, easy-to-administer clinical stereopsis tests include the Randot and the Titmus. (5) Given the importance of viewer comfort on user acceptability, and the implications this may have for performance, future S3D investigators should consider measuring and reporting any presence of discomfort attributable to their experimental manipulations (see the Introduction for citations relevant to viewing discomfort, eyestrain, and/or fatigue induced by S3D displays). The Simulator Sickness Questionnaire [31] is a standardized subjective measurement tool for virtual environment discomfort which includes an oculomotor subscale, and can easily be used for such an assessment. There are also specifically visual discomfort rating scales that might be useful for such purposes, including the Convergence Insufficiency Symptom Survey

J.P. McIntire et al. / Displays 35 (2014) 18–26

(e.g., see [61]; see also [8]). Many other researchers seem to have had success measuring discomfort via internally-developed brief rating scale questionnaires focusing on visual discomfort, headaches, and related questions.

[14]

[15]

The results of this review concur with those we found in a previous less comprehensive review [40]. In that work, we focused on only the human factors and human–computer interaction literature. We conclude again with the observation that S3D displays come with a host of unique human factors challenges including the simulator-sickness symptoms of eyestrain, headache, fatigue, disorientation, nausea, and malaise. These side effects appear within a large proportion of the population. For instance, a survey conducted by the American Optometric Association reported that at least a quarter of people who watched S3D films, television, or videogames experienced such symptoms [1]. An informal online survey by HomeTheater.com found that 53% of people who have viewed S3D content have experienced these kinds of symptoms [76]. This is limited data, but it is possible that as many as 25– 50% of the general population may have uncomfortable experiences when viewing S3D displays. Since S3D displays seem to offer performance benefits for specific (depth-related spatial) tasks, these human factors concerns suggest that the technology should be wielded delicately and applied carefully in order to assure both high comfort and high performance for users.

[16] [17] [18]

[19] [20]

[21]

[22] [23] [24]

[25]

[26]

Appendix A. Supplementary material Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.displa.2013. 10.004.

[27]

[28]

References [1] American Optometric Association (2010), ‘‘3-D TV and movies look to attract viewers but not everyone can ‘see’ what all the hype is about’’. [digital press release] on May 18, 2010. . [2] T. Bando, A. Iijima, S. Yano, Visual fatigue caused by stereoscopic images and the search for the requirement to prevent them: a review, Displays 33 (2) (2011) 76–83. [3] S.V. Bemis, J.L. Leeds, E.A. Winer, Operator performance as a function of types of display: conventional versus perspective, Human Factors 30 (1988) 163– 169. [4] K.R. Boff, Critical research issues on cockpit applications of 3-D displays, in: D.J. Getty (Ed.), Three-Dimensional Displays: Perceptual Research and Applications to Military Systems. Proceedings, National Academy of Sciences, National Research Council: Committee on Human Factors, ONR technical report, A122577, 1982. [5] M.F. Bradshaw, K.M. Elliott, S.J. Watt, P.B. Hibbard, I.R. Davies, P.J. Simpson, Binocular cues and the control of prehension, Spatial Vision 17 (1–2) (2004) 95–110. [6] M.E. Brown, J.J. Gallimore, Visualization of three-dimensional structure during computer-aided design, International Journal of Human–Computer Interaction 7 (1) (1995) 37–56. [7] J.Y.C. Chen, E.C. Haas, M.J. Barnes, Human performance issues and user interface design for teleoperated robots, IEEE Transactions on Systems, Man, & Cybernetics – Part C: Applications and Reviews 37 (6) (2007) 1231–1245. [8] E.G. Conlon, W.J. Lovegrove, E. Chekaluk, P.E. Pattison, Measuring visual discomfort, Visual Cognition 6 (1999) 637–663. [9] J.E. Cutting, P.M. Vishton, Perceiving layout and knowing distances: the integration, relative potency, and contextual use of different information about depth, in: W. Epstein, S. Rogers (Eds.), Handbook of Perception and Cognition, vol. 5: Perception of Space and Motion, Academic Press, San Diego, CA, 1995, pp. 69–117. [10] S.C. de Vries, P. Padmos, Steering a simulated unmanned aerial vehicle using a head-slaved camera and HMD, Proceedings of SPIE 3058 (1997) 24–33. [11] S.C. de Vries, P. Padmos, Steering a simulated unmanned aerial vehicle using a head-slaved camera and HMD: Effects of HMD quality, visible vehicle references and extended stereos cueing, Proceedings of SPIE 3362 (1998) 80–91. [12] S. Dixon, E. Fitzhugh, D. Aleva, Human factors guidelines for applications of 3D perspectives: a literature review, Proceedings of SPIE 7327 (2009) 73270K-1. [13] J.V. Draper, S. Handel, C.C. Hood, C.T. Kring, Three experiments with stereoscopic television: when it works and why, Proceedings of the IEEE

[29]

[30] [31]

[32]

[33]

[34]

[35]

[36]

[37] [38]

[39]

[40]

[41]

[42]

25

International Conference on Decision Aiding for Complex Systems – Systems, Man, & Cybernetics 2 (1991) 1047–1052. D. Drasic, Skill acquisition and task performance in teleoperation using monoscopic and stereoscopic video remote viewing, in: Proceedings of the Human Factors Society 35th Annual Meeting, 1991, pp. 1367–1371. D. Drasic, J.J. Grodski, Using stereoscopic video for defence teleoperation, Proceedings of SPIE 1915 (1993) 58–69. A.R. Fielder, M.J. Moseley, Does stereopsis matter in humans?, Eye 10 (1996) 233–238 J.J. Gallimore, M.E. Brown, Visualization of 3-D computer-aided design objects, International Journal of Human–Computer Interaction 5 (4) (1993) 361–382. D.J. Getty, Three-dimensional displays: perceptual research and applications to military systems, in: Proceedings, National Academy of Sciences, National Research Council: Committee on Human Factors, ONR technical report, A122577, 1982. D.J. Getty, P.J. Green, Clinical applications for stereoscopic 3-D displays, Journal of the Society for Information Display 15 (6) (2007) 377–384. V.S. Grinberg, G. Podnar, M.W. Siegel, Geometry of binocular imaging, in: Proceedings of the IST/SPIE Symposium on Electronic Imaging, Stereoscopic Displays and Applications, 1994, pp. 2177. I.D. Haskell, C.D. Wickens, Two- and three-dimensional displays for aviation: a theoretical and empirical comparison, International Journal of Aviation Psychology 3 (1) (1993) 87–109. M. Hayhoe, B. Gillam, K. Chajka, E. Vecellio, The role of binocular vision in walking, Visual Neuroscience 26 (1) (2009) 73–80. R.T. Held, T.T. Hui, A guide to stereoscopic 3D displays in medicine, Academic Radiology 18 (8) (2011) 1035–1048. J. Hofmeister, T.G. Frank, A. Cuschieri, N.J. Wade, Perceptual aspects of twodimensional and stereoscopic display techniques in endoscopic surgery: review and current problems, Seminars in Laparoscopic Surgery 8 (1) (2001) 12–24. N. Holliman, 3D display systems, in: Handbook of optoelectronics, IOP Press, London, 2005. . P.A. Howarth, Potential hazards of viewing 3-D stereoscopic television, cinema and computer games: a review, Ophthalmic and Physiological Optics 31 (2011) 111–122. J. Hsu, Z. Pizlo, D.M. Chelberg, C.F. Babbs, E.J. Delp, Issues in the design of studies to test the effectiveness of stereo imaging, IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 26 (6) (1996) 810–819. W.A. IJsselsteijn, P.J.H. Seuntiëns, L.M.J. Meesters, Human factors of 3D Displays. Chapter 12, in: Schreer, Kauff, Sikora (Eds.), 3D Videocommunication: Algorithms, Concepts, and Real-Time Systems in Human Centered Communication, Wiley & Sons, Ltd., 2006 (doi: 10.1002/ 0470022736.ch12). R.K. Jones, D.N. Lee, Why two eyes are better than one: the two views of binocular vision, Journal of Experimental Psychology: Human Perception and Performance 7 (1) (1981) 30–40. G. Jones, D. Lee, N. Holliman, D. Ezra, Controlling perceived depth in stereoscopic images, Proceedings of SPIE 4297 (2001) 42–53. R.S. Kennedy, N.E. Lane, K.S. Berbaum, Lilienthal, Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness, International Journal of Aviation Psychology 3 (3) (1993) 203–220. D. Kim, S. Choi, K. Sohn, Depth adjustment for stereoscopic images and subjective preference evaluation, Journal of Electronic Imaging 20 (3) (2011) 033011-1. W.S. Kim, F. Tendick, L.W. Stark, Visual enhancements in pick-and-place tasks: human operators controlling a simulated cylindrical manipulator, IEEE Journal of Robotics and Automation RA-3 (5) (1987) 418–425. A. Kulshreshth, J. Schild, J.J. LaViola, Evaluating user performance in 3D stereo and motion enabled video games, in: Proceedings of the International Conference on the Foundations of Digital Games (FDG ’12), 2012, pp. 33–40. M.T.M. Lambooij, W.A. IJsselsteijn, I. Heyndrickx, Visual discomfort in stereoscopic displays: a review, Journal of Imaging Science and Technology 53 (3) (2007) 30201-1. M.S. Landy, L.T. Maloney, E.B. Johnston, M. Young, Measurement and modeling of depth cue combination: in defense of weak fusion, Vision Research 35 (3) (1995) 389–412. J.M. Lee, J. MacLachlan, W.A. Wallace, The effects of 3D imagery on managerial data interpretation, MIS Quarterly 10 (3) (1986) 257–269. D.M. Litynski, M. Grabowski, W.A. Wallace, The relationship between threedimensional imaging and group decision making: an exploratory study, IEEE Transactions on Systems, Man, and Cybernetics–Part A: Systems and Humans 27 (4) (1997) 402–411. A. Loftus, P. Servos, M.A. Goodale, N. Mendarozqueta, M. Mon-Williams, When two eyes are better than one in prehension: monocular viewing and end-point variance, Experimental Brain Research 158 (3) (2004) 317–327. J.P. McIntire, P.R. Havig, E.E. Geiselman, What is 3D good for? A review of human performance on stereoscopic 3D displays, Proceedings of SPIE 8383 (2012) 83830X-1. S.P. McKee, S.N.J. Watamaniuk, J.M. Harris, H.S. Smallman, D.G. Taylor, Is stereopsis effective in breaking camouflage for moving targets?, Vision Research 37 (15) (1997) 2047–2055 D.R. Melmoth, S. Grant, Advantages of binocular vision for the control of reaching and grasping, Experimental Brain Research 171 (3) (2006) 371–388.

26

J.P. McIntire et al. / Displays 35 (2014) 18–26

[43] D.R. Melmoth, A.L. Finlay, M.J. Morgan, S. Grant, Grasping deficits and adaptations in adults with stereo vision losses, Investigative Ophthalmology and Visual Science 50 (8) (2009) 3711–3720. [44] F. Miksicek, Causes of visual fatigue and its improvements in stereoscopy, Technical Report DCSE/TR-2006-04, Department of Computer Sciences and Engineering, University of West Bohemia, Czech Republic, 2006. [45] R.R. Mourant, L. Parsi, Training in a virtual stereoscopic environment, in: Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting, 2002, pp. 2206–2209. [46] N. Naikar, Perspective displays: a review of human factors issues, Aeronautical and Maritime Research Laboratory, Defense Science & Technology Organisation, Australia; technical report DSTO-TR-0630, 1998. [47] A.C. Neubauer, S. Bergner, M. Schatz, Two- vs. three-dimensional presentation of mental rotation tasks: sex differences and effects of training on performance and brain activation, Intelligence 38 (5) (2010) 529–539. [48] C.A. Ntuen, M. Goings, M. Reddin, K. Holmes, Comparison between 2-D & 3-D using an autostereoscopic display: the effects of viewing field and illumination on performance and visual fatigue, International Journal of Industrial Ergonomics 39 (2) (2008) 388–395. [49] A.R. O’Connor, E.E. Birch, S. Anderson, H. Draper, The FSOS Research Group, The functional significance of stereopsis, Investigative Ophthalmology and Visual Science 51 (4) (2010) 2019–2023. [50] A.E. Patla, E. Niechwiej, V. Racco, M.A. Goodale, Understanding the contribution of binocular vision to the control of adaptive locomotion, Experimental Brain Research 142 (4) (2002) 551–561. [51] R. Patterson, Human factors of 3-D displays, Journal of the Society for Information Display 15 (2007) 861–871. [52] R. Patterson, Review paper: human factors of stereo displays: an update, Journal of the Society for Information Display 17 (2009) 987–996. [53] R. Patterson, W.L. Martin, Human stereopsis, Human Factors 34 (6) (1992) 669–692. [54] R. Patterson, A. Silzars, Immersive stereo displays, intuitive reasoning, and cognitive engineering, Journal of the Society for Information Display 17 (2009) 443–448. [55] S. Pastoor, Human factors of 3D displays in advanced image communications, Displays 14 (3) (1993) 150–157. [56] S. Pastoor, M. Wöpking, 3-D displays: a review of current technologies, Displays 17 (1997) 100–110. [57] E. Peinsipp-Byma, N. Rehfeld, R. Eck, Evaluation of stereoscopic 3D displays for image analysis tasks, Proceedings of SPIE 7237 (2009) 72370L. [58] A.H. Reinhardt-Rutland, Remote operation: a selective review of research into visual depth perception, Journal of General Psychology 123 (3) (1996) 237– 248. [59] J.M. Reising, K.M. Mazur, 3-D displays for cockpits: where they payoff, Proceedings of SPIE 1256 (1990) 35–43. [60] R. Rosenthal, The file drawer problem and tolerance for null results, Psychological Bulletin 86 (3) (1979) 638–641. [61] M. Rouse, E. Borsting, G.L. Mitchell, S.A. Cotter, M. Kulp, M. Scheiman, C. Barnhardt, A. Bade, T. Yamada, Validity of the convergence insufficiency symptom survey: a confirmatory study, Optometry and Vision Science 86 (2009) 357–363.

[62] P. Servos, M.A. Goodale, L.S. Jakobson, The role of binocular vision in prehension: a kinematic analysis, Vision Research 32 (8) (1992) 1513–1521. [63] M.J. Singer, J. Ehrlich, S. Cinq-Mars, J.-P. Papin, Task performance in virtual environments: stereoscopic versus monoscopic displays and head-coupling. U.S. Army Research Institute, Technical Report 1034, 1995. [64] H.S. Smallman, M. St. John, Naïve realism: misplaced faith in realistic displays, Ergonomics in Design: The Quarterly of Human Factors Applications 13 (3) (2005) 6–13. [65] M. St. John, M.B. Cowen, H.S. Smallman, H.M. Oonk, The use of 2D and 3D displays for shape-understanding versus relative-position tasks, Human Factors 43 (1) (2001) 79–98. [66] B.A. Steiner, D.A. Dotson, The use of 3-D stereo display of tactical information, in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, October, vol. 34, No. (1), 1990, pp. 36–40. [67] W.J. Tam, F. Speranza, S. Yano, K. Shimono, H. Ono, Stereoscopic 3D-TV: visual comfort, IEEE Transactions on Broadcasting 57 (2) (2011) 335–346. [68] K. Ukai, Human factors for stereoscopic images, IEEE International Conference on Multimedia and Expo (2006) 1697–1700. [69] M.H.P.H. van Beurden, G. van Hoey, H. Hatzakis, W.A. IJsselsteijn, Stereoscopic displays in medical domains: a review of perception and performance effects, Proceedings of SPIE 7240 (2009) 72400A-1. [70] K. Votanopoulos, F.C. Brunicardi, J. Thornby, C.F. Bellows, Impact of threedimensional vision in laparoscopic training, World Journal of Surgery 32 (2008) 110–118. [71] J.P. Wann, S. Rushton, M. Mon-Williams, Natural problems for stereoscopic depth perception in virtual environments, Vision Research 35 (19) (1995) 2731–2736. [72] C. Ware, C. Gobrecht, M.A. Paton, Dynamic adjustment of stereo display parameters, IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans 28 (1) (1998) 56–65. [73] S.J. Watt, M.F. Bradshaw, The visual control of reaching and grasping: binocular disparity and motion parallax, Journal of Experimental Psychology: Human Perception and Performance 29 (2) (2003) 404–415. [74] G. Westheimer, Three-dimensional displays and stereo vision, Proceedings of the Royal Society B 278 (2011) 2241–2248. [75] C. Wickens, The when and how of using 2D and 3D displays for operational tasks, in: Proceedings of the IEA 2000/HFES 2000 Congress, 2000, pp. 3-403–3-406. [76] S. Wilkinson, 3D Sickness Explained?’’ Home Theater [web article], 25 January 2011. . [77] P. Willemsen, A.A. Gooch, W.B. Thompsen, S.H. Creem-Regehr, Effects of stereo viewing conditions on distance perception in virtual environments, Presence 17 (1) (2008) 91–101. [78] S.P. Williams, R.V. Parrish, New computational control techniques and increased understanding for stereo 3-D displays, Proceedings of SPIE 1256 (1990) 73–82. [79] A. Woods, T. Docherty, R. Koch, Image distortions in stereoscopic video systems, Proceedings of the SPIE, Stereoscopic Displays and Applications (1993) 1915. [80] J. Zeidner, R. Sadacca, A.I. Schwartz, Human factors studies in image interpretation: the value of stereoscopic viewing, Adjutant General’s Office (Army) Washington, D.C., technical report, June 1961.