CID-8, KTH,Stockholm, Sweden 1997
Effects of Image Resolution on Depth Perception in Stereo and Non-stereo Images Kai-Mikael Jää-Aro and Lars Kjelldahl
Kai-Mikael Jää-Aro and Lars Kjelldahl Effects of Image Resolution on Depth Perception in Stereo and Non-stereo Images Report number: CID-8 Publication date: August 1997 E-mail of author:
[email protected],
[email protected] URL of author: http://www.nada.kth.se/~kai, http://www.nada.kth.se/~lassekj
Reports can be ordered from: CID, Centre for User Oriented IT Design Nada, Dept. Computing Science KTH, Royal Institute of Technology S-100 44 Stockhom, Sweden telephone: + 46 8 790 91 00 fax: + 46 8 790 90 99 e-mail:
[email protected] URL: http://www.nada.kth.se/cid/
Effects of image resolution on depth perception in stereo and non-stereo images Kai-Mikael J¨aa¨ -Aro and Lars Kjelldahl Interaction and Presentation Laboratory Department of Numerical Analysis and Computing Science Royal Institute of Technology SE-100 44 Stockholm SWEDEN
ABSTRACT Head-mounted displays (HMDs) in use today have fairly limited resolution, but the extents to which this low resolution may be detrimental to various tasks is not well known. We have studied the effect of low resolution on distance perception by letting experimental subjects estimate distances to objects in computer-generated 3D scenes. The images have been presented at varying resolutions both binocularly and biocularly on a workstation screen viewed through a Cyberscope stereoscopic device and biocularly in a Flight Helmet HMD. Our results indicate:
For good distance judgements, anti-aliasing is more important than stereo. In fact, stereo may even be detrimental in low resolution images. Anti-aliasing works well as a resolution-enhancer. The fuzziness of the LCD HMD screens gives an anti-aliasing effect, partially offsetting the low resolution. Subjects utilise different estimation strategies depending on the resolution of the images to minimise their estimation errors. Error estimates vary depending on the target object shape—there is a tendency for better estimates for those objects whose sides line up with the coordinate axes. The differences between subjects are considerably larger than those within subjects.
Keywords: Head-mounted display, stereoscopy, depth perception, anti-aliasing
1 Background Depth perception, defined as the ability to visually determine which objects are closer to and which further away from the observer, is something in which humans excel, while distance perception, the determining of the absolute distance to an object, is more difficult.1, 2 Depth perception is an intensely studied subject and a number of criteria for how various parameters in the presented scene affect the ability of the perceiver to judge depth have been formulated.3–11 Most of these studies are made using physical objects, or CRT images with as high resolution as possible. For computergenerated images, a relevant but less explored factor is the resolution of the image, in particular for images presented in current LCD-based (= affordable) HMDs with their low resolution (the addressable resolution typically being 300 200 pixels for a 1 1 rad field of view12 ). The email addresses of the authors are fkai,
[email protected]. ftp://ftp.nada.kth.se/IPLab/TechReports/IPLab-116.ps.Z
A machine-readable copy of this paper can be found in
Figure 1: A lowest-resolution (52 35 pixels) stereo pair of stellated tetrahedra. The target object is at a virtual distance of 3.1 m.
Figure 2: The object types presented to the subjects: Tetrahedron, cube, stellated cube, stellated octahedron, octahedron, stellated tetrahedron.
As low resolution causes quantisation errors in images, disparity should be subject to artifacts and stereopsis consequently negatively affected. We wanted to quantify how badly viewers were affected by the resolution of a HMD and possibly give some recommendations for minimum resolutions. Furthermore we wanted to see if particular object shapes were affected more than others and therefore should be avoided or presented with additional depth cues.
2 Experimental setup The subjects were presented images at different resolutions of objects for which they were asked to make distance estimates. Each image contained three identical objects as shown in fig 1. The subjects were told that the rightmost and middle object were 1 and 2 m away, respectively, and were asked to write on a paper form the distance to the leftmost object to within 0.1 m. This third object was placed 2–4 m away (the subjects were not told this restriction). The objects were cubes, octahedra, tetrahedra and stellated versions of the same (see fig 2). No shadows or textures were used in the images. Images containing all combinations of objects and resolutions were generated, a separate set for each experimental condition. For each object/resolution combination, two images were generated with the target object at two different distances. This series was then doubled and the resulting series of images shown in random order to the subjects. Before the series of images to be judged the subjects were shown a random dot stereogram as a check for stereo-deficiency. The subjects were manually timed for the time taken for each third of the display series. The images were generated in rayshade13 and presented in xv.14 The field of view in the images was fixed at 0.7 rad and the interocular distance at 64 mm; the zero disparity viewpoint was set to the middle object, placed 2 m away right in front of the viewer. As the images were displayed on a raster screen, in a fixed position and lacking many depth cues, the experimental task should not be confused with depth perception “in the wild”, but we believe it to be typical of depth estimates that have to be done for perspective-rendered 3D objects presented on computer displays.
(a) Cyberscope
(b) Flight Helmet
Figure 3: The display devices used in the experiment.
2.1 Images presented on workstation In the first three experiments we displayed the images on a Sun workstation screen through a Simsalabim Systems Cyberscope15 (fig 3(a)). The screen had a resolution of 1152 900 pixels and a measured size of 360 270 mm. According to the manufacturer’s literature on the Cyberscope the optical path length is 320 mm, and the horisontal field of view 0.8 rad. Each subject had to judge 2 60 scenes. The images were generated at five leves of resolution—832 560, 416 280, 208 140, 104 70 and 52 35 pixels—ranging from the full workstation resolution of 3 pixels/mm (1040 pixels/rad) to a 16th of the resolution, thus bracketing the range of typical LCD screens. The lower resolution images were scaled by pixel replication to be the same size as the highest resolution images. Three different experiments were performed with this setup: 1. Anti-aliased stereo images. 2. Non-anti-aliased stereo images. 3. Non-anti-aliased non-stereo (biocular) images.
2.2 Images presented in HMD In the last experiment we used a Virtual Research Flight Helmet for image presentation (fig 3(b)). Due to lack of stereo hardware we could only show the scenes in non-stereo (biocularly). Each subject had to judge 2 48 scenes. The images were generated at two levels of resolution, 160 120 and 80 60, scaled 4 and 8 times respectively, corresponding to the maximum resolvable resolution on the HMD screens (approximately 90 pixels/rad at a field of view of 1.7 rad) and half that. Both sets of images were presented with and without anti-aliasing.
3 Results The subjects were mainly male undergraduate students1 in their early 20s, who received course credit in exchange for their participation, as well as a few volunteers. The number of subjects for the four experimental series were 22, 21, 17 and 12, respectively. Some subjects participated in several series, so the total number of subjects was 66. We kept track of which subjects wore spectacles, but there was no significant difference in estimates between those who did and those who did not (as indeed there should not have been if the lenses were of the proper strength), so we will not further consider this issue. Four subjects failed the stereogram test but did not seem to perform worse than the stereo normal subjects 1 The skewed subject population is obviously due to easy availability, but it is probably also representative for the set of users that are likely to be using HMDs in the near future. . .
,12
,1
,1
,08
,08 Mean relative error
Mean relative error
,12
,06 ,04 ,02 0
,06 ,04 ,02 0
1
2
4
8
16
1
Scaling
(a) Anti-aliased, stereo
4
8
16
(b) Non-anti-aliased, stereo
,12
,1
,1 Mean relative error
,12
,08 Mean relative error
2 Scaling
,06 ,04
,08 ,06 ,04 ,02
,02
0
0 1
2
4
8
16
4, Anti-aliased 8, Anti-aliased 4, Non-anti-aliased 8, Non-anti-aliased
Scaling
(c) Non-anti-aliased, non-stereo
(d) Non-stereo, HMD
Figure 4: The relative error for each image resolution.
(this is an indication that stereopsis was a less important factor for these distance estimates), so they have not been excluded from the analyses. Nor was there any significant relation between the time taken for estimates and their accuracy. We have chosen to consider the relative error ((true distance , estimate)=true distance) as a useful statistic. Under all experimental conditions the subjects tended to underestimate the distance to the target object with on average 5– 8%; in fact the the stated distance decreased with an average of 40–90 mm for the second presentation of a scene—while small, a highly significant effect (p < 0:0001 in a paired t-test, N = 1320, 1260, 1020 and 576 for the four experimental series, respectively). According to studies by Patterson et al,10 stereo presentations with uncrossed parity lead to underestimation of distances, but since the effect we measured persisted for non-stereo images as well, it must be due to something else.
3.1 Effects of resolution Low resolution was, as expected, a negative factor for distance judgements, but estimation errors are not a monotonic function of resolution, as can be seen in figure 4. For the anti-aliased images only the lowest resolution images had significantly (at the 5%-level as determined by an unpaired t-test) worse estimates. For the non-anti-aliased images there was an interesting effect in that the lowest resolution images (marked 16) in fact had lower estimation errors than the second-lowest resolution images (8). We believe that the pixels became sufficiently coarse in the lowest resolution images that the subjects consciously or unconsciously switched to counting pixels along the polyhedron sides for this case and that this gave better results than whatever strategies they used otherwise.
,12
,1
,1
,08
,08 Mean relative error
Mean relative error
,12
,06 ,04 ,02 0
,06 ,04 ,02 0
Cube StellatedCube Octahedron StellatedOctahedron Tetrahedron StellatedTetrahedron
Cube StellatedCube Octahedron StellatedOctahedron Tetrahedron StellatedTetrahedron
(b) Non-anti-aliased, stereo
,12
,12
,1
,1
,08
,08 Mean relative error
Mean relative error
(a) Anti-aliased, stereo
,06 ,04 ,02 0
,06 ,04 ,02 0
Cube StellatedCube Octahedron StellatedOctahedron Tetrahedron StellatedTetrahedron
(c) Non-anti-aliased, non-stereo
Cube StellatedCube Octahedron StellatedOctahedron Tetrahedron StellatedTetrahedron
(d) Non-stereo, HMD
Figure 5: The relative error for each object type.
We also note that the difference in estimates between anti-aliased and non-anti-aliased stereo images is larger than that between stereo and non-stereo non-anti-aliased images, which seems to imply that anti-aliasing is more important for accurate estimates than stereo is, and in fact it seems that the stereo images get worse estimates at low resolutions than the non-stereo images. This can be explained by pixels not lining up in the two images, creating ghost shapes with ambiguous disparity, confounding the subjects. Interestingly enough the images shown in the HMD got errors almost as low as those for the anti-aliased stereo images and the errors were similar whether the HMD images were anti-aliased or not. We believe that this is due to the low contrast of the HMD screens, causing a blurring of the images similar to anti-aliasing.
3.2 Effects of object type There is a complex interaction between the type of object shown, the image resolution, the distance to the object and the presentation form (cf fig 5). However, we have not been able to isolate any single factor that is correlated with the size of the error estimate—neither the number of vertices, sides or faces, nor the slope of the sides seems to affect the distance estimates. Subjects have reported familiarity (e g cubes being more familiar objects than stellated tetrahedra) as subjectively improving distance estimates. We measured the disparity differences between adjacent depth planes for the stereo images and the difference between images at adjacent depth planes for the non-stereo images on the assumption that object types that tend to have large differences will be easy to estimate the distance to. We have ascertained that for all placements of all objects under all resolutions, the displayable depth planes will be spaced closer than the required reporting accuracy. In some cases the perceivable difference
,3
,2
,2 Mean relative error
Mean relative error
,3
,1 0 -,1 -,2
,1 0 -,1 -,2
-,3
-,3 1
3 2
5 4
7 6
9 11 13 15 17 19 21 8 10 12 14 16 18 20 22
1
3 2
,3
,3
,2
,2
,1 0 -,1 -,2
7 6
9 8
11 13 15 17 19 21 10 12 14 16 18 20
(b) Non-anti-aliased, stereo
Mean relative error
Mean relative error
(a) Anti-aliased, stereo
5 4
,1 0 -,1 -,2
-,3
-,3 1
3 2
5 4
7 6
9 8
11 13 15 17 10 12 14 16
(c) Non-anti-aliased, non-stereo
1
3 2
5 4
7 6
9 8
11 10
12
(d) Non-stereo, HMD
Figure 6: Mean relative errors for the different subjects for all experimental conditions.
will be quite small, though, primarily at long distances and low resolutions; in particular the cube, octahedron and tetrahedron seem sensitive to this, whereas the more complex stellated objects apparently contain enough detail that they will be different at most distances. In the non-stereo images, the stellated cube also suffers from collapsed depth planes. As before, the anti-aliased stereo images and the HMD images give similar estimates. The stellated cube has the largest relative error with the octahedron coming in second and the cube and tetrahedron having the smallest errors. For the nonanti-aliased images this changes and the errors for the cube and tetrahedron increase. Presumably the order in the anti-aliased images is the “natural” order, such that cubes and tetrahedra are easier to estimate the distance to, perhaps since they have surfaces parallel with the horizon; when anti-aliasing is removed, the collapsed depth planes make it more difficult to judge the distance to these objects than to the others which were more difficult to begin with, but do not suffer as greatly from collapsed depth planes.
4 Discussion For all experimental conditions the subject is the most important factor that affects depth estimates—the difference between subjects is larger than that between experimental conditions for the subjects (cf fig 6). It seems reasonable that different subjects naturally vary in their ability to make depth estimates, whatever the circumstances, but the experimental conditions may have added to this—the fixed inter-ocular distance (IOD) in the displays may have affected depth estimates adversely for those whose IODs differed from the set value; subjects may have different tolerance to watching degraded images; while all subjects were
frequent computer users, some may have had more occasion to make depth judgements on computer displays, e g when using a CAD system or playing 3D games. It could be argued that the reason anti-aliasing improves distance estimates is that we in fact get increased resolution through the use of more colours, but this is belied by the fact that we get the same effect in merely blurred images in the HMD. Perhaps the fuzziness of contours supports counteracts image artifacts, or possibly we are in fact seeing two separate mechanisms that each on their own can help distance estimates. There are some explanations for the underestimation of distances that should be explored:
Screen being closer than image plane, forcing a compromise between accommodation and other depth cues. Differences in subject IOD and image IOD. Subjects estimating the distance to a point on the surface of the polyhedron, rather than its centre. Artifacts in the perspective rendering.
5 Conclusions We summarize the most important results of our experiments:
Subjects have a very large variability in their ability to make correct distance estimates. For the images presented on the workstation anti-aliasing not only increased the ability to perceive depth, but kept it constant for all but the worst image resolution. For the lowest resolution images, stereo seemed more of a hindrance than a benefit for distance estimates in the absence of anti-aliasing. For the non-anti-aliased images, it was actually better to have so low resolution that the subjects could count pixels. Images presented in an HMD seem to benefit from the low contrast, giving an anti-aliasing effect.
Acknowledgements We wish to thank KTH for the opportunity for Dr Kjelldahl to do research within his duties as senior lecturer and Ericsson and Telia for funding Mr J¨aa¨ -Aro’s research.
References [1] Gerald Westheimer. The spatial sense of the eye. Investigative Ophthalmology and Visual Science, 18(9):893–912, 1979. [2] Robert Patterson. Human stereopsis. Human Factors, 34(6):669–692, 1992. [3] Ross L. Pepper, Robert E. Cole, and Edward H. Spain. The influence of camera separation and head movement on perceptual performance under direct and tv-displayed conditions. Proceedings of the SID, 24(1):73–80, 1983. [4] Edward H. Spain, Robert E. Cole, and Ross L. Pepper. Visual performance and fatigue with stereoscopic television displays. In Proceedings of the 1984 National Topical Meeting on Robotics and Remote Handling in Hostile Environments, pages 61–67, 1984. [5] Stanley N. Roscoe. Judgments of size and distance with imaging displays. Human Factors, 26:617–629, 1984. [6] Shojiro Nagata. How to reinforce perception of depth in single two-dimensional pictures. Proceedings of the SID, 25(3):239–246, 1984. [7] Yei-Yu Yeh and Louis D. Silverstein. Using electronic stereoscopic color displays: Limits of fusion and depth discrimination. In Woodrow E. Robbins and Scott S. Fisher, editors, Three-Dimensional Visualization and Display Technologies, volume 1083 of SPIE Proceedings, pages 196–204, 1989.
[8] Christopher D. Wickens. Three-dimensional stereoscopic display implementation: Guidelines derived from human visual capabilities. In Merritt and Fisher,16 pages 2–11. [9] William F. Reinhart, Robert J. Beaton, and Harry L. Snyder. Comparison of depth cues for relative depth judgments. In Merritt and Fisher,16 pages 12–21. [10] Robert Patterson, Linda Moe, and Tiger Hewitt. Factors that affect depth perception in stereoscopic displays. Human Factors, 34(6):655–667, 1992. [11] R. Troy Surdick, Elizabeth T. Davis, Robert A. King, Gregory M. Corso, Alexander Shapiro, Larry Hodges, and Kelly Elliot. Relevant cues for the visual perception of depth: Is where you see it where it is? In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting, volume 2, pages 1305–1309, October 1994. [12] Head mounted display survey: A comprehensive round-up of products. Real Time Graphics, 4(2):8–13, August 1995. [13] Craig
E.
Kolb.
Rayshade
User’s
Guide
and
Reference
Manual,
0.4
edition,
January
1992.
3.10a
edition,
1994.
http://www-graphics.stanford.edu/~cek/rayshade/doc/guide/guide.html.
[14] John
Bradley.
xv—Interactive
Image
f
Display
for
the
X
Window
g
System,
ftp://ftp.cis.upenn.edu/pub/xv/docs/ xvdocs.pdf,xvdocs.ps.Z .
[15] Simsalabim Systems, Inc, 1994. http://www.nscee.edu/nscee/Research/Cyberscope/cyber.html. [16] John O. Merritt and Scott S. Fisher, editors. Stereoscopic Displays and Applications, volume 1256 of SPIE Proceedings, 1990.
