18 A 3-D Display System Using Motion Parallax

18 A 3-D Display System Using Motion Parallax Hiroshi Ono and Sumio Yano

Abstract Motion parallax was described as a cue to depth over 300 years ago. Despite this long history, little research has been addressed to it in comparison to binocular parallax. In recent years experimental interest in motion parallax has increased, following the re-discovery of the idea of yoking stimulus motion to head movement. Exploiting what is now known about observer-produced parallax, we propose a novel display system in which 3-D perception results from synchronizing the movement of stimuli on a 2-D screen to observers’ side-to-side head movements, or back and forth head rotations. After briefly reviewing the relevant literature, this chapter suggests how one might go about creating such a display system.

18.1 Introduction1 Motion parallax is described in most modern textbooks on perception. Most textbook descriptions of motion parallax focus on the perception of depth, but several also discuss perceived motion. Despite its widespread use, the term “motion parallax” is relatively new, but the concept itself has been applied to the perception of depth for over three centuries [2]. Although meager 1

This chapter is not a comprehensive review of the area of motion parallax. For such a review, we recommend the chapter entitled “Depth from motion parallax” in Howard and Rogers [1]. They are currently writing a more up-to-date review that will appear soon.

H. Ono Centre for Vision Research and Department of Psychology, York University, Toronto, Ontario, M3J 1P3 (Canada) e-mail: [email protected]

B. Javidi et al. (eds.), Three-Dimensional Imaging, Visualization, and Display, C Springer Science+Business Media, LLC 2009 DOI 10.1007/978-0-387-79335-1 18, 

363

364

H. Ono and S. Yano

compared to the extensive literature on binocular parallax, we now have sufficient knowledge on the depth perception from motion parallax to conceive an experimental 3-D display system. In this chapter, we first discuss the background material needed to understand the basis for the system, and then discuss, in general terms, how such a system might be made. The literature on motion parallax makes a distinction between observerproduced parallax and object-produced parallax. This chapter focuses on observer-produced parallax in making the 3-D display. See footnote 2 for a brief discussion of object-produced parallax.

18.2 Early Accounts of Motion Parallax and Early and Recent Experimental Studies Early historical accounts of motion parallax are detailed in Ono and Wade [2, 3] and, therefore, will not be elaborated here. To present a sample of the early writings we quote from Wheatstone [4]. Having invented the stereoscope, he was confronted with a question regarding depth perception by those who did not have binocular vision. He wrote: . . . the same solid object is represented to the mind by different pairs of monocular pictures, according as they are placed at different distances before the eyes, and the perception of these differences (though we seem to be unconscious of them) may assist in suggesting to the mind the distance of the object. . . . The mind associates with the idea of a solid object every different projection of it which experience has hitherto afforded; a single projection may be ambiguous, from its being one of the projections of a picture, or of a different solid object; but when different projections of the same objects are successively presented, they cannot all belong to another object, and the form to which they belong is completely characterized. [4, p. 377]

Although several accounts of motion parallax were given before the time of Wheatstone, the concept did not receive the concerted experimental attention accorded to binocular parallax (for a recent discussion of the surge of experiments using a stereoscope soon after its invention, see Ono et al. [5]). It was not until the turn of the last century that the first experiment on motion parallax was conducted. Bourdon [6, 7] demonstrated that judgments of the separation between two stimuli, equated in visual angle at different distances from an observer, were made accurately when the head moved, but not when the head remained stationary. Soon after Bourdon, Heine [8] controlled the rate of retinal image motion (or eye movement) with respect to the head movement and successfully simulated two stationary rods with two moving rods (see Fig. 18.1). He mechanically yoked his lateral body movement to the movements of the rods using a shoulder harness: the closer rod physically moved in the same lateral direction as the head/body movement and the far one physically moved

18 A 3-D Display System Using Motion Parallax

365

Fig. 18.1. An illustration of the apparatus used by Heine [8] to study depth perception from self-generated retinal motion (A and B) and the virtual reality created when an observer moves from side to side (C). The rod shown on the left side in Fig. 18.1A was attached to an observer’s shoulder harness. His/her lateral movement moved the rod that pivots at B and produced the movement of Stimulus 1 and 2. Adapted from Ono and Wade [2]. See text for a discussion

in the opposite lateral direction. That is, the normal relationship between retinal image displacement and head movement was reversed. Under these conditions, the perception of depth was opposite to the actual depth; the near stimulus appeared farther than the far one, which provided convincing evidence that the retinal image motion produced by a head/body movement is a cue to depth perception. Moreover, he found that yoking or “slaving” stimulus movements to head movements leads to unambiguous depth perception. (By “unambiguous” we mean the direction of the depth perceptions is stable, unlike that of the kinetic depth effect.) Modern counter parts of these two studies are Gonz´ alez, et al. [9] and Rogers and Graham [10]. Gonz´ alez, et al. used a procedure similar to that of Bourdon [6] and measured depth thresholds with and without head movement. The measured thresholds with head movement were considerably smaller than those without head movement. Rogers and Graham [10] developed a technique somewhat similar to that devised by Heine [8]. Their apparatus consisted of electronically yoking dot movements on a screen to lateral head movements. They successfully simulated different stationary surfaces (square, sine, triangular, or saw tooth) by yoking random dot movements to the observer’s head

366

H. Ono and S. Yano

Fig. 18.2. An illustration of the apparatus used by Rogers and Graham [10] to study depth perception from self-generated retinal motion (18-2A) and the virtual reality created when an observer moved from side to side (18-2B and 18-2C). Adapted from Ono and Wade [16]. See text for a discussion

movement2 (see Fig. 18.2). The perception produced was analogous to that produced by the random dot stereograms devised by Julesz [11]. As to the lower threshold of depth perception from motion parallax, Rogers and Graham’s [12] data indicated that it is not as low as that of binocular parallax, but the shape of the thresholds as a function of spatial frequency of corrugation was similar. The lowest threshold was located at 0.3 cpd for both the motion and binocular parallaxes (see Fig. 18.3). Subsequently, using a similar technique, Ono and Ujike [13] showed that (a) small parallax magnitudes led to the perception of depth without motion; (b) larger magnitudes led to the perception of depth with concomitant motion—apparent motion that occurs in synchrony with a head movement, and (c) yet larger magnitudes led to no depth perception with a large concomitant motion (see Appendix for a definition of the magnitude of motion parallax and for what is called “equivalent disparity”). The last finding indicated that the range of effectiveness of motion parallax for depth perception is limited, just as the range of effectiveness of binocular parallax is limited. Also as with binocular parallax Nawrot [14] and Ono, et al. [15] found that the distance of the screen scales the magnitude of parallax; for a given parallax magnitude, the apparent depth was smaller when the distance of the screen was greater.

2

Rogers and Graham [10] also produced these apparent surfaces by yoking the relative dot positions to the position of a moving screen (the object-produced parallax). See the arrow bars near the oscilloscope indicating the movement of the screen in Fig. 18.2. This finding suggests that a moving stimulus on a screen can be given an apparent depth by yoking parts of the stimulus with the position of it on the screen, i.e., a moving object can be made to have apparent depth in animation.

18 A 3-D Display System Using Motion Parallax

367

Fig. 18.3. The lower depth thresholds with motion parallax and binocular parallax as a function of corrugation frequency. Redrawn from Rogers and Graham [12]

Both the experimental techniques and the findings of Heine [8] and Rogers and Graham [10] are particularly relevant to the construction of a 3-D display system using motion parallax. They successfully produced “identical incoming messages,” or what Ames called “equivalent configurations” from different external physical arrangements (see Ittelson [17]) and created a virtual reality of depth. Just as a stereoscope provides the “identical incoming messages” received by the two eyes from a “natural” object or scene, the apparatus of Heine and that of Rogers and Graham provided the identical incoming messages received by one eye from a natural object or scene. Note from Fig. 18.1 that whether the far (or near) simulated stationary point is fixated or the actually moving far (or near) stimulus is pursued, the extent of eye movement relative to head movement is identical. Also note that if there were two actual stationary rods located at “Perceived Location 1” and “Perceived Location 2” the incoming message would be the same as that produced by Heine’s apparatus. What has been discussed above lays the foundation upon which we base our proposed 3-D display system. Before we do that, however, readers are asked to view the demonstrations located at the Web site: http://www.yorku. ca/hono/parallax˙demo. The demonstrations were made for a classroom setting [16], but viewing them on a computer screen can produce the 3-D experience. Remember to use one eye only and to synchronize your head movement with the moving marker at the bottom of the demonstration.

18.3 Demonstration The demonstration consists of two moving bars analogous to those used by Heine [8], or the moving dots that produced one of the four simulated surfaces analogous to those created by Rogers and Graham [10]. Moving along the

368

H. Ono and S. Yano

Fig. 18.4. An illustration of how the demonstrations are viewed with one eye

bottom of the screen is a marker used to guide the head movements (see Fig. 18.4). When you move your head from side to side in synchrony with the moving marker, the bars or the dots will move on your retina as though produced by your head movement. (The reason that these demonstrations work when viewed on the Web site is that the yoking of stimulus movements to head movements need not be as exact as in Heine’s or Rogers and Graham’s experiments.) The appropriate extent of head movement depends on how far you are from the screen. For these demonstrations, the greater the magnitude of lateral head movement, the smaller the parallax magnitude, because the parallax magnitude is the ratio of retinal image displacement (or velocity) to the displacement (or velocity) of head movement. Therefore, if you are near the screen, you need to move your head by a greater extent than if you are farther from the screen to produce the same parallax magnitude. Moreover, if you perceive concomitant motion, you need to move your head a greater extent (or move farther from the screen) to see stationary bars or a stationary surface. Once you find the appropriate extent of head movement to see a stationary stimulus with depth, move your head in the opposite direction to the moving marker on the bottom of the screen; you will see a reversal of the direction of depth.

18.4 The Suggested 3-D Display System To be completely faithful to Heine’s [8] or Rogers and Graham’s [10] idea, one can design a system where the lateral movement of the video camera is driven by the head movement. The signal from the camera can be presented on a screen in front of an observer while s/he keeps moving his/her head from side to side with one eye occluded; this arrangement eliminates the need for the moving marker on the bottom of the screen. The video signal obtained with

18 A 3-D Display System Using Motion Parallax

369

the camera moving leftward and rightward provides the identical message as an observer viewing the scene while moving his/her head from side to side. However, this system is impractical for 3-D tele-presence or remote sensing purposes, since the video signal would be delayed relative to the head position. A delay of half a cycle of head movement would reverse the direction of depth, as the demonstration indicated. Therefore, for the first experimental 3-D display system, it is suggested that you include a moving marker to indicate the position of the camera. The suggested 3-D display system using motion parallax is illustrated in Fig. 18.5. Figure 18.5A shows the recording system—the video camera moves

Fig. 18.5. Suggested 3-D display system. 18-5A shows the recording of a scene and 18-5B shows an observer viewing the scene on a television screen

370

H. Ono and S. Yano

from side to side on a rail while the camera is fixed on the person standing in a field. We suggest that the movement of the camera on the rail be controlled by a motorized crank system that produces reciprocating motion, but with a human operator, to select an object of interest and control the direction in which the camera is pointed. In Fig. 18.5A, the camera is pointed to the human standing in the field. The screen for the display can be of any size; a large screen in a lecture hall as the demonstrations discussed in the previous section or a TV or computer screen. Figure 18.5B shows the display system using a relatively small TV screen. Whatever the screen size, an observer moves his head from side to side in synchrony with the moving marker on the screen. The figure also shows the movement of stimuli located at different parts of the screen; the point to which the camera was fixed remains stationary, whereas a part above (farther than) the stationary point moves in the same direction as the head and a part below (nearer than) this point moves in the opposite direction. The frequency of head movement matches that of the frequency of the reciprocating motion of the camera. The extent of head movement required by the observer depends on the extent of the excursion made by the camera, the viewing distance of the screen, and the reduction of the visual angles of the stimuli on the screen. It is recommended that these factors be combined so that the frequency and the amplitude of the required head movement will have a peak velocity greater than 15 cm/s: i.e., above 1/2 Hz for 10 cm, and above 1/4 Hz above 20 cm excursion to ensure that the depth threshold is lowest with these head movements [18]. We expect that the motion parallax cue for depth produced by this system would combine with other depth cues such as the perspective cue, but how effectively it would combine remains to be determined. Also yet to be determined are the consequences of image parameters such as contrast ratio, resolution, and the color appearance of the display system. Nonetheless, this system would likely have advantages over a binocular 3-D display system in that it does not have to deal with (a) stereoblindness, (b) the visual fatigues usually associated with a binocular display system, and (c) two channels of video signals. Moreover, a 3-D experience can be created with present internet technology as seen in the demonstration. (Instead of lateral head movement, a horizontal rotation of the head can be used, as it would translate the viewing eye. See Steinbach, et al. [19]).3

18.5 Summary Based upon what is known about motion parallax, a 3-D display system can be created. As a display for games or entertainment, having to keep moving the head from side to side may not be an appealing feature, but the suggested 3

An up-and-down or a forward-and-backward head movement when yoked to appropriate retinal image motion is also effective in producing depth [19, 20, 21].

18 A 3-D Display System Using Motion Parallax

371

system may be useful as a tele-presence system. Creating the suggested experimental system would be useful for examining its cost effectiveness and exploring those stimulus situations that have an advantage over binocular display systems. Acknowledgements Writing of this chapter was supported by a grant from the Natural Sciences and Engineering Research Council of Canada. We wish to thank Esther Gonz´ alez, Linda Lillakas, Al Mapp, and Daniel Randles for their helpful comments on earlier versions of this chapter, and Linda Lillakas, for preparing the figures.

Appendix The geometry for specifying the magnitude of motion parallax is almost identical to that of binocular parallax, and vice versa. With this understanding, we now define the magnitude of motion parallax, which is the relative retinal image motion per head movement. If we were to specify the extent of retinal image motions in terms of visual angles (∝. and ß. in Fig. 18.6) (or velocities), the difference between the two divided by the extent (or velocity) of head movement is the parallax magnitude. If we were to compute (∝. – ß) when the head moves 6.2 cm, we would have a unit called “Equivalent Disparity” [12] that is equal to the unit of retinal disparity for the identical depth at a given distance.

Fig. 18.6. An illustration for defining the magnitude of motion parallax

372

H. Ono and S. Yano

References [1] Howard IP, Rogers BJ (2002) Seeing in Depth: Vol. II. Depth perception. Toronto: Porteous Publisher [2] Ono H, Wade NJ (2005) Depth and motion in historical descriptions of motion parallax. Percept 34:1263–1273 [3] Ono H, Wade NJ (2007) The paradoxes of parallaxes. Percept [4] Wheatstone C (1838) Contributions to the physiology of vision – Part the first. On some remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical Transactions of the Royal Society 128:371–394 [5] Ono H, Lillakas L, Wade NJ (2007) Seeing Double and Depth with Wheatstone’s Stereograms. Percept 36:1611–1623 [6] Bourdon B (1898) La perception monoculaire de la profondeur. Revue Philosophique 46:124–145 [7] Bourdon B (1902) La perception visuelle de lespace. Paris: Librairie Schleincher Freres ¨ [8] Heine L (1905) Uber Wahrnehmung und Vorstellung von Entfernungsunterschieden. Albrecht von Graefes Archiv f¨ ur klinische und experimentelle Opthamologie 61:484–498 [9] Gonz´ alez EG, Steinbach MJ, Ono H, Wolf ME (1989) Depth perception in children enucleated at an early age. Clinical Vision Sciences 4:173–177 [10] Rogers BJ, Graham ME (1979) Motion parallax as an independent cue for depth perception. Percept 8:125–134 [11] Julesz B (1971) Foundations of Cyclopean Perception. Chicago: University of Chicago Press [12] Rogers BJ, Graham ME (1982) Similarities between motion parallax and stereopsis in human depth perception. Vision Research 22:261–270 [13] Ono H, Ujike H (2005) Motion parallax driven by head movement: conditions for visual stability, perceived depth, and perceived concomitant motion. Percept 34:477–490 [14] Nawrot M (2003) Depth from motion parallax scales with eye movement gain. Journal of Vision 3:841–851 [15] Ono ME, Rivest J, Ono H (1986). Depth perception as a function of motion parallax and absolute-distance information. Journal of Experimental Psychology: Human Perception and Performance 12:331–337 [16] Ono H, Wade NJ (2006) Depth and motion perceptions produced by motion parallax. Teaching of Psychology 33:199–202 [17] Ittelson WH (1960) Visual Space Perception. New York: Springer [18] Ujike H, Ono H (2001) Depth thresholds of motion parallax as a function of head movement velocity. Vision Research 41:2835–2843 [19] Steinbach MJ, Ono H, Wolf M (1991) Motion parallax judgments of depth as a function of the direction and type of head movement. Canadian Journal of Psychology 45:92–98 [20] Sakurai K, Ono H (2000) Depth perception, motion perception and their tradeoff while viewing stimulus motion yoked to head movement. Japanese Psychological Research 42:230–236 [21] Yajima T, Ujike H, Uchikawa K (1998) Apparent depth with retinal image motion of expansion and contraction yoked to head movement. Percept 27: 937–949