Super Wide Viewer Using Catadioptrical Optics Hajime Nagahara
Yasushi Yagi
Masahiko Yachida
Graduate School of Engineering Science, Osaka University 1-3, Machikaneyama-cho, Toyonaka, Osaka, Japan
The institute of Scientific and Industrial Research, Osaka University 8-1, Mihogaoka, Ibaraki, Osaka, 567-0047, Japan
Graduate School of Engineering Science, Osaka University 1-3, Machikaneyama-cho, Toyonaka, Osaka, Japan
[email protected]
[email protected]
ABSTRACT Many applications have used a Head-Mounted Display (HMD), such as in virtual and mixed realities, and tele-presence. The advantage of HMD systems is the ease of feeling a 3D world in the display of animation or movies. However, the field of view (FOV) of commercial HMD systems is too narrow for feeling immersion. The horizontal FOV of many commercial HMDs is around 60 degrees, significantly narrower than that of humans. In this paper, we propose a super wide field of view head-mounted display consisting of an ellipsoidal and a hyperboloidal curved mirror. The horizontal FOV of the proposed HMD is 180 degrees and includes the peripheral view of humans. It increases reality and immersion of users. As well, the central region (60 degrees) of the FOV can measure 3D distances using stereoscopics.
Categories and Subject Descriptors I.3.1 [Computing Methodologies]: COMPUTER GRAPHICS—Hardware Architecture, Three-dimensional displays; I.3.7 [Computing Methodologies]: COMPUTER GRAPHICS— Three-Dimensional Graphics and Realism, Virtual reality; I.3.8 [Computing Methodologies]: COMPUTER GRAPHICS—Picture/Image Generation, Display algorithms
General Terms Design
Keywords Wide field of view, Head mount display, Catadioptrical optics, Tele-presence
1.
INTRODUCTION
Many applications have used a Head-Mounted Display (HMD), such as in virtual and mixed realities, and tele-presence.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. VRST'03, October 1-3, 2003, Osaka JAPAN. Copyright 2003 ACM 1-58113-569-6/03/0010...$5.00
[email protected]
The advantage of an HMD system is the ease of feeling a 3D world in the display of animation or movies. However, the field of view (FOV) of commercial HMD systems is too narrow for feeling immersion. The horizontal FOV of many commercial HMDs is around 60 degrees, significantly narrower than that of humans. In this paper, we propose a super wide field of view head-mounted display consisting of an ellipsoidal and a hyperboloidal curved mirror. The horizontal FOV of the proposed HMD is 180 degrees and so includes the human peripheral view and thus increases the reality and immersion for users. It is known that a lack of peripheral vision seriously influences postural control in humans [1]. Furthermore, Furness reported that for a human operator a FOV over 80 degrees was required for a feeling of immersion and reality [2]. Takahashi et al. [3] also measured the influence of a wide FOV on human attitude control. They concluded that an HMD with a wide FOV, such as 140 degrees, was better than a standard one. Caldwell et al. [4] reported that a narrow FOV hindered task efficiency and reality in tele-operations, even though they compared only 30 and 60 degree FOVs. These studies indicate that a wide FOV along with peripheral vision are important factors for increasing reality and immersion. In previous studies, several researchers and product makers have investigated HMDs with wider FOVs. Eyephone02 (VPL inc.) had an 80 degree horizontal FOV. Datavisor 80 (n-Vision inc.), Sim Eye XL100A (Kaiser Electro-Optics inc.), Gemini-Eye 3 (CAE USE inc.) and Fiber-Optic HMD (CAE inc.) increased the horizontal FOV to 120 degrees, 100 degrees, 100 degrees and 120 degrees, respectively. Takahashi et al. [3] constructed a 140 degrees FOV HMD system using 4 LCD panels and fresnel lenses. Inami et al. [5] developed an HMD, which extended the horizontal FOV to 110 degrees using Maxwellian optics. However, the FOV of the earlier HMDs are still smaller than that of human vision. In this paper, we propose a HMD with 180 degrees of FOV, which covers the peripheral vision of humans. The optics consist of a hyperboloidal convex and an ellipsoidal concave mirror. The proposed HMD does not require an eyepiece lens system, so the FOV is not limited by the lens size of the eyepiece lens system. Each eye of the proposed HMD optics covers a 120 degree horizontal view and a 60 degree vertical view. Thus, a 180 degree horizontal FOV is achieved by
Hyperboloidal mirror
Ellipsoldal mirror
\
Observer's eye (J
Virtual image
2J
Planer mirror
(G
` 8
` 8
` 8
Lens
[
(J
LCD RK
(a) Hyperboloid Figure 1: Optics of the proposed catadioptrical unit G\
G
utilizing both eyes. A stereopsis view can be achieved within a 60 degree horizontal and 60 degree vertical FOV. In the next section, we show the fundamental optical relation of our HMD and the omnidirectional video-based virtual reality system with our proposed HMD, as well as experimental results of the evaluation of the effectiveness of our HMD.
(G
G` 8
G
2G
G[ G` 8 G
(G
(b) Ellipsoid
2.
OPTICS OF PROPOSED HMD
Figure 1 shows the components and optics of the proposed HMD; consisting of planer, hyperboloidal and ellipsoidal mirrors, a lens and an LCD. The lens is aligned on the focus of the hyperboloidal mirror. The planer mirror between the lens and the hyperboloidal mirror inclines the rays to avoid interference with the observer’s face. The axis of the ellipsoidal mirror is inclined at 50 degrees to avoid the hyperboloidal mirror obstructing the FOV of the observer’s eye. Using this property, the proposed HMD can display an FOV virtual image of about H120×V60 degrees to the observer using simple optics. The proposed optics consists of a hyperboloidal convex mirror and an ellipsoidal concave mirror. Figure 2 shows the hyperboloidal, ellipsoidal mirrors and the geometrical relation of these. In this figure, bold lines indicates the hyperbolidal and ellipsoidal part of HMD optics. Generally, the hyperboloid and the ellipsoid are defined by equations 1-2 and both have two focal points F h+ (0, 0, ch ), F h− (0, 0, −ch ) and F e+ (0, 0, ce ), F e− (0, 0, −ce ), as shown in figure 2. The ellipsoidal coordinate [e x, e y, e z]T is rotated and shifted from the hyperboloidal coordinate [x, y, z]T as described in 3. M is the rotation and translati/on matrix. The focal point F h+ of the hyperboloidal mirror is aligned with the focal point F e+ of the ellipsoidal mirror in the proposed optics.
Figure 2: Characteristics of hyperboloids and ellipsoids
z2 x2 + y 2 − 2 = −1 2 ah bh e 2 x + e y2 z + 2 =1 2 ae be
(1)
e 2
where,
(2)
a2h + b2h = c2h , b2e − c2e = a2e [e x, e y, e z, 1]T = M [x, y, z, 1]T
(3)
On the ellipsoid, the normal vector at point e P e bisects the angle between lines passing through an arbitrary point e P e on the ellipsoid and the focal points e F e+ or e F e− in the ˜ 1 and ellipsoidal coordinate as shown in figure 2-b. Here, e V e˜ V 2 are defined as ray vectors along these lines, the ray e˜ V 1 is reflected to the ray e V˜ 2 as discribed in equations 5, 6. Where, λ1 in equation 5 is derived from equations 2 and 5.
60deg. 180deg. 60deg.
Overlap view 120deg.
Left view
Right view
Figure 3: Binocular alignment and horizontal FOV Figure 4: Prototype HMD
e
V˜ 1 = M V˜ 1 e ˜ a + e F e− P e = λ1 e V e
V˜ 2 = e F e+ − e P e
(4)
Omnidirectional Image sensor
(5)
3D magnetic tracker
(6)
Because the focal point F h+ of the hyperboloidal mirror is set on the focal point F e+ of the ellipsoidal mirror, e V˜ 1 and e˜ ˜ 2 respectively as discribed V 2 are expressed by V˜ 1 and V in equations 4, 7.
Omnidirectional image (1296X1026pix 15Hz)
Transmitter
Head motion (30 Hz)
L ch R ch
On the hyperboloid, as with the ellipsoid, the normal vector at point P h aliquots the angle between lines passing through an arbitrary point P h on hyperboloid and the focal points F h+ or F h− of hyperboloid as shown in figure 2-a. Here, V˜ 2 and V˜ 3 are defined as ray vectors along these lines, the ˜ 3 as discribed in equation 8, ray V˜ 2 is reflected to the ray V 9. Where, λ2 in equation 8 is derived from equations 1 and 8.
Hard drive
Work station Stored omnidirectional image (1024X1024pix 10Hz)
Proposed HMD Transformed binocular image (1024X768pix 20Hz)
Figure 5: Experimental system
with wide angle easily by such simple optics and display the wide filed of view distortionless image. ˜ 2 = M −1e V˜ 2 V P h = λ2 V˜ 2 + F h+
(7)
˜ 3 = F h− − P h V pi = AV˜ 3
(9)
(8) (10)
Based on these characteristics, we designed a new optics of HMD. Practically, we carefully decided the parameter of hyperboloidal and ellipsoidal mirrors, and lens for prototype HMD by using the commercial optical simulation software ZEMAX(Focal software Inc.). A projector consisting of an LCD and a projection lens is set on the focal point F h− of the hyperboloidal mirror. An observer eye is set on the focal point F e− of the ellipsoidal mirror, he can see the image from LCD. We can derive V˜ 3 from an arbitrary ray vector V˜ 1 . The point on LCD plane pi is projected from ˜ 3 as discribed in equation 10. Where Ais the the vector V 2D projection matrix of the LCD plane and lens set on the focal point F h− . Therefore we can completely compensate the optical distortion by using the relation of pi and V˜ 1 as discribed in equations 4-10. From these characteristics of a hyperboloid and an ellipsoid, we can spread and gather rays
Figure 3 shows the layout of both the HMD optical units and the relationship of both the FOVs. The HMD units are aligned with 60 degrees rotated to parallel around the vertical axis, as shown in figure 3. Each catadioptrical unit has 120 horizontal and 60 vertical degrees of FOVs. Therefore, the HMD can cover a 180 horizontal degree × 60 vertical degree FOV including a 60 degree overlap area which gives stereo capability. The proposed HMD displays a stereo image with an overlapped area and a wide FOV including peripheral vision.
3. PROTOTYPE HMD SYSTEM Figure 4 shows a prototype of the catadioptrical HMD system. Usually, pupil position differs between individuals. Therefore, the proposed HMD has a mechanism for adjusting the positions of the optical units according to the pupil position of each observer. Each optical unit is mounted on a helmet by adjusters. Each adjuster consists of three 19 mm adjustable linear movements. Each movement is set orthogonally, and the totally optical unit has 3 degrees of freedom. The LCD module is 1.44 million pixels of a 0.5 inch device that can project a 1024×768 pixels color image. The
Figure 8: Binocular transformed image (H180×V60 degree FOV) Figure 6: Omnidirectional image
Object
Omnidirectional images L
t-d-1 t+d
ed
t
f line o Base ints o p w vie
t+1
Figure 9: Displayed image HMD(H180×V60 degree FOV)
of
the
proposed
Figure 10: Displayed image HMD(H60×V40 degree FOV)
of
common
Camera path
Figure 7: Generation of disparity images by motion stereo criteria
LCD and backlight module are components of a commercial HMD (Sony: Grasstron). A magnetic motion tracker, The Flock of Birds (Ascension inc.), is attached to the top of the helmet to detect the observer’s head motion. The weight of HMD optical unit is 200g. Total weight of prototype HMD including helmet, mount and counter balance weight is 3750g. The HMD was evaluated by showing a recorded video. Figure 5 shows the experimental system for evaluating the proposed HMD. The system consists of the proposed HMD, an omnidirectional image sensor HyperOmni Vision [6], and a graphic work station with a SCSI160 hard disk unit. The input video is captured by the omnidirectional image sensor and recorded by the SCSI160 hard disk unit (1296×1026 pixels, 15 Hz). Figure 6 shows an example of an omnidirectional input image, which then is transformed to LCD images (1024×768 pixels) by a graphic workstation (Octain 2: SGI). Figure 8 shows an example of the transformed LCD images. The lines in figure 8 indicate longitude and latitude in the spherical coordinate system. Note that the transformed LCD images are reversed horizontally due to reflection in the mirrors and are deformed to compensate for the distortion caused by the HMD optics. Figre 9 shows the displayed image of the proposed HMD. The FOV of displayed image is terrifically wider than that of common HMD as shown in figure 10. The image can present the view including the peripheral vision. The system updates the image at 25 Hz for head motion
and at 10 Hz for changes in the environment. Onoe et al. [7] also constructed a similar system. However, their system used a commercial HMD with a normal FOV (about 60 degrees horizontally). HMD have an overlapped area that can present stereo images, as shown in figure 3, and the system can create binocular disparity images from the motion stereo criteria. The input omnidirectional images were captured on a moving camera path. Each image is the image at a different viewpoint as described by the positions of t-d and t in figure 7. The differential of the input image frame d is described by equation 11. In equation 11, r and s, are the frame rate [frame/s] and the sensor moving speed [mm/s] when the input images were captured. L is a base line of both eyes and was set at 68 mm. θd indicates the viewing direction. If θd is close to a right-angle, it means that the viewing direction is parallel to the camera path; the differential d would be big and the images are dissimilar to ideal disparity images. Therefore, this method can be used for cases where the viewing direction is nearly perpendicular to the camera path; we have to switch to monocular images when the direction becomes nearly parallel to the path, if this method is to be applied for tele-presence applications. The left and right eye images, with disparity, can be transformed from different input image frames t-d and t as described in figure 7. Figure 11 shows a sample of the disparity binocular images. At the bottom of this figure are the disparity images on the overlapped area. These images are transformed to perspective images to enable its disparity to
11
Input image sequence
20
10
10
9
0
Latitude [degree]
t Resolution [pixel/degree]
t-d
-10
8
-20
7
-30 6
-40
5 4 -90
-60
-30
0
30
60
90
Longitude [degree]
(a) Resolution attribute against horizontal Binocular disparity images (projectied image) 10 9
0 30 60 90
8 7 6
Longitude [degree]
Resolution [pixel/degree]
11
5 4 -40
Disparity image of overlaped area (perspective)
be easily recognized. The system can display the binocular stereo images with disparity from a monocular omnidirectional image sequence to estimate the stereo capability of the HMD.
rL s cos(θd )
-10
0
10
20
(b) Resolution attribute against vertical Figure 12: Resolution of projected image on the HMD
(11) Vertical
The angular resolution of the HMD is shown in figure 12. Resolution decreases when the depression angle increases, which is the same angular specification as in HyperOmni Vision. Thus, the proposed HMD does not result in any loss of visual information when the omnidirectional input image is transformed to the LCD image.
4.
-20
Latitude [degree]
Figure 11: Binocular disparity transformed image
d =
-30
EXPERIMENTS
We estimated the prototype HMD system and the effect of wide field of view in experiments. At first, we evaluate the displaying image quality, such as focus and resolution, on the prototype HMD. We showed subjects test charts and asked the subjects which size of chart can be recognized to evaluate a resolution visually. We used three types of test charts; vertical, horizontal and Landolt circles as shown in figure 13 and put on the chart each direction of view area. Figure 14 shows the sample of test charts on the LCD images. Five subjects were recruited for this experiment. We confirmed the limit size recognized by the subjects by changing the four different sizes of the chart corresponding to angular resolutions of; 1.25, 2.5, 5.0 and 10 pixel/degree. Figure 15-a-c shows the results of the
Horizontal
Landolt circle
Figure 13: Test charts for confirmation of image quality
average resolution for each chart by the five subjects. Figure 15-d shows the ideal resolution attribute from LCD limitation. The results include multiple effects of the resolution attributes shown in figure 15-d and the optical focus. The area at the center of view area was well focused in comparison with figure 15-a-c and d. On the other hand, the resolution of the peripheral area is not high, especially over the 60 degree area in figure 15-b, because of optical blur. However, as human peripheral vision is insensitive to resolution, we believe that the image quality is sufficiently good for this type of display. Next, we estimated a depth sensitivity of stereopsis. The proposed HMD has an overlapped area. We estimated a stereo capability of the proposed HMD to display stereo disparity images on the overlapped area. We showed subjects synthetic disparity images of a moving stick between two static sticks shown in figure 16. The static sticks are set on
-1 0
-1 0
6 5
10
4 3 20
2 1
9 8 7 6 10
5 4 3
20
2 1 0
0 30
30 -2 0
- 10
0
10
20
30
40
50
60
70
-2 0
80
- 10
0
10
20
Longitude [degree]
30
40
50
60
70
80
Longitude [degree]
(a) Verttical Chart
(b) Horizontal Chart
-1 0
-1 0
0
8 7 6
10
5 4 3
20
2 1
10 9
Latitude [degree]
Latitude [degree]
9
Resolution [pixel/degree]
10 0
8
10
5
7 6
4 3 20
2 1
0 30
Resolution [pixel/degree]
Latitude [degree]
7
10
0
Resolution [pixel/degree]
8
0
Latitude [degree]
9
Resolution [pixel/degree]
10
0 30
-2 0
- 10
0
10
20
30
40
50
60
70
80
Longitude [degree]
(c) Verttical Chart
-2 0
- 10
0
10
20
30
40
50
60
70
80
Longitude [degree]
(d) LCD resolution limit
Figure 15: Result of image quality test by test chart
Table 1: Estimation of depth perception by displaying stereo images Average [mm] standard deviation [mm] 22.35 16.149
Table 2: Estimation of eye movable limit without vignetting Average [deg.] standard deviation [deg.] Horizontal ± 16.6 5.13 Vertical ± 15.5 4.33 Figure 14: Sample of test chart image on LCD
the position 500 mm far from user and the moving stick is moved ± 200 mm back and forward from the static sticks. We measured the position when the subjects think a moving stick stand on the same depth of the static sticks, and defined the position difference as a depth perception error. Table 1 shows the average and standard deviation of depth perception error by ten subjects. The result shows the position error is about 20 mm. On the other hand, the resolution of center of overlapped is about 7 pixel/degree as shown in figure 12. This resolution expresses the disparity corresponding to 20 mm depth difference. Therefore the centered area was enough to focus and the proposed HMD gave the depth perception within the limit of LCD resolution. The prototype HMD has a vignetting problem. The vignetting problem is that the displayed image is dropped out with large observer’s eye rotation. If the observer’s pupil
moves when the eye is rotated, the observer’s pupil obstructs the rays from the HMD. The position of the observer’s eye is also critical because the observational pupil is small on the prototype HMD. Therefore, we estimated the applicable area without vignetting by eye rotation. Ten subjects were studied in this experiment. Table 2 shows the average and standard deviation of the eye rotation angle without vignetting. This result shows that the applicable angle without the vignetting is about 30 degree both vertical and horizontal directions. We also estimate the influence of vignetting caused by a small observational pupil. In this experiment, we assume a tele-presence application and shows the movie captured on the moving car. The prototype telepresence system can display dynamic view accommodated with head motion. We set the two conditions of a constant view and an accommodated view with head motion in this experiment. Table 3 shows the results for the 10 subjects. Table 3 indicates the number of subjects who felt the vignetting under unconscious eye motion. This result shows
Table 3: Influence of vignetting Accommodation Occurence of vignetting with head motion Off 6/10 On 1/10
=OO?
=OO?
Table 4: Estimation of prototype wide FOV HMD Advantage of wide FOV Extension of FOV 10/10 Immersion 10/10 Feeling of movement 10/10
that over half of the subjects claim vignetting in the case of constant view. However, when the view accommodated with head motion was enabled, most subjects were not concerned about the problem. In the case of constant view, the user tracked viewing objects by eye motion. In the case of accommodated view with head motion, the user can track the viewing object by head motion. So the eye motion was decreased and the vignetting problem was moderated. In this way we showed that the current prototype HMD would be applicable for uses involving unconscious eye motion. The improvement of vignetting problem by optics and design is addressed in future work. We compared a wide FOV (H180×V60 degree) and a narrow FOV (H60×V40 degree) assuming common HMDs to estimate the effect of the wide FOV. Figure 10 shows the narrow binocular FOV images for the limited FOV used in the experiment. In this experiment we used images of a moving car. Table 4 shows the results of the comparisons between the wide and common FOV for the subjects. This result shows that the wide FOV improved the factors of reality; an extension of the FOV, immersion and a feeling of movement. We confirmed in practice that the proposed HMD can display H180×V60 wide FOV images using the experimental system presented. From these results, the proposed wide FOV HMD would be effective for use in virtual reality and robotics applications because the wide FOV, including its peripheral vision, contributed to reality and immersion in a virtual environment.
5.
CONCLUSIONS
In this paper, we proposed a catadioptrical HMD composed of hyperboloidal and ellipsoidal mirrors. The proposed HMD can display a 180 degree wide horizontal view with a 60 degree stereopsis view. We constructed a prototype HMD, evaluated its capability, and confirmed the practicality of the wide FOV displayed by the HMD. We also confirmed its advantage of peripheral vision. The proposed HMD has a 60 degree overlap area where it has stereo capability. The prototype system can display stereo images with disparity from image sequences. We confirmed the HMD gave the depth perception to display a stereo disparity images on overlapped area.
=OO?
Figure 16: Setting of depth perception test
The current prototype system is too heavy because the mirrors and adjuster of the prototype HMD are made of aluminum, so we are planning to make the HMD lighter by modifying with plastic parts. We are also planning to evaluate the reality and task efficiency of the system in applications such as tele-presence and virtual reality.
6. REFERENCES [1] J. D. Dickinson and J. A. Leonald: “The role of peripheral vision in static balancing”, EGONOMICS, Vol.10, pp.421-429, 1967. [2] T. A. Furness: “Creating better virtual worlds”, Technical Report HITL-M-89-3, 1989. [3] M. Takahashi, K. Arai and K. Yamamoto: “Wide field of view using a 4LCD HMD is effective for postural control”, Second International Conference on Psychophysiology in Ergonomics, 1998. [4] D. G. Caldwell, K. Reddy, O. Kocak and A. Wardle: “Sensory Requirements and Performance Assessment of Tele-Presence Controlled Robots”, Proc. IEEE Int. Conf. Robotics and Automation, pp.1375-1380, 1996. [5] M. Inami, N. Kawakami, T. Maeda, Y. Yanagida and S. Tachi: “A Stereoscopic Display with Large Field of View Using Maxwellian Optics”, Proc. Int. Conf. Artificial Reality and Tele-Exsistance, pp.71-76, 1997. [6] K. Yamazawa, Y. Yagi and M. Yachida: ”New real-time omnidirectional image sensor with hyperboloidal mirror”, Proc. 8th Scandinavian Conf. Image Processing, Vol. 2, pp. 1381-1387, 1993. [7] Y. Onoe, K. Yamazawa, H. Takemura and N. Yokoya: “Telepresence by real-time view-dependent image generation from omnidirectional video stream”, Computer Vision and Image Understanding, Vol. 71, No. 2, pp.154-165, 1998.
