High-resolution optical see-through multi-focal ... - OSA Publishing

High-resolution optical see-through multi-focalplane head-mounted display using freeform optics Xinda Hu and Hong Hua* 3DVIS Lab, College of Optical Sciences, University of Arizona, 1630 East University Boulevard, Tucson, Arizona 85721, USA * [email protected]

Abstract: Conventional stereoscopic displays force an unnatural decoupling of the accommodation and convergence cues, which may contribute to various visual artifacts and have adverse effects on depth perception accuracy. In this paper, we present the design and implementation of a high-resolution optical see-through multi-focal-plane head-mounted display enabled by state-of-the-art freeform optics. The prototype system is capable of rendering nearly-correct focus cues for a large volume of 3D space, extending into a depth range from 0 to 3 diopters. The freeform optics, consisting of a freeform prism eyepiece and a freeform lens, demonstrates an angular resolution of 1.8 arcminutes across a 40-degree diagonal field of view in the virtual display path while providing a 0.5 arcminutes angular resolution to the see-through view. © 2014 Optical Society of America OCIS codes: (120.2040) Displays; (120.2820) Heads-up displays; (120.4570) Optical design of instruments; (220.1250) Aspherics; (220.3620) Lens system design; (330.7338) Visually coupled optical systems.

References and links 1.

S. Suyama, S. Ohtsuka, H. Takada, K. Uehira, and S. Sakai, “Apparent 3-D image perceived from luminancemodulated two 2-D images displayed at different depths,” Vision Res. 44(8), 785–793 (2004). 2. K. Akeley, S. J. Watt, A. R. Girshick, and M. S. Banks, “A stereo display prototype with multiple focal distances,” ACM Trans. Graph. 23(3), 804–813 (2004). 3. G. D. Love, D. M. Hoffman, P. J. W. Hands, J. Gao, A. K. Kirby, and M. S. Banks, “High-speed switchable lens enables the development of a volumetric stereoscopic display,” Opt. Express 17(18), 15716–15725 (2009). 4. S. Liu, H. Hua, and D. Cheng, “A novel prototype for an optical see-through head-mounted display with addressable focus cues,” IEEE Trans. Vis. Comput. Graph. 16(3), 381–393 (2010). 5. X. Hu and Hua, “Distinguished student paper: a depth-fused multi-focal-plane display prototype enabling focus cues in stereoscopic displays,” in SID Symposium Digest of Technical Papers (2011), Vol. 42, pp. 691–694. 6. S. Liu and H. Hua, “A systematic method for designing depth-fused multi-focal plane three-dimensional displays,” Opt. Express 18(11), 11562–11573 (2010). 7. S. Ravikumar, K. Akeley, and M. S. Banks, “Creating effective focus cues in multi-plane 3D displays,” Opt. Express 19(21), 20940–20952 (2011). 8. K. J. MacKenzie, D. M. Hoffman, and S. J. Watt, “Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control,” J. Vis. 10(8), 22 (2010). 9. X. Hu and H. Hua, “Design and assessment of a depth-fused multi-focal-plane display prototype,” J. Display Technol. 10(4), 308–316 (2014). 10. D. Cheng, Y. Wang, H. Hua, and M. M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009). 11. Q. Wang, D. Cheng, Y. Wang, H. Hua, and G. Jin, “Design, tolerance, and fabrication of an optical see-through head-mounted display with free-form surface elements,” Appl. Opt. 52(7), C88–C99 (2013).

1. Introduction The multi-focal-plane (MFP) display is one of the technologies proposed to minimize and eliminate the accommodation and convergence conflict — a fundamental problem in conventional stereoscopic 3D displays [1–5]. By presenting multiple image planes at different

#209099 - $15.00 USDReceived 31 Mar 2014; revised 20 May 2014; accepted 21 May 2014; published 30 May 2014 (C) 2014 OSA 2 June 2014 | Vol. 22, No. 11 | DOI:10.1364/OE.22.013896 | OPTICS EXPRESS 13896

accommodation distances to the viewer and aligning them appropriately for each eye, threedimensional scenes correctly rendered by MFP displays can correctly stimulate the eyes to accommodate at the same distances to which the eyes converge. The accommodationconvergence response when viewing MFP display is very similar to the one in viewing natural environments where the convergence cue and accommodation cue are matched. Therefore MFP displays have demonstrated great potentials in reducing perceptual distortions, improving visual performance and reducing visual discomfort and fatigue. Many valuable works have advanced the MFP system designs as well as depth-rendering algorithms to improve image quality and to accurately cue eye accommodation within discrete focal planes [1–9]. It is still technically challenging, however, to design a compact, high frame rate, and high performance MFP display with adequate number of focal planes. The challenge is further aggregated for the use of this technology in see-through headmounted displays where viewers such as in medical and military applications can benefit most from accurate depth perception and improved visual performance. Recently, we presented the implementation of a depth-fused six-focal-plane prototype system — the first system capable of displaying 6-focal-plane depth-fused 3D scene extending from 3 diopters to infinity at a flicker-free speed [5, 9]. However, built from offthe-shelf lenses, the preliminary prototype suffered from unsatisfactory image resolution and contrast, and most importantly it lacked the optical see-through capability, which significantly limited its usefulness in general 3D display application as well as fundamental depthperception vision research. In this paper, we present an optical see-through MFP-HMD system, which is a substantial improvement over our previous preliminary implementation. This system utilizes state-of-theart freeform design technique and elegantly couples freeform prism eyepieces with rotationalsymmetrical lens systems. As a result, the prototype not only achieves high quality imagery across a large 3D volume for the virtual display path but it also maintains better than 0.5 arcminutes visual resolution of the see-through view. This paper is structured as follows. In section 2, the system components and functions are described and overall system design approaches as well as results are presented. In section 3, the system prototype is presented with qualitative experiments demonstrating the 3D rendering and see-through optical quality. 2. Design approach 2.1 System functions and layout The overall optical system layout of a monocular MFP display is shown in Fig. 1. Each monocular setup consists of two parts: the composite optical see-though eyepiece and the image generation subsystem (IGS). Figure 1(a) shows the optical layout of the right-eye setup looking from the top, where the freeform eyepiece folds the virtual display optical path to the right and the rotational symmetric IGS, illustrated in Fig. 1(b), stands vertically to the temple side of the human head. Rays from the center and edge field angles are drawn with different colors to show conjugated pupil and image locations more clearly. The composite eyepiece based on freeform optics is the core component enabling the successful integration of multi-focal-plane display and OST-HMD system. It consists of a wedge-shaped freeform eyepiece, a freeform see-through compensator lens and a cylindrical lens. The freeform eyepiece, formed by three freeform surfaces S1, S2 and S3, serves as a single-element HMD eyepiece for magnifying the intermediate image created by the IGS and presenting the virtual content to the viewer. The light from the IGS propagates into the freeform eyepiece through S3, after total-internally reflected on S1 and partially reflected by the beamsplitter coating on S2, exits the eyepiece structure and enters the viewer’s eye. Unlike conventional HMD eyepiece with rotational symmetric optical surfaces, which requires components to be collinear with the optical axis and uses a separate beamsplitter to fold the system, the freeform prism eyepiece folds the light path within a single plastic


element so that not only the system weight, size and complexity are greatly reduced, the system at the same time achieves a large field of view, high image quality and very desirable long eye-relief. The optical see-through capability of the system is enabled by cementing the freeform eyepiece with a compensator lens, formed by two freeform surfaces S2 and S4, to correct the undesirable aberrations and distortions introduced by the prism eyepiece. In this prototype, a cylindrical lens was further incorporated into the optical see-through path design, which helps to correct both aberration-induced blur and distortion of the real-world view to a negligible level.

Fig. 1. (a) top-view optical layout of the right-eye module of the MFP-HMD prototype. (b) detailed layout of the Image Generation Subsystem (IGS).

The second part of the system, the IGS, with detailed optical layout in Fig. 1(b), achieves the core function of generating the multi-focal-plane contents. It consists of a high-speed digital micromirror device (DMD) microdisplay by Texas Instruments, a deformable membrane mirror device (DMMD) and relay lens groups. The IGS is essentially a zoomed relay system based on an active optical element (the deformable mirror) whose optical power can be electronically controlled at a high speed as fast as 1 kHz. The DMD microdisplay, illuminated by an RGB LED light source, is first magnified by two field lenses and then relayed by a double-pass double-telecentric lens group, forming an intermediate image between the IGS and the eyepiece The DMMD is placed at the stop of the relay lens group and its surface shape can change from a plane to a concave parabola when voltage is applied. The change of optical power on the DMMD rapidly changes the axial location of the relayed intermediate image and in turn creates a stack of addressable virtual focal planes with different distances to the viewer in the viewing space. The depth volume of the 3D scene appearing to the viewer and the spacing of the focal planes can be controlled by adjusting the voltage control on the DMMD. By synchronizing the voltage applied on the deformable mirror, the contents being rendered on microdisplay and the RGB LED illumination at a high speed, multiple focal planes are time-multiplexed and different virtual contents can be generated at their correct depths, thus forming a continuous true-3D space even with the monocular setup [5].


2.2 System optical design In the virtual display path, the design goal of a MFP display is to present to the viewer a large volume of high-quality 3D contents, rendered with correct accommodation cues. This requires the display to have a wide field of view as well as a large addressable depth range of the focal planes. In this prototype, we’ve selected the virtual display field of view to be 40° diagonally, or 33° × 25° in the horizontal and vertical directions, respectively. The source of the virtual image is the 0.7-inch DMD microdisplay with a pixel resolution of 1024 × 768 (XGA) and a pixel size of 14µm. Therefore in the visual space, the virtual display equivalently affords an angular resolution of 1.8 arcminutes per pixel. As described in Section 2.1, a group of lenses relays the DMD microdisplay image to the freeform eyepiece. Since the double-pass doubletelecentric relay group provides no magnification, to help reduce the optical power of the freeform eyepiece, i.e. its design difficulty, two field lenses were designed to pre-magnify the image as well as to correct aberrations (mainly lateral chromatic aberration). On the other hand, in this prototype the focal-plane depth can be addressed from 33cm to infinity in front of the viewer, in a total range of 3 diopters. This depth range is optically magnified from the 10mm-diameter DMMD, which only has an adjustable optical power range of approximately 1.2 diopters. From the first-order design analysis in [9], we have identified the trade-off between the addressable depth range and the exit pupil diameter (EPD). To magnify the focal-plane depth range to 3 diopters from DMMD’s 1.2 diopters, the focal lengths of the freeform eyepiece and the half relay-lens group (the two doublet lenses and a singlet lens close to DMMD) must maintain a ratio of 0.632 ( f eyepiece / f halfrelay ) . This in return limits the maximum achievable non-vignetted EPD of the display to 6.3mm, since the DMMD is conjugated to the exit pupil in this pupil-forming display. Taking into account many system design factors, such as the placement of IGS relative to the head, the desired long 25mm eye-clearance to accommodate most viewers wearing eyeglasses and the overall optical design difficulties, we chose the focal length of the freeform eyepiece (formed by S1-3) to be 27.5mm and a F-number of 4.3. The design form of this freeform prism eyepiece is well-known and its basic design methods can be found in many literatures such as Refs. [10] and [11]. In this design, the three freeform surfaces are described by plane-symmetric 10th-order XY-polynomials and they are decentered and tilted to form the prism structure on PMMA plastic. Besides typical requirements of wide FOV, large pupil and high image contrast, the optical design of the freeform eyepiece and the overall design of our MFP-HMD system face several unique challenges not in typical HMD designs where freeform eyepiece directly magnifies a microdisplay. The first challenge results from the coupling between the freeform optics with the rotational symmetric IGS. The advantages of freeform eyepiece came from breaking the system symmetry and folding the optical path with freeform surfaces. However this also introduces a substantial amount of undesired aberrations not presented in rotationalsymmetric systems. Additionally, due the pupil-forming nature of the overall optical system and the fact that the IGS is mainly composed of off-the-shelf lenses, the freeform prism needs to be strategically optimized to match the pupil and to correct aberrations of the IGS, pushing the limits of the prism eyepiece structure. Secondly, the depth of focal-plane can be addressed within a large range of 3 diopters by shifting the intermediate image plane with the zoomed IGS. Therefore the through-focus performance of the optical system must be optimized to ensure high image quality on focal planes of any depth. Last but not the least, in order to properly apply depth-fusion technique [6–9] in MFP-displays which requires that the fusing pixels on different focal planes maintain constant angular magnification across the entire depth range of interest, the optical system needs to be optimized to ensure high telecentricity and low distortion variation across the 3 diopters of depth range.


In the optical see-through path, the freeform compensator (formed by S2 and S4, described with 10th-order XY-Polynomials) and an off-the-shelf cylindrical lens were designed together to achieve a see-through field of view of 50° × 45° in the horizontal and vertical directions, respectively. The see-through field of view is much larger than the virtual display to minimize the obstruction and “tunnel-feeling” to the real-world view. The central 40° of FOV, where virtual 3D contents can augment the real-world view, was optimized to have MTF values of about 50% at human normal visual acuity of 1 arcminute and minimal distortion so that the viewer’s vision of the real-world is not degraded. Although the seethrough path does not have a confined eye box, the compensator optics was optimized for a large 10mm pupil to ensure high visual quality and viewing comfort with eye rotation. 2.3 Optical design results The optical design results of the virtual display path and optical see-through path are presented separately in this section. Figure 2(a) shows the full-field display of the polychromatic modulation transfer function (MTF) of the virtual display path evaluated with 3mm centered pupil for spatial frequencies of 20 lp/mm and 35 lp/mm, where 35 lp/mm is the Nyquist frequency of the DMD microdisplay. The sizes of the circles are proportional to the MTF values. Across the entire 40° of FOV the image contrast is very high and uniform, with an average MTF of 0.58 at 20 lp/mm and 0.36 at 35 lp/mm. Figure 2(b) shows the distortion grid of the virtual display. The maximum distortion of the virtual display is kept very low, less than 5% at the edge field of 40°. With very strict telecentricity of less than 0.15° at the intermediate image space, the maximum pixel shift across the focal planes is less than 0.25 pixels, making fusing multi-focal-plane images an ease. As shown in Fig. 2(c), the optical performance through the 3-diopter focus range was also very well balanced. Calculated from the average RMS spot sizes, it varies from + 3% to −8% across the 3 diopter range with respect to the nominal performance on dioptric mid-plane of 1.5 diopters.

Fig. 2. (a) Full-field MTF plots of the virtual display; (b) distortion grid of the virtual display; (c) Through-focus performance across 3 diopters of accommodation range.

Figure 3(a) shows the MTF full-field plot of the see-through optical path with a 3mm centered pupil for the spatial frequency of 0.5 cycle/arcminute and 1.0 cycle/arcminute, which corresponds to an angular resolution of 1 arcminute and 0.5 arcminute in the visual space, respectively. The performance across the 40° is almost all diffraction-limited by the eye pupil. The see-though view has very high contrast, with an average MTF value of 0.49 at human #209099 - $15.00 USDReceived 31 Mar 2014; revised 20 May 2014; accepted 21 May 2014; published 30 May 2014 (C) 2014 OSA 2 June 2014 | Vol. 22, No. 11 | DOI:10.1364/OE.22.013896 | OPTICS EXPRESS 13900

normal visual acuity of 1 arcminute and 0.16 at 0.5 arc minute. Figure 3(b) shows the distortion grid of the see-through view. The distortion caused by the composite eyepiece is negligible with less than 2% pincushion at the edge fields of the temporal side.

Fig. 3. (a) Full-field MTF plots of the see-through path; (b) distortion grid of the see-through view showing minimal distortion at the corners of the temporal side.

3. System prototype Figure 4(a) shows the freeform composite eyepiece assembly in the mounting hardware. The freeform optics was prototyped by diamond-turning on PMMA plastic. The 3D geometry of the as-built binocular bench prototype is shown in Fig. 4(b). The mechanical mountings for the image generation subsystem of the right eye module are hidden to show the lens assembly more clearly.

Fig. 4. (a) the freeform composite eyepiece assembly; (b) 3D model of the as-built binocular bench prototype with part of the mechanical mountings moved to show the optical lenses.

Figure 5 shows photos of the multi-focal-plane 3D scene rendered by the virtual display and captured with a camera placed at the exit pupil position. Six focal planes were timesequentially formed at 3.0 diopters, 2.4 diopters, 1.8 diopters, 1.2 diopters, 0.6 diopters and 0.0 diopter respectively, by controlling the voltage applied on the DMMD. Each focal plane displays a different part of the 3D scene whose depth are near that focal plane, and by incorporating depth-weighted luminance blending technique [6–9], a large 3D space can be rendered continuously with only a small number of focal planes. The 3D scene displayed in #209099 - $15.00 USDReceived 31 Mar 2014; revised 20 May 2014; accepted 21 May 2014; published 30 May 2014 (C) 2014 OSA 2 June 2014 | Vol. 22, No. 11 | DOI:10.1364/OE.22.013896 | OPTICS EXPRESS 13901

Fig. 5 consists of a green floor grid extending from 3.0 diopters to 0.6 diopters, a green wall grid at 0.6 diopters with UofA logo on it and a moveable resolution target with bar targets of varying spatial resolutions. The resolution target was intentionally made to have a constant angular size despite of its rendered distance in the 3D scene. However its axial placement can be identified by looking at its blocking of the floor grid.

Fig. 5. Three-dimensional scenes displayed with correct focus cues by depth-fusing 6 discrete focal-planes placed at 0.0D, 0.6D, 1.2D, 1.8D, 2.4D and 3.0D, where: in (a) and (b) the resolution target is rendered at a near distance of 2.7D using two focal planes of 2.4D and 3D; in (c) and (d) the resolution target is rendered at a far distance of 0.9D using two focal planes of 0.6D and 1.2D.

These results clearly demonstrated superb image quality from the optical design of the freeform lenses, as well as the true-3D experience with correct focus cues enabled by the multi-focal-plane display technology. In Fig. 5(a), the resolution target, virtually placed at 2.7 diopters distance from the eye, was rendered with the depth-fusing technique by the two physical focal planes at 2.4 diopters and 3.0 diopters. When the camera was focused at the near target, the far end of grid floor and the wall at 0.6 diopters clearly show natural, differential out-of-focus blur, where the resolution target and the near end of the floor grid show clear in-focus image. In Fig. 5(c), the resolution target, virtually placed at 0.9 diopters distance from the eye, was rendered with depth-fusing technique by the two focal planes at 0.6 diopters and 1.2 diopters. With the camera focused at the far target, the back wall and the logo were in sharp focus and the floor grid extending from far to near gradually blurred out. Figures 5(b) and 5(d) are magnified views of the central part on the resolution target, corresponding to 5(a) and 5(c), respectively. The center smallest bar targets correspond to the cutoff spatial frequency of 2 pixels/cycle. Attributed to the telecentric design, the results in #209099 - $15.00 USDReceived 31 Mar 2014; revised 20 May 2014; accepted 21 May 2014; published 30 May 2014 (C) 2014 OSA 2 June 2014 | Vol. 22, No. 11 | DOI:10.1364/OE.22.013896 | OPTICS EXPRESS 13902

Figs. 5(b) and 5(d) clearly demonstrated that the bars of the resolution target fused by pixels from two spatially-separated focal planes (i.e. with a 0.6 dioptric spacing) achieved high contrast. The nearly perfect overlay of pixels from separated focal planes allows that the depth-fusion technique can be directly applied to the multi-focal plane images without calibrations of pixel mapping. From the two magnified views, good through-focus performance can also be seen without noticeable difference between targets rendered at near or far distances. Figure 6(a) shows an image of the see-through view captured with a camera for a resolution target placed at approximately 75cm away, covering the FOV of approximately 40 degrees diagonally. The image of the resolution target demonstrated superb quality and negligible distortion. Figure 6(b) shows an image of the see-through view for viewing a standard Snellen eye chart placed at 20 feet, where Group 8 (20/20 vision) corresponds to the angular resolution of 1 arcminute and Group 11 (20/10 vision) corresponds to the angular resolution of 0.5 arcminute. The camera was configured to have an angular resolution of 0.23 arcminutes per pixel. From the image, all letters from Group 8 to Group 11 are readable and have very good contrast. These results clearly demonstrated that little degradation and negligible distortion on the real-world view is introduced by the see-through optics.

Fig. 6. (a) Camera captured see-through view of 40-degree resolution target; (b) Standard eye chart placed at 20ft seeing through the composite freeform eyepiece.

4. Conclusion In this paper, we presented the optical design and implementation of an optical see-through multi-focal-plane head-mounted display prototype. By incorporating the state-of-art freeform optics, we substantially improved our previous implementation of multi-focal-plane display. Not only the virtual display quality across the large 3D volume is improved, most importantly the freeform optics grants the system optical see-through capability with excellent image quality. Although current prototype is implemented as a bench prototype and the overall system does not have a compact size and light weight, future works can be done to further improve the optics, electronics as well as mechanics for a more ergonomic head-mounted display platform. Acknowledgments This work is funded by National Science Foundation Grant Awards 1115489 and 0915035.
