Design of a foveated imaging system using a two-axis MEMS mirror

Design of a foveated imaging system using a two-axis MEMS mirror Sheng Liu, Craig Pansing, and Hong Hua* 3DVIS Lab, College of Optical Sciences, University of Arizona, Tucson, AZ, USA 85721 ABSTRACT We present a novel foveated imaging system which consists of three major components: a foveated imager, a peripheral imager, and a 2D scanning sub-system. The foveated imager captures a narrow field of view (FOV) with high angular resolution to mimic the foveated region of interest (FRoI), and the peripheral imager captures a wide FOV with low angular resolution to provide the peripheral region for context. The scanning system, implemented with a single-element two-axis MEMS mirror, is capable of sweeping the FRoI across the entire FOV at a maximum speed of 100Hz. A bench prototype and experimental results are presented as verifications of our optical design concept and feasibility.

Key words: foveated imaging, multi-resolution, micro-electro-mechanical system (MEMS) 1.

INTRODUCTION

High-resolution, wide field of view (FOV), and video rate image capturing is highly desirable in many applications such as tele-conferencing, surveillance, and robot navigation [1]. However, conventional imaging techniques have spread the finite number of pixels evenly across the entire FOV. Consequently, the FOV and the angular resolution of these conventional digital cameras are limited by the choice of imaging sensors and optics. Using a 640×480 NTSC CCD detector as an example, a camera with 60° FOV only offers an angular resolution of about 5.6 arc minutes at maximum. Furthermore, even though higher resolution sensors become available, real-time processing of high resolution images and fast transmission of them over a network are still challenging.

2.

FOVEATED IMAGING SYSTEM

By exploring properties of the human visual system, foveated imaging techniques have emerged as alternatives to traditional imaging approaches [2]. These means, by making tradeoffs between the viewer’s primary and peripheral regions of interest, have been used for image processing, video coding [3] and stereoscopic displays [4] to accelerate the image processing and to reduce the bandwidth requirement. Several approaches have been explored to achieve foveated imaging/display systems. Etienne-Cummings [5] presented a bionics approach by designing CCD detectors analogous to the structure of human retina. Martinez and Wick presented foveated imaging systems in which a transmissive [6] or a reflective [7] liquid crystal (LC) spatial light modulator (SLM) is used to dynamically correct the aberrations at the foveated region of interest (FRoI). Applying a similar concept to the foveated imaging techniques, several researchers have explored the design of foveated displays. For example, Godin [8] described a system using multiple projectors to achieve a 3D dual-resolution display. Rolland [9] also presented the design of head-mounted displays with high-resolution insets corresponding to FRoIs. In this paper, we present a dual-resolution foveated imaging system, which is not only easily controlled, but also lowcost. The system consists of three major components: a foveated imager, a peripheral imager, and a 2D scanning subsystem. The foveated imager captures a narrow FOV with high angular resolution to mimic the FRoI. The peripheral imager, which can be implemented with low-resolution detectors and low-cost optics, captures a wide FOV with low angular resolution to provide the peripheral region for context. The scanning system, implemented with a 2D scanning MEMS mirror, is capable of sweeping the high-resolution FRoI across the entire low-resolution visual field at a speed up to 100Hz. Furthermore, by dynamically capturing the narrow FOV of the FRoI, our foveated imaging system could be ________________________________________ *Email: [email protected]; Tel: 520-626-8703

International Optical Design Conference 2006, edited by G. Groot Gregory, Joseph M. Howard, R. John Koshel, SPIE Vol. 6342, 63422W, © 2006 SPIE-OSA · 0277-786X/06/$15 · doi: 10.1117/12.695275 SPIE-OSA/ Vol. 6342 63422W-1

exempt from using expensive high resolution detectors while maintaining high angular resolutions. In one possible mode of the system, the peripheral imager is mainly used for detecting and aiming the FRoI for target recognition and task engagement.

3. OPTICAL DESIGN 2.1 Design Requirements To achieve dual-resolution imaging, two CCD imagers are used, one for capturing the FRoI with high angular resolution and small FOV, and the other for the peripheral imager with low angular resolution and large FOV. Furthermore, it is critical to ensure the condition of single-viewpoint imaging, in which the foveated and peripheral imagers share the same entrance pupil. The single-viewpoint imaging condition allows the mosaic of the two images independent of the scene depth and requires minimal post-processing. Additionally, the position of the scanning MEMS mirror is required to coincide with the intermediate pupil of the system such that the pupil position of the imaging system is independent of the scanning process. Finally, for reasons of compactness, the two imagers are combined together by sharing their optical paths and components as much as possible. 2.2 Schematic Design With the major considerations discussed above, the schematic design of the foveated imaging system is shown in Figure 1. The foveated imager consists of two groups of lenses: an objective lens (b) and a collimation lens (f). The foveated imager is conceptually similar to a Keplerian telescope design. However, as light reflected by the MEMS (g) is sent back to the collimation lens, the same lens is also used as the imaging lens of the foveated imager for compactness. Though the MEMS can be utilized as the physical stop of the system, the projection area of the MEMS surface varies with its scanning angle, which consequently causes the dependence of the effective aperture F-number of the system. Instead, a physical stop is placed at the front focal plane of the front lens group (b), which makes the system telecentric between the objective lens (b) and the collimation lens (f). The MEMS mirror is placed at the back focal point of the collimation lens and is conjugate to the physical stop (a). By tilting the MEMS mirror instantaneously towards the region of interest, the collimated off-axis rays are redirected onto axis and focused on the foveated imaging CCD (d). The peripheral imager shares the stop and the objective lens with the foveated imager. An intermediate image with full FOV is formed and refocused onto the peripheral imaging CCD (i) by the lens (h). In order to reduce the light loss through beamsplitting, especially for the foveated imager in which the beam is split twice, a polarizing beam splitter (PBS) together with a quarter wave plate [10] is utilized as shown in Figure 1. It theoretically results in a 50/50 beam splitting between the foveated and peripheral image without losses for both beams and leads to a very compact design as well. (i) peripheral imaging CCD (h) positive lens

(a) stop

(b) (c) polarized objective beam splitter lens

(e) quarter wave plate (d) foveated imaging CCD

fb

(f) collimation lens

fb ff ff Fig. 1. Schematic diagram of the foveated imaging system

SPIE-OSA/ Vol. 6342 63422W-2

(g) MEMS

(a)

19.23 MM

ORA

DIFFRACTION LIMIT T 0.0 FIELD ( 0.00 ) WAVELENGTH R 656.3 NM T 0.7 FIELD (16.00 ) 587.6 NM R 486.1 NM T R 1.0 FIELD (22.50 )

01-Jun-06

WEIGHT 1 1 1

DEFOCUSING 0.00000

1.0

FIELD POSITION 0.00, 1.00 0.000,22.50 DG

0.9

MODULATION

0.8 0.7 0.00, 0.69 0.000,16.00 DG

0.6 0.5 0.4 0.3 0.2

0.00, 0.00 0.000,0.000 DG

0.1 6.00

12.00

18.00

SPATIAL FREQUENCY (CYCLES/MM)

.876E-01 MM

24.00 DEFOCUSING

(b)

0.00000

(c)

Fig. 2. (a) Optical layout of the peripheral imager (b) MTF and (c) Root-mean-square spot diameter

2.3 Optical System Design Based on the conceptual layout in Figure 1, we designed a foveated imaging system, in which the image quality of the foveated and the peripheral paths was optimized separately using CODE V [11]. Figure 2 and Figure 3 show the lens layouts and the optical performance of the peripheral imager and the foveated imager, respectively. Both of the imagers share an ERFLE eyepiece and the system stop. The stop is placed at the front focal plane of the eyepiece. The telecentric intermediate image is formed near the back focal plane of the eyepiece. A beam splitter behind the eyepiece splits the rays into the peripheral beam and the foveated beam by 50/50. The major components in the peripheral beam are two singlets, which refocus the intermediate image onto a 2/3” CCD. The major components of the foveated imager consist of a doublet and a MEMS mirror. The MEMS mirror is placed at the conjugate position of the system stop, which is the back focal plane of the doublet as well. The MEMS mirror folds the foveated beam collimated by the doublet which then focuses onto the 1/4” CCD. As shown in Figure 2(b), the modulation transfer function (MTF) of the peripheral imager drops to 20% near 24lp/mm, while the spatial resolution of the 2/3” CCD used in the peripheral imager is 36 lp/mm. It is worth pointing out that the

SPIE-OSA/ Vol. 6342 63422W-3

20.83 MM

20.83 MM

(a) DIFFRACTION LIMIT T 0.5 FIELD (12.50 ) WAVELENGTH R 656.3 NM T 0.8 FIELD (17.50 ) 587.6 NM R 486.1 NM T R 1.0 FIELD (22.50 )

25-May-06

WEIGHT 1 1 1

ORA

DEFOCUSING 0.00000

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

MODULATION

MODULATION

ORA

(b)

18.00

36.00

54.00

25-May-06

WEIGHT 1 1 1

DEFOCUSING 0.00000

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

72.00

SPATIAL FREQUENCY (CYCLES/MM)

DIFFRACTION LIMIT T -1.0FIELD (-5.00 ) WAVELENGTH R 656.3 NM T 0.0 FIELD ( 0.00 ) 587.6 NM R T ) 486.1 NM R 1.0 FIELD ( 5.00

18.00

36.00

54.00

72.00

SPATIAL FREQUENCY (CYCLES/MM)

(c) (d) Fig. 3. Lens layouts and optical performance of the foveated imager when MEMS tilts by -4.25° (a) and (c) and MEMS tilts by 0° (b) and (d)

MTF of the peripheral beam can be further improved by adding more elements between the two singlets, as refer to Figure 2. However, since the peripheral image is simply to provide the low-resolution peripheral context, the high quality performance is unnecessary. As shown in Figures 3(c) and 3(d), the MTF of the foveated imager remains above 20% near 72lp/mm which corresponds to the spatial resolution of the foveated CCD. We optimized the foveated beam to offer similar high image quality both at the center FOV with the MEMS tilted at 0° angle and at the peripheral FOV with the MEMS tilted at -4.25° angle. This prevents the image quality of the foveated imager from degrading while the MEMS mirror scans the FRoI across the entire FOV. Table 1. Optical performances of the foveated and the peripheral imagers

Peripheral imager Foveated imager

EPD

EFL

F/#

1.5mm 1.5mm

13.2mm 22.9mm

F/9 F/15

RMS spot diameter 50µm 10µm

FOV 45º 10º

Effective angular resolution 13arcmin 1.5arcmin

Image sensor 2/3” CCD 1/4” CCD

In summary, Table 1 lists the optimized first-order data and basic performance for both imagers. The full FOV captured by the peripheral imager is 45º, while approximately 10º FOV out of the full 45º is instantaneously captured by the foveated imager with an effective angular resolution of 1.5arcmin, which is optimized to be about 9 times better than that of the peripheral imager with full FOV. The MEMS tilts to scan the foveated imager against the FOV of the peripheral imager. Therefore, the foveated imager is capable of capturing equivalently the full FOV at high resolution. To illustrate how the dual-resolution foveated imaging approach is advantageous, let us consider the non-foveated case in which the same angular resolution of 1.5 arc minutes is desired for the 45º FOV peripheral image captured by a 2/3” CCD. First of all, it requires for a detector with at least 1800 pixels in diagonal, which is not only costly but also makes the optical design difficult. Furthermore, it is challenging to process and transmit such a high-resolution image in realtime, since the total number of pixels increases by about 130% compared with the sum of the foveated and peripheral images. In the foveated imaging scheme, it is worth further noting that only the foveated image needs to be processed or transmitted in real-time for task analysis. Finally, our analysis of the system shows, by tilting the MEMS to steer collimated off-axis lights back onto axis, the performance of the foveated beam can be enhanced, compared with the case when the same FOV is captured without the scanning mechanism.

3. EXPERIMENTS Using off-the-shelf optical components, we built a bench prototype of the foveated imaging system, which is shown in Figure 4. The prototype approximately matches the optimized design summarized in Table 1. A 28mm EFL eyepiece

SPIE-OSA/ Vol. 6342 63422W-4

(i) (h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Fig.4. The bench prototype (Inset left: Bare MEMS chip, and Inset right: pupil conjugation on the MEMS surface)

Fig. 5. Real MEMS chip compared to a quarter

assembly was used as the objective lens (b), a 60mm EFL doublet is used as the collimation and imaging lens (f), and a 50mm EFL two-element lens group is used as the lens group (h). Matching the FOV partitions for the foveated and the peripheral imagers, a 1/4” CCD (768×494, SNOY) is utilized to take the foveated image and a 2/3” CCD (640×480, Pulnix) for the peripheral image. Figure 5 shows the snap shot of the MEMS chip (Model TALP1010 by Texas Instruments), which has an elliptical active area (3.2mm short axis and 3.6mm long axis) and ±5º dual-axis mechanical tilt capability. The MEMS mirror is compactly implemented into the optics and is utilized to sweep the 10° foveated FOV across the full visual field. Alpha/Beta (degree) vs. U (V) 6 alpha = 1.0982*U - 0.0022 2 R =1

4 2

beta = 0.9757*U + 0.0134 2 R =1

0 -5

-4

-3

-2

-1

0

1

2

3

4

5

-2

-4 -6

Fig. 6. MEMS tilt angle alpha (x) and beta (y) vs. Applied Voltage curves

SPIE-OSA/ Vol. 6342 63422W-5

•1a4. r---J b

a

•

:r -

Mtldentutflon

0,

-.

2,,

MriISC'6s, '

'.tMOOe1

1's"

1bkSMdI

/f/ f,//,///,/'/,c.,Wfl.tit#'// P'* 4' Mwt Ctrns*ry

y/(7; .../,tl

oewbc

9ancu

hI

-2.9º

QY1T

j1

-3.6º

'IL

MEMS tilts -4.3º

Iui

Fig. 7. Left: low-resolution peripheral image with 46.5° diagonal FOV Right: high-resolution foveated image with 10° diagonal FOV

'I.

-1.5º

I

-2.2º

1'

I-

-0.7º

I

0.0º 0.7º

2.2º

ii

I'

10º

II

1.5º

2.9º 3.7º 4.3º

46.5º

Fig. 8. Superposition of high-resolution foregrounds and the low-resolution background by operating MEMS at different angles

As shown in Figure 6, the MEMS chip is calibrated prior to take the foveated images. Analog signals from -5V to +5V are applied to both x- and y-axis of the MEMS mirror. In x and y directions, the tilt angle (alpha in x, beta in y) of the MEMS chip has very good linearity across the entire tilt range albeit a slightly difference in retardence. The compactness and ease of use of the MEMS chip promise a compact design of the foveated imaging system prototype.

SPIE-OSA/ Vol. 6342 63422W-6

/IiIv)

QJOO*mtp I

Ii

I

-C cttc

p 7bh,ith

4

Cwrnrv

rSt: Se

Fig. 9. Matching details of the foveated images on the peripheral background near the center FOV (lower left) and the peripheral FOV (upper left)

Captured by the 2/3” peripheral imager, Figure 7 left shows the low-resolution peripheral background with 46.5° diagonal FOV. The high-resolution foveated foreground with 10° diagonal FOV is captured by the 1/4” foveated imager simultaneously, which is shown in Figure 7 right. By tilting the MEMS from applying -3V to +3V with 0.5V increments parallel to both x and y channels, an integrated foveated image sequence of the University of Arizona campus map is captured by superimposing the foveated foregrounds onto the peripheral background, which is shown in Figure 8. The 10° foveated FOV is steered across the entire 46.5° peripheral FOV by tilt the MEMS mirror from -4.3° to +4.3° with approximately 0.7° increments. After some simple post-processing steps, such as distortion correction and image flip, the FRoI foreground images were well aligned with their corresponding regions on the peripheral background, as shown in Figure 9.

4. CONCLUSION In this paper, we present a novel foveated imaging design which consists of a foveated imager, a peripheral imager and a MEMS scanning subsystem. The angular resolution of the foveated imager with a small FOV is optimized in CODE V to be about nine times better than that of the peripheral background with full FOV. The low power consuming scanning subsystem, implemented with a two-axis MEMS mirror, is capable to sweep the FRoI across the entire FOV at a speed up to 100Hz. The off the shelf bench prototype and the integrated image sequence by superimposing high-resolution foregrounds onto the low resolution background are also shown to verify the optical design concept and feasibility. Future work may include building up the customized system based on optimized lenses to increase the resolution of the foveated foreground, applying multiple exposures at different FOV and the corresponding MEMS angle to reconstruct high resolution large FOV image and applications in fast image processing and transmission over the network.

ACKNOWLODGEMENTS This work is partially supported by the National Science Foundation grant IIS/HCI 04-11578.

SPIE-OSA/ Vol. 6342 63422W-7

REFERENCES 1.

Z. G. Zhu, K. D. Rajasekar, E. M. Riseman, and A. R. Hanson, “Panoramic virtual stereo vision of cooperative mobile robots for localizing 3D moving objects,” in Proc. of IEEE Workshop on Omnidirectional Vision, 29-36. (2000). 2. D. A. Scribner, L. J. Buckley, M. Satyshur, R. R. Sands, and G. Zuccarello, “Bio-inspired optics,” in Infrared Technology and Applications XXIX, Proc. SPIE 5074, 312-317 (2003). 3. W. S. Geisler and J. S. Perry, “Real-time foveated multiresolution system for low-bandwidth video communication,” in Human Vision and Electronic Imaging III, Proc. SPIE 3299, 294-305 (1998). 4. T. Ienaga, K. Matsunaga, K. Shidoji, K. Goshi, Y. Matsuki and H. Nagata, “Stereoscopic video system with embedded high spatial resolution images using two channels for transmission,” in Proc. of the ACM Symposium on Virtual Reality Software and Technology, 111-118 (2001). 5. R. Etienne-Cummings, J. Van der Spiegel, P. Mueller and M. Z. Zhang, “A foveated silicon retina for twodimensional tracking”, IEEE Trans. Circuits Syst. II 47, 504-517 (2000). 6. T. Martinez, D. V. Wick and S. R. Restaino, “Foveated, wide field-of-view imaging system using a liquid crystal spatial light modulator,” Opt. Express 8, 555-560 (2001). 7. D. V. Wick, T. Martinez, S. R. Restaino and B. R. Stone, “Foveated imaging demonstration,” Opt. Express 10, 6065 (2002). 8. G. Godin, P. Massicotte, and L. Borgeat, “High-resolution insets in projector-based stereoscopic displays: principles and techniques,” in Stereoscopic Displays and Virtual Reality Systems XIII, Proc. SPIE 6055, 60550F1-12 (2006). 9. J. P. Rolland, A. Yoshida, L. D. Davis and J. H. Reif, “High-resolution inset head-mounted display,” Appl. Opt. 37, 4183-4193 (1998). 10. H. Hua, C. Gao, “A polarized head-mounted projective display,” in Proc. of IEEE and ACM Symposium on Mixed and Augmented Reality, 32-35. (2005). 11. Optical Research Associates, http://www.opticalres.com

SPIE-OSA/ Vol. 6342 63422W-8