Study on far-field image using micro-lens array integrated with LCD Fugui Yang, Hong Chang, Lei Dong, Anting Wang*, Hai Ming Department of Physics, Anhui Key Laboratory of Optoelectronic Science and Technology, University of Science and Technology of China, Hefei, Anhui, China 230026 ABSTRACT We firstly propose the technique to realize far-filed image by integrating the microlenses array with LCD. This technique is expected to resolve eye-gazing problem in common teleconferencing system without affecting the display performance of LCD. The characteristics of far-field diffraction image and geometric image formed by single microlens are simulated and analyzed. The simulation shows that the quality of image is seriously reduced by geometric aberration and diffraction of the small aperture diameter for aspherical and spherical surface profile refractive microlens. The main problem of image acquisition with high resolution is that the signal obtained by cell detector is contributed not only by the sampling point in the axis of the corresponding cell microlens but also sampling points nearby. In simulation, Microlenses of 300μm diameter same with pixel size of LCD were used in orthogonal array at 1mm pitch, which is also the sampling interval in object space. These microlenses array with unit number 213×246 are placed up to 600mm away from object surface. Monte Carlo optimization algorithm is adopted to give accurate inversion results. The typical results are presented with our main conclusions. Keywords: LCD, microlens array, far-filed, image, simulation, integration
1. INTRODUCTION The quick development of the internet technique makes it possible and convenient to communicate with people by video and audio instantaneously as in real world even both sides are far away. However, videoconferencing does not seem to be as widely spread as predicted even thought it could cut down the cost compared with conventional conference model. Among many problems faced in video-teleconferencing, such as cost, network bandwidth, and resolution, the lack of eye-contact has a profound impact on communication and seems to be the most difficult one to overcome [1,2]. The reason for the loss of gaze-awareness when a participant is looking at someone’s image on his display is that the camera and display screen cannot be physically aligned in a typical desktop environment and often mounted above, below or beside the displays. Several techniques have been developed to correct eye gaze and bring video-teleconferencing one step closer to mass market. The most popular method is image processing technique which effectively utilizes the high performance of computer [3-5] without any of special hardware appended. The approach taken involves three steps: pose tracking, view matching, and view synthesis. A pair of calibrated stereo cameras and a personalized face model to track the head pose in 3D, the use of strong domain knowledge (a personalized face model) and a stereo camera pair greatly increase the robustness and accuracy of the 3D head pose tracking. The results from the head tracking and stereo matching are combined to generate a virtual view. However, the speed and effect of image processing limits application of this technique. The other way is to directly image with special hardware system [6-8]. The concept of direct imaging system with contact is that the camera is aligned with display panel without affecting the effect of display. Half-silvered mirrors or transparent screens with projectors were used to allow the camera to be placed on the optical path of the display, which means that when the local participants look at the display showing remote participants and they are also looking directly into the camera and providing true eye contact (with zero gaze angle) to the far end sites. The expensive cost and bulky setup prevent them to be widely used in commerce. This paper presents the technique to realize far-filed image by integrating the microlenses array with LCD (Liquid Crystal Display). The system, called Micro-Lens LCD Far-Field Imaging (MLFI) system, is expected to resolve eyegazing problem in common teleconferencing system without affecting the display effect of LCD. The characteristics of far-field diffraction image and geometric image formed by single microlens are simulated and analyzed. The main *
Corresponding author:
[email protected] 2009 International Conference on Optical Instruments and Technology: Optoelectronic Devices and Integration, edited by Xuping Zhang, Wojtek J. Bock, Xuejun Lu, Hai Ming, Proc. of SPIE Vol. 7509, 750904 · © 2009 SPIE CCC code: 0277-786X/09/$18 · doi: 10.1117/12.838094 Proc. of SPIE Vol. 7509 750904-1
problem of image acquisition with high resolution is that the signal obtained by cell detector is contributed not only by the sampling point in the axis of the corresponding cell microlens but also sampling points nearby. And Monte Carlo optimization algorithm is adopted to retrieve image with high resolution. In Section.2 we present the architecture of MLFI system. In Section.3 we simulate and analyze image performance of single micro lens and microlenses array to decide relevant parameters of the system. In Section.4 Monte Carlo optimization algorithm is introduced and applied in image reconstruction and optimization.
2. MICRO-LENS LCD FAR-FIELD IMAGING (MLFI) SYSTEM 2.1 Architecture The system under consideration solves the eye gaze problem by integrating the microlens array with LCD which has been the popular display commercially. The thought of far-field image of the microlens is different from common image function of microlens array that has been used to form 3-D image [9] or 2-D image with high resolution [10-12].
Fig.1.. Schematic diagram of MFLI system
The schematic diagram of MLFI system is shown in Figure 1. Components of the optical module are a microlens array, a single separation layer and a photodetector array situated in front of the LCD panel. Microlens array images object far away to the plane that detector array is positioned. Only a photodiode cell is integrated along the optical axis of each microlens. In order to prevent light collected by adjacent unit cross talk, a separation layer with opaque wall is put between the microlens array sheet and photodetector array plane. The common LCD display images by pixel array effect, where 4-sub-pixels (Red/Green/Blue/White) construct one pixel. This structure makes it possible to integrate microlens to White sub-pixel without affecting the display function that R/G/B sub-pixels play and similar integration technique has been applied to 3D display where microlens[13]. For simplicity, one-dimension is used to analysis in following context. The schematic diagram of optical module in the system in Fig.2 gives a clear explanation of the principle of the MLFI. Dimension of each detector is so small (dimension of single CCD pixel) that only light from paraxial domain corresponding to the optical axis is recorded. That is to say only adjacent sampling points will contribute to the signals detected if we sample the object plane with a certain resolution Δx . Thus, certain function between detected signals and light intensity distribution at these sampling points, which is easily got by measurement or computation, can be used to reconstruct the image. In simplest situation that only one sampling point remains the paraxial domain, the detected signals are the direct representation of the object. For general situation, reconstruction algorithm is needed. The 2-D MLFI system is characterized by unit number N×K, diameter of microlens Dlens , focal length f , detector size d , contributing cell number n×n.
Proc. of SPIE Vol. 7509 750904-2
Fig.2.. Optical system of MFLI
2.2 Diffraction and Aberration The analysis above is an ideal consideration which doesn’t take small dimension into account and geometric aberration. Owing to the small dimension of microlens (diameter 200um-500um, focal length 500um-2000um) and far away object distance, the effect of diffraction cannot be neglected. Strictly speaking, the single microlens dose not image but collect the light from the small angle in MLFI system if object distance large than focal length, which is more similar with the function that telescope acts in this mean. Figure 3 gives light collection by single microlens in i-th unit, where the number of contributing sampling point n is confined to 3. The light from one sampling point will broaden its width by DAiry , diameter of Airy disk in focal plane, because of effect of diffraction. Geometric aberration is another key impact in image system, which seriously decreases the image quality. And it also causes PSF (Point Spread Function) broadened same with diffraction. According to our need, we only focus on the off-axis aberration like spherical aberration, which is the main aberration in paraxial domain. So generally it is necessary to value the effect of diffraction in physical optics and aberration in geometrical optics.
Fig.3.. Schematic diagram for far-field image of the microlens
3. MICROLENS ARRAY Under the original idea of the proposal, the use of micro-lenses array is just in order to get more sampling points in object and there should be no interference with each other in imaging space (behind the microlenses array) for each
Proc. of SPIE Vol. 7509 750904-3
micro-lens cell. So the case of a single micro-lens, which has been shown in Fig.3, is studied rather than analysis of microlenses array, which reduces the degree of complexity and difficulty to a large extent in the simulation and experimental study. The important work is to specify series of parameters in MLFI system based on the theoretically study above. Here we select surface profile refractive micro-lens, spherical or aspherical type, on consideration of difficulty and cost of fabrication. The study is performed by ray trace in powerful optical design and analysis softwareZemax, which provides diffraction analysis function at the same time. Linking to the practical application, this system is decomposed into three components: light source, microlenses array and detectors. The light source is actually the imaging object, which determines that the light is non-coherent. Spherical surface is most commonly used and simplest. Table.1 gives the lens data edition, which specifies each surface with radius/thickness to the next surface/glass type/semi-diameter/conic in this system, and Fig.4 (a) shows the layout. The substrate is quartz with refractive index 1.45 and the lens material is resin with 1.65. The spot diagram in Fig.4 (b) shows that the RMS radius of the points in image plane intersected by the rays traced is 23.72um, which is much larger than that of Airy disk 2.33um. Analysis by synthesizing these results show that the aberration of the system is too heavy compared with diffraction effect, which is also directly perceived through the senses in layout of the system. The MTF (modulation transfer function) shown in Fig.4(c) certifies our conclusion that the quality of image seriously degrades due to the image aberration especially spherical aberration which is the main aberration in paraxial region. Table. 1. Lens data edit for spherical surface profile refractive microlens image system
(a)
(b)
(c) Fig.4. Simulation and analysis of microlens with spherical surface: (a) layout of the system ;(b) spot diagram; (c) MTF
Proc. of SPIE Vol. 7509 750904-4
(a)
(b)
(c) (d) Fig.5.. Simulation and analysis of microlens with aspherical surface: (a) layout ;(b) spot diagram; (c) MTF; (d)geometric image
Although limiting the numerical aperture of spherical surface profile microlens is a way to minimize the spherical aberration [14], this causes the efficiency of light collection decrease obviously and broadens the PSF because of increase of the radius of the Airy disk. Aspherical surface profile microlens, which is widely used in common image system to correct spherical aberration, becomes a moderate choice. The lens data about the system using aspherical lens is listed in Table 2. The field in object space is chosen at 0mm/4mm/8mm.The MTF approaches diffraction limit for paraxial object without decreasing its efficiency of light collection as shown in Figure 5(b). Even though there are large amount image off-axis aberration e.g. coma/distortion, diffraction becomes the main aspect affecting the image performance of the microlens referring to Figure5 (b) and (c). Image of grid is simulated by geometric ray tracing in figure 5(d) and the results prove our conclusion above. Then diffraction image analysis which is also provided in Zemax is done in order to design the far-field imaging system. One dimension Images of five spot resources spaced by 4mm and 2D dimension image of grid with 20mm width are shown in Figure 6(a) and (b). The most interesting information is that the light concentrates in zero-level diffraction spot and the peak of PSF dose not decrease much as the increase of the field except that the angle amplification
α ( x, y ) is not uniform. Thus we can analysis the distribution in the zero-level
diffraction spot instead of computing the distribution in whole image plane. For one dimension, the relation between the pixel detector size d and the number of sampling points（2m+1）on the assumption that the object is placed at far field, diffraction images of which contributes to the detected signal, is given by
Proc. of SPIE Vol. 7509 750904-5
⎛ mΔx ⎛ ( m + 1) Δx D ⎞ D ⎞ 2⎜ α ( mΔx ) f + Airy ⎟ ≤ d ≤ 2 ⎜ α ( ( m + 1) Δx ) f − Airy ⎟ ⎜ Lobj ⎜ Lobj 2 ⎟⎠ 2 ⎟⎠ ⎝ ⎝ ⎛ Δx D ⎞ α ( Δx ) f − Airy ⎟ d ≤ 2⎜ ⎜L 2 ⎟⎠ ⎝ obj
m>0
(1)
m=0
(2)
where Lobj is the object distance. Thus we can determine the detector size d required for a system with certain resolution Δx and angle amplification
α ( x) .
Table. 2. Lens data edit for aspherical surface profile refractive microlens image system
(a)
(b)
Fig.6.. Diffraction image analysis of microlens with spherical surface: (a) spot resource; (b) grid
4. IMAGE RECONSTRUCTION AND OPTIMIZATION To reconstruct the image of the object from the signals captured by the photodiode array, we study two kind of condition based on whether it needs optimizing process. 4.1 Direct image When the contributing sampling point is just the one locating on axis of the microlens, the signal recorded has a linear relation with the intensity of the sampling point. Thus the array signal is a direct mapping of the object without any process. However satisfaction of the term that the detector pixel size should be small enough according to equation (1) is not easy to reach because of technique of detector fabrication. The smallest size of CCD pixel common used is about 1-2 micrometers. When applied to the system shown in Table 2, the resolution is lower than 1mm and the relation between the detector pixel size and resolution is shown in Fig.7, where angle amplification has been assumed to be uniform. To decrease the requirement of detector pixel size without changing the resolution, the focal length should be as long as enough. The other advantage of this direct image condition is that the depth of field can be change too much if the system parameter is properly selected.
Proc. of SPIE Vol. 7509 750904-6
Fig.7.. The relation between detector pixel size and focal length with resolution 1mm
4.2 Optimization To increase the quality of the reconstructed images, optimization of the signals captured by the photodetector array
(a)
(b)
(c) (d) Fig.8.. Simulation of image retrieval by the IMC method: (a) object, (b) detected signals, (c) retrieved image by IMC method, (d) image after noise filter by neighborhood arrangement method.
Proc. of SPIE Vol. 7509 750904-7
should be utilized in processing. Because the certain relation between the sampling points on the object and the photodetector can be describe as discussed above, we can calculate the object image using inverse method. Monte Carlo optimization algorism is adopted here.
r
r
For simplicity, we consider the optical system of a one-dimensional model with vectors f and g and matrix A ,
r
r
where f and g are the intensity of sampling points and the signals of the detector array, respectively, and the contributing matrix. Then a matrix equation is given by:
A denotes
r r g = Af
(3) The characteristics of the matrix equation that variable number is larger than that of equations pretends the direct solution of solving linear equations such as elimination method, iterative method. An optimization algorithm-IMC (inverse Monte-Carlo technique), the aim of which is to find the value that makes the objective function minimum by optimization processing, becomes a feasible method and has been developed based on Monte Carlo random walk. Implicit in IMC method is a positivity constraint on the solution for f j . We initialize the walkers with the detected
f j for each sampling points. Then the r r total relative difference between the fit data g fit and experimental data g is defined following: signal which is a representation of the image and bin to get the initial intensity
⎛ g fit − gi ⎞ R = ∑⎜ i ⎟ gi ⎠ i ⎝
2
2
(4)
We then allow all walkers to move randomly by step (-1, 0, 1) and calculate a new difference R after each move. If the difference R is reduced we accept the new location of the walkers. We repeat this step until some maximum number of steps is reached or until the difference R between the measured data and the fit to data is less than a selected tolerance Rmax. The tolerance is selected to set a goal as to the quality of the fit desired. To verify the characteristics of the image of the MLFI, a computer simulation of the system described in Table 2 is executed. Figure 8 shows an example of the image reconstructed by IMC method with resolution 1mm. The contributing grid in object is chosen 5×5 and the number of sampling points is 213×246, which has simulated actual circumstance. As one can see from the result, the reconstructed image has a higher resolution and contrast compared with detected signals even though there is a little more noise spot appearing. In order to reduce the dark noise spots, neighborhood arrangement method is adopted and the result shows good. The problem of this reconstruction method is the low efficiency because the random evaluation for each variables.
5. CONCLUSIONS A compact image-capturing called MFLI system which integrates LCD with far-field image system has been presented. The general principle for choosing the microlens is discussed with considering image aberration and diffraction. For image reconstruction, IMC method provides good performance. Simulation results verify the principle of the MFLI system.
REFERENCES [1] [2] [3] [4] [5]
L. Mhlbach, B. Kellner, A. Prussog, and G. Romahn. "The Importance of Eye Contact in a Videotelephone Service". In 11th Interational Symposium on Human Factors in Telecommunications, Cesson Sevigne, France, (1985) Michael Argyle, [Bodily Communication], International Universities Press, Madison, Connecticut, (1988) Zhengyou Zhang, Ruigang Yang, "Video-teleconferencing system with eye-gaze correction", US Pat. 6771303,2004 M. Ott, J. Lewis, and I. Cox. Teleconferencing Eye Contact Using a Virtual Camera. In INTERCHI‘93, pages 119 – 110(1993) T.J. Cham and M. Jones. Gaze Correction for Video Conferencing. Compaq Cambridge Research Laboratory, http://www.crl.research.digital.com/vision/interfaces/corga.
Proc. of SPIE Vol. 7509 750904-8
[6] [7] [8] [9] [10] [11] [12]
[13] [14]
R. Kollarits, C. Woodworth, J. Ribera, and R. Gitlin. "An Eye-Contact Camera/Display System for Videophone Applications Using a Conventional Direct-View LCD". SID Digest, (1995) Jeffrey Machtig, Steve McNelly. "Videoconferencing eye contact spatial imaging display", US Pat.6042235, (2000) Kuriki et al. "Display and image capture apparatus which enables eye contact". US Pat.5317405,(1994) F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt., vol. 36, no. 7, pp. 1598–1603, (1997) Cees van Berkel, "Microlens arrays for 2D large area image sensors", Pure Appl. Opt. 3 177-182. Printed in the UK(1994) Kouichi Nitta, Rui Shogenji, Shigehiro Miyatake, and Jun Tanida, "Image reconstruction for thin observation module by bound optics by using the iterative backprojection method," Appl. Opt. 45, 2893-2900 (2006) Jun Tanida, Tomoya Kumagai, Kenji Yamada, Shigehiro Miyatake, Kouichi Ishida, Takashi Morimoto, Noriyuki Kondou, Daisuke Miyazaki, and Yoshiki Ichioka, "Thin Observation Module by Bound Optics (TOMBO): Concept and Experimental Verification," Appl. Opt. 40, 1806-1813 (2001) Scott S. Fisher; John O. Merritt; Mark T. Bolas, “Multiview 3D LCD”, Stereoscopic Displays and Virtual Reality Systems III, Proc. SPIE Vol. 2653, p. 32-39, (1996) Slyusarev,G.G.,[Aberration and optical design theory], J W Arrowsmith Ltd press, Bristol, page 245-251(1984)
Proc. of SPIE Vol. 7509 750904-9
