Lensless three-dimensional integral imaging using

Lensless three-dimensional integral imaging using variable and time multiplexed pinhole array Ariel Schwarz, 1, Jingang Wang, 1 Amir Shemer, 2 Zeev Zalevsky, 2 and Bahram Javidi 1* 1

Department of Electrical and Computer Engineering, University of Connecticut, Unit 4157, Storrs, Connecticut 06269, USA 2 Faculty of Engineering, Bar Ilan University, Ramat Gan 529002, Israel *Corresponding author: [email protected] Received Month X, XXXX; revised Month X, XXXX; accepted Month X, XXXX; posted Month X, XXXX (Doc. ID XXXXX); published Month X, XXXX We present a multi variable coded aperture (MVCA) for lensless three-dimensional integral imaging (3D II) systems. The new configuration is based on a time multiplexing method using a variable pinholes array design. The system provides higher resolution3D images with improved light intensity and signal to noise ratio as compared to single pinhole system. The MVCA 3D II system configuration can be designed to achieve high light intensity for practical use as micro lenslets arrays. This new configuration preserves the advantages of pinhole optics while solving the resolution limitation problem and the long exposure time of such systems. The three dimensional images are obtained with improved resolution, signal to noise ratio and sensitivity efficiency. This integral imaging lensless system is characterized by large depth of focus, simplicity and low cost. . © 2014 Optical Society of America OCIS Codes: (100.6890) Three-dimensional image processing; (110.3010) Image reconstruction techniques; (110.6880) Three- dimensional image acquisition; (080.0080) Geometrical optics; (110.0110) Imaging systems. http://dx.doi.org/10.1364/OL.99.099999

Micro lenslets arrays are currently used in threedimensional integral imaging (II) for optical reconstruction. In integral imaging, lenses have been preferred over pinholes [1-13] for a number of reasons. Low resolution, low light level image reconstruction, and long exposure time to capture enough photons are the disadvantages of pinhole arrays for imaging. However, the advantages of pinhole optics, besides simplicity, are complete freedom from linear distortion, virtually infinite depth of focus and a very wide angular field that can be made to exceed 90o. In order to preserve the advantages of pinhole optics, together with solving the problems of low light intensity and signal to noise ratio (SNR), we will use a multipinhole array configuration that is composed of variable coded aperture arrays replacing the pinholes positions in the multi-pinhole array. In multi pinhole array and coded aperture imaging a higher number of pinholes are used in order to obtain light intensity and SNR improvement. The unique variable coded aperture design is based on variable and time multiplexed pinholes array. The advantage of time modulated coded aperture over a stationary design is to preserve the image frequency contents more efficiently. This design is based on our previous work of variable coded aperture presented for 2D imaging with very limited 3-D ability [14]. In this previous design the 3-D imaging can be obtained only for the near field case by using laminography approach and even then its performance of the depth resolution is very low. This is caused due to the overlapped array design multiplexed by time presented in that work. In our present work we extended our work to full 3-D imaging for far field or near field cases with improved depth resolution. This was done by presenting the multi variable coded aperture (MVCA) design that is composed of several variable coded apertures in a non-overlapped structure. This multi variable coded aperture system (MVCA) is able

to achieve higher resolution imaging (not just as a result to SNR and light intensity enhancement, but also by overcome the loss of spectral information), while the multi-pinhole non-overlapped matrix design is able to provide increased depth of field for 3D II. The system provides the advantages of simplicity, freedom from linear distortion, virtually infinite depth of focus and a very wide angular field. It is achieved while providing higher resolution images with improved signal to noise ratio, high light intensity and with the same time exposure as compared to a single pinhole system or lens array II system. This MVCA technique allows a wide range of design options for many applications that determine the characteristics of the optical imaging system. The light efficiency improvement factor and the signal to noise ratio determined by the variable coded aperture and multipinhole matrix designs will be presented. In pinhole optics, the resolution may be characterized by the geometric optics limit for large pinholes and by diffraction limit for very small pinholes while the analysis in between large and very small pinholes is more complicated [15-17]. When the pinhole is large, the image is a uniform disk which is just the geometrical shadow of the pinhole. Thus, the size of the image is proportional to the pinhole size. When the pinhole is very small, the image is constructed according to Fresnel or Fraunhofer diffraction. Due to those two limits, in order to achieve the best resolution, the optimum pinhole size (R) will be a compromise between the large spot image produced by geometric optics and the one produced by diffraction from a small pinhole: R~(0.61λf/1+M)1/2, where λ is the wavelength, f is the pinhole focal length and M is the magnification factor . The smallest image thus occurs roughly where the geometrical optics and diffraction approximations give the same result. In this Letter, we present a MVCA technique for lensless 3D II system. The computational integral

imaging system is composed of two steps as shown in Fig. 1. The first is the pickup process that creates the elemental images from the MVCA optical imaging system. The 3D reconstructed output image is obtained by reconstruction in the visualization or display process. This computational integral imaging reconstruction simulates the geometrical optics using a virtual pinhole array and produces a series of slice or depth images representing a 3D scene. The optical imaging system of one elemental image based on a variable coded aperture with K different positions consists of a set of K arrays of pinholes. The switch methods of the array position are related to the system application and can be mechanical, electrical etc. A single pinhole causes loss of information due to narrow optical transfer function (OTF). A multi-pinhole array also causes loss of information and will adversely affect the final reconstruction. The reason for this is that multipinhole arrays also have many zero points in their optical transfer function plane. Using a time variable array can remedy this problem to some degree. Although each array in the set have zero points in the OTF, the overall array maintains higher spatial frequencies of the object to allow higher resoled images. The usage of a set of multi-pinhole arrays instead of a single multi-pinhole array with the same total number of holes is essential in order to cover the loss of information from a single multi-pinhole array. The variable array creates the complement encoding aperture with minimum loss of information.

of variable pinhole array does not extend the exposure time and enables us to perform the reconstruction after a single capture as in a single array. The final MCVA array is composed of a JxJ multi-pinhole array in a nonoverlapped matrix configuration design of multiple variable coded apertures which are used instead of the array's pinholes. In the MCVA system, the original elemental image, I(x,y) becomes blurred due to the variable coded aperture multiplexing. The final captured elemental image, Icapture(x,y) consists of product of the object according to the pinholes number and localization in the array and according to the variability of the array during exposure. The original elemental image needs to be reconstructed after the time multiplexing. Since the final captured elemental image, Icapture(x,y) is obtained while the array changes during the entire exposure period, it can be considered as the sum of all images in the arrays in the set as following: K

I capture( x, y )   ik ( x, y )  t k k 1

,

N

i k ( x, y )  t k  t k   i ( x  p x , k n 1

(n)

, y  p y ,k

(n)

(1)

)

where ik(x,y) is the image contribution due to array configuration in each time interval and i(x,y) is the original image accumulating in time. The time parameter for each array capture is variable and depends on the variable coded aperture design. The variable coded aperture which changes K times can actually be considered as an updatable filter H(ξ,η) applied in the spatial frequency domain multiplied with the spectrum of the imaged object. This can be seen by computing the spectrum of the spatial distribution of a single time interval capture and to obtain the variable coded aperture filter H(ξ,η) in each time interval: N

H k ( , )   e

 2i ( p x , k ( n )  p y , k ( n ) )

,

(2)

n 1

The spectrum of the spatial distribution is given by the Fourier transform and so the final elemental image in the frequency domain is: K

Fig. 1. The pickup process (top) and the visualization or display process (bottom) of the proposed lensless 3-D integral imaging system.

In order to preserve all the spatial frequencies of the original object, we must control the variable coded aperture design. There are four free parameters in the variable coded aperture configuration: the number of positions in the variable coded aperture (K), the number of pinholes in each array position (N), the spatial order of the pinholes in each array (px,y(n)), and the capture time intervals for each array position (tk). The use of such kind

Iˆcapture( , )   iˆk ( , )  t k k 1

,

(3)

K

 Iˆ( , )   H k ( , )  t k k 1

In our experiments, the reconstruction of the elemental images in the pickup process can be performed by deconvolution method such as Wiener filtering or inverse filtering [19-21] to remove the distortion function, H(ξ,η):

  K   Iˆ( , )   Iˆ( , )   H k ( , )  t k   H ( , ) 1 k 1     H ( , )  

,

(4)

In the far field case (Z>>F), the difference between the replication distances in the image (due to the number of pinholes in the array) for all 3D object planes will be negligible. However in the near field situation the distances between the object’s replications in the image plane are not the same as the distances of the pinholes in the array. The ratio between the two set of distances is related to the realized magnification:

p x' , y

(n)

 (1  M )  p x, y

( n)

,

(5)

where the magnification factor is M=F/Z. As a result, in the near field case, the elemental image reconstruction is performed by using the inverse of H(ξ,η) which corresponds to the object range and ratio in Eq. 5 for each z-plane. The fused image is obtained from all z-plane reconstructed images. The final elemental image in the pickup process is generated by a multi-focus image reconstruction algorithm using the wavelet transform [22]. We now present experiments for the far field. We have used the same pinhole diameter (D) of 150µm for all the pinholes in the MVCA. However, different diameter for pinholes can be used in the array design. This additional parameter (Dk,n) can be added to the array design with some limitations. In order to achieve high resolution, the angular separation must be as small as possible. The optimal focal length F=16.1mm was calculated for visible light (≈570nm) and according to the considerations for tradeoff between the angular separation due to geometric limit (D2>>F) and the angular separation due to diffraction limit (D2F), and M is negligible. The relative light intensity improvement (RLII) of one variable coded aperture as compared to a single pinhole system is determined by the number, size and time exposure of each pinhole in the variable array: K

RLII 

N

 D k 1 n 1

2 k ,n

 tk

2 T  Dreference

,

The elemental unit of one variable coded aperture is equivalent to one microlens in the lenslet array, and consists of a total of 17 pinholes in set of three arrays (K=3). Although the exposure time for each array position is one of the free parameters, for simplicity reasons in this experiment the configuration was designed without zero points in the OTF with equal time intervals(tk=T/K) [14]. The comparison between the transfer functions of the designed configuration and single pinhole are shown in Fig. 2. Thus, relative light intensity improvement as compared to a single pinhole system was 5.67 and the total relative light intensity improvement of the MVCA system was 51 (J=3). Two configurations were tested with different objects to array distances. Each setup was used with two 3-D objects in different distances.

(7)

where T is the total exposure time and Dreference is the single pinhole diameter. The final relative light intensity improvement of the MVCA system will be the relative light intensity improvement of one variable coded aperture multiplied by the number of variable coded apertures in the MVCA array (JxJ). This is accomplished due the non-overlapping feature of the multi-pinhole array by using a simple position summation algorithm.

Fig. 2. The transfer function of the system (top line) vs. single pinhole. In the first test, the experimental setup parameters were as following. The object #1 to array distance (Z1) was 138-144mm and the object #2 to array distance (Z2) was 160-166mm. The multi variable design purpose is to overcome the loss of information of single pinhole with improved light intensity. The reference systems were single pinhole system with the same 150µm pinhole and 350µm pinhole for better light intensity. Relative light intensity improvement was5.44 as compared to single 150µm system. The MCVA array dimensions were 12mm x 12mm and consisted of 3 x 3 elemental units replacing the standard integral imaging microlens array. The elemental images that were captured in the MVCA system were first reconstructed as part of the pickup process. The comparison of resolution and light intensity between one variable coded aperture before and after reconstruction, and the reference systems are shown in Fig. 3. The visualization and display process consisted of computational reconstruction to produce a series of slice images for different z-plane (distances) representing a 3D space. The z-plane slices range was 120-175mm in steps of 1mm. The slice images with the depth information of the two 3D objects are presented in Fig. 4 and the PSNR improvement is presented in table 1. In the second test, the experimental setup parameters were as following. Object #1 to array distance (Z1) was 0.91-0.99m and object #2 to array distance (Z2) was 1.151.2m (see Fig. 4). The MVCA array dimensions were 36mm x 36mm and consisted of 3 x 3 elemental units replacing the standard integral imaging microlens array. The reference single pinhole systems were 150µm pinhole and 350µm pinhole.

Fig. 3. The comparison of resolution and light intensity in the pickup process. (a). The reconstructed elemental image in one variable coded aperture (150µm); (b). The captured elemental image in one variable coded aperture (150µm); (c). The elemental image in the single pinhole system (150µm); (d). The elemental image in the single pinhole system (350µm).

Fig. 7. The elemental images in the integral imaging pickup process: (a). Elemental image reconstructed by variable coded aperture with (150µm); (b). Elemental image captured by variable coded aperture with (150µm); (c-d). Elemental image captured by single pinhole system (150µm and 350µm).

In conclusion, we have presented a lensless integral imaging system with the ability to obtain high resolution and high light intensity 3D images using MVCA system. The system advantages are large depth of focus, short exposure time, high resolution images, improved PSNR, simplicity and low cost.

Fig. 8. Images reconstructed in different depth with integral imaging for the second experiment. Fig. 4. The integral imaging reconstruction in the first experiment. Various Depth slices of reconstructed images.

In order to show that with variable coded aperture the overall array maintains high spatial frequencies to achieve high resolution images, we conducted the following experiment. A comparison was performed between the variable coded aperture reconstructed image and a single pinhole system with the same pinhole size (150µm), same position and light intensity. To achieve the same light intensity between the systems, a long exposure time (~x6) was used in the single pinhole system. The high resolution image is presented in Fig. 6. In the integral imaging pickup process, the elemental images that were captured were reconstructed and comparisons of resolution and light intensity were made between one variable coded aperture (before and after reconstruction) and the reference as shown in Fig. 7. The range of reconstructed image planes was 0.90-1.35m in steps of 1mm. The reconstructed images with depth information of the two 3D objects are presented in Fig. 8 and the PSNR improvement is presented in table 1.

Table 1. PSNR results

First experiment Second experiment

One VCA reconstructed image

Improved PSNR

3.0631

8.8588

5.7857

4.3042

7.8000

3.4958

References 1. 2. 3. 4. 5. 6. 7. 8. 9.

Fig. 6. (a). High resolution elemental image for variable coded aperture reconstruction (150µm); (b). Elemental image of single pinhole system (150µm) with long exposure time (x6).

Single pinhole

10. 11. 12. 13. 14. 15. 16.

G. Lippmann, C. R. Acad. Sci. 146, 446 (1908). H. E. Ives, J. Opt. Soc. Am. 21, 171 (1931). C. B. Burckhardt, J. Opt. Soc. Am. 58, 71 (1968). T. Okoshi, Three-dimensional imaging techniques, New York: Academic Press (1976). L. Yang, M. McCornick, and N. Davies, Appl. Opt. 27, 4529 (1988). F. Okano, H. Hoshino, J. Arai, and I. Yuyama, Appl. Opt. 36, 1598 (1997). J.-S. Jang and B. Javidi, Opt. Lett. 27, 1144 (2002). S. Hong, J. Jang, and B. Javidi, Opt. Express 12, 483 (2004). F. Okano, J. Arai, K. Mitani, and M. Okui, Proc. IEEE 94, 490 (2006). A. Stern and B. Javidi, Proc. IEEE 94, 591 (2006). B. Javidi, F. Okano, and J. Y. Son, Three-Dimensional Imaging, Visualization, and Display (Springer, 2009). R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, Proc. IEEE 97, 1067 (2009). M. Daneshpanah , B. Javidi, Opt. Lett. 36, 600 (2011). A. Schwarz, A. Shemer, and Z. Zalevsky, Appl. Opt. 53, 4483 (2014). A. C. Hardy and F. Perrin, The Principles of Optics, New York: McGraw-Hill (1932). E. W. H. Selwyn, Photographic J. 90B, 47 (1950).

17. L. C. Martin and W. T. Welford, Technical Optics, Vol. I, London: Pitman (1966). 18. R. Willingale, M. R. Sims, and M. J. L. Turner, Nucl. Instrum. Methods Phys. Res. 221, 60 (1984). 19. M. Antweiler and H. D. Lüke, Frequenz 48, 58 (1994). 20. H. D. Lüke and A. Busboom, Appl. Opt., 37, 856 (1998). 21. J. W. Woods, M. P. Ekstrom, T. M. Palmieri and R. E. Twogood, IEEE Trans. Nucl. Sci. NS-22, 379 (1975). 22. X. Yang, W. Yang and J. Pei, Proc. of Third Inter. Conf. on Inform. Fusion 1, 3 (2000).

18. R. Willingale, M. R. Sims, and M. J. L. Turner,

19. 20.

21.

Full References 1. 2.

3.

4. 5.

6.

7.

8.

9.

10.

11. 12.

13.

14.

15. 16. 17.

G. Lippmann, “La photographie integrale,” C. R. Acad. Sci.146, 446–451 (1908). H. E. Ives, “Optical properties of a Lippman lenticulated sheet,” J. Opt. Soc. Am. 21, 171-176 (1931). B. Burckhardt, “Optimum parameters and resolution limitation of integral photography,” J. Opt. Soc. Am. A 58, 71–74 (1968). T. Okoshi, Three-dimensional imaging techniques, New York: Academic Press (1976). L. Yang, M. McCormick, and N. Davies, “Discussion of the optics of a new 3-D imaging system,” Appl. Opt.27 (21), 4529–4534 (1988). F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Realtime pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36, 1598– 1603 (1997). J. S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. 27, 1144-1146 (2002). S. Hong, J.-S. Jang, B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12, 483–491 (2004) F. Okano, J. Arai, K. Mitani, and M. Okui, “Real-time integral imaging based on extremely high resolution video system,” Proc. IEEE 94, 490-501 (2006). A. Stern and B. Javidi, “3D Image Sensing, Visualization, and Processing using Integral Imaging,” Proceedings of the IEEE Journal 94(3), 591-608 (2006). B. Javidi, F. Okano, and J. Y. Son, Three-Dimensional Imaging, Visualization, and Display (Springer, 2009). R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Progress in 3-D Multiperspective Display by Integral Imaging,” Proceedings of the IEEE Journal 97(6), 1067-1077 (2009). M. Daneshpanah and B. Javidi, “Three-dimensional imaging with detector arrays on arbitrarily shaped surfaces,” Optics Letters 36(5), 600-602 (2011). A. Schwarz, A. Shemer and Z. Zalevsky, “Light intensity and SNR improvement for high-resolution optical imaging via time multiplexed pinhole arrays,” Appl. Opt. 53, 4483-4492 (2014). A. C. Hardy and F. Perrin, The Principles of Optics, New York: McGraw-Hill (1932). E. W. H. Selwyn, “The Pin-Hole Camera,” Photographic J. 90B, 47-52 (1950). L. C. Martin and W. T. Welford, Technical Optics, Vol. I, London: Pitman (1966).

22.

“Advanced deconvolution techniques for coded aperture imaging,” Nucl. Instrum. Methods Phys. Res. 221, 60-66 (1984). M. Antweiler and H. D. Lüke, “Mismatched–Filterung periodischer Binärarrays,” Frequenz 48, 58-65 (1994). H. D. Lüke and A. Busboom, “Mismatched filtering of periodic and odd–periodic binary arrays,” Appl. Opt., 37, 856–864 (1998). J. W. Woods, M. P. Ekstrom, T. M. Palmieri and R. E. Twogood, “Best Linear Decoding of Random Mask Images,” IEEE Trans. Nucl. Sci. NS-22(1), 379-383 (1975). X. Yang, W. Yang and J. Pei, “Different focus points images fusion based on wavelet decomposition,” Proc. of Third Inter. Conf. on Inform. Fusion 1, 3-8 (2000). Table 2. Notations and parameters

Notation

Definition

R λ

pinhole size (radius) wavelength

F M

pinhole focal length magnification

K

number of positions number of pinholes in each array position spatial order of the pinholes

N p tk T J I Icapture i ik H D Z

time interval of one position total capture time non-overlapped matrix dimension original elemental image final captured elemental image original image accumulating in time image contribution due to array configuration in each time interval variable coded aperture filter pinhole diameter object to pinhole distance

Appearance in text In text In text, Eq. 6 In text, Fig 1, Eq. 6 In text, Eq. 5-6 In text, Eq. 1, 34, 7 In text, Eq. 1-2, 7 In text, Eq. 1-2, 5 In text, Eq. 1, 34, 7 In text, Eq. 7 In text In text, Eq. 3-4 In text, Eq. 1, 3 In text In text, Eq. 1, 3 In text, Eq. 2-4 In text, Eq. 6-7 In text, Fig. 1