parallax in three-dimensional (3-D) space without ... parallax, an alternative stereoscopic vision, cannot .... feasibility study, we performed CT scanning to take.
H. Liao, M. Iwahara, N. Hata, I. Sakuma, T. Dohi, T. Koike, Y. Momoi, T. Minakawa, M. Yamasaki, F. Tajima, H. Takeda: High-resolution integral videography autostereoscopic display using multi-projector; Proceedings of the Ninth International Display Workshops, IDW’02, pp.1229-1232
High-resolution integral videography autostereoscopic display using multi-projector H. Liao, M. Iwahara, N. Hata, I. Sakuma, T. Dohi, T. Koike*, Y. Momoi*, T. Minakawa*, M. Yamasaki*, F. Tajima*, H. Takeda* The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113, Japan; *Hitachi Ltd, 4-6, Surugadai, Chiyoda-ku, Tokyo 101-8010, Japan ABSTRACT We present a high-resolution autostereoscopic display using the integral videography (IV) and multi-projector. The multi rear projection are used for creating a high-resolution image and long zoom lens projection is used for developing small size high-density image, with corresponding digital camera based measurement of the position and color features of the pixels projected on the screen via micro convex lens array to achieve highresolution seamless image. This system can display geometrically accurate high-quality autostereoscopic images and reproduce motion parallax in three-dimensional (3-D) space without using special viewing glasses. We evaluate the feasibility of this display by developing a 3-D CT autostereoscopic image and applying it to imageguided surgery.
INTRODUCTION Stereoscopic technique has been taking an important roll in a lot of filed recently, with various modes of visualization on offer [1-3]. Among previous reported methods use polarized or shuttering glasses originally developed in augmented reality domain and applied in surgical simulation and diagnosis. However, glass-based method does not always give an observer an accurate sense of depth, as the depth perception depends on the spacing of the observer's eyes, which is not always the same as the fixed lenses. In the case of this kind of stereoscopic displays, they can observe pre-fixed 2-D images, which create quasi 3-D images with distortion. Subjectively they perceive it as 3-D, but there may be significant inaccuracies in registration. Motion parallax, an alternative stereoscopic vision, cannot be reproduced without wearing a tracking device. Our previous reports [4] proposed an approach to overcome these issues of binocular stereoscopic vision by using Integral Videography (IV), which was an animated extension of integral photography (IP) originally proposed by M.G.Lippmann [5] in 1908. IV projects a computergenerated graphical object generated by multiple rays coming through micro convex lens array. The
detailed description of IP and IV can be found in [6,7]. Though the advantage of IV has been proven in both feasibility studies and clinical applications [4,6,7], one of the issues still unsolved is the limitation of pixel density, originally caused by of the inadequate resolution of currently available display devices. Thus, the motivation of this paper is to overcome these issues on limited resolution by using multiple projection display system and parallel computing for fast rendering. We evaluate the feasibility of this display by developing a 3-D CT autostereoscopic image and applying it to surgical planning. The main engineering contribution of this paper is use of parallel visualization and multiple optical projection systems for autostereoscopic IV image generation, which thought to be an effective solution for generating high-resolution autostereoscopic display. This engineering contribution deploys the clinical significance of the study that the newly proposed method enables high-resolution autostereoscopic visualization of the 3-D reconstructed MR, CT, or US images.
MATERIALS AND METHODS A schematic presentation of the multi-projector IV display is presented in Fig. 1. Four major components of the display system are: (1) Parallel rendering and parallel display; (2) High-resolution multi-projection system and image calibration; (3) Long zoom lens projection technique; (4) special image formation. Display cluster
Image processing
High-resolution and Spatial image high density image formation
IV image Image calibration High performance compute cluster
Micro lens array
System area network Data storage and server console
Fig.1. Schema of autostereoscopic display
3D image data （ MRI, CT, etc）
the
high-resolution
IV
1229
H. Liao, M. Iwahara, N. Hata, I. Sakuma, T. Dohi, T. Koike, Y. Momoi, T. Minakawa, M. Yamasaki, F. Tajima, H. Takeda: High-resolution integral videography autostereoscopic display using multi-projector; Proceedings of the Ninth International Display Workshops, IDW’02, pp.1229-1232 Parallel rendering and parallel display In this case, an important issue is the coordination of multiple commodity projectors to We integrated parallel rendering and display achieve seamless edge blending and precise system of the IV autostereoscopic display system alignment. Seamless edge blending can remove with several components (Fig.1), including 1) a the visible discontinuities between adjacent display cluster with multi PCs, which are used for projectors. Edge blending techniques overlap the parallel display of high-resolution image; 2) a edges of projected, tiled images and blend the compute cluster with high performance computer overlapped pixels to smooth the luminance and or workstation, which was used for parallel chromaticity transition from one image to another. calculation; 3) a console PC controls execution of We obtain precise alignment (or misalignment) system, including data input and storage. All the information with a digital camera (Fig.2) by PCs are connected together with a 100 Base-T measuring the position and color features of the Ethernet network. pixels projected on a screen via micro convex lens Here we face all three general types of research array directly, and then feedback the calibration challenges: coordination of PCs and graphics information to the image-processing hardware to accelerators to create consistent, real-time images; make fine adjustments of the source image before communication among multi PCs; and resource display by misaligned projectors. Using this allocation to achieve good utilization. We method, it requires only the coarsest physical integrated Message Passing Interface (MPI) [8] to alignment of the projectors. increase the rendering performance of IV. The parallelization is based on static partitioning of the Long zoom lens projection rendering image. Each node in a cluster computer Although we can achieve high-resolution image is assigned a section of the line to reconstruct. The by using a normal multi-projection technique, the node elaborates their section in parallel and, once pixel density of projected image is not suitable for finished, send reconstructed section to a master IV image creating. The pixel of the projected image process. The master process is in charge of I/O, or used in IV autostereoscopic display must be highnetwork communication API, which collects source density [7]. In this study, we developed long zoom images and sends the rendered image to parallel lens projection technique to achieve a high-density display cluster. pixel image. We redesign the projector by altering Multi-projector seamless display system the combination and arrangement of lenses to achieve new long zoom lens projection optics Despite much recent progress in the (Fig.3). The density of pixel on the projected image development of new display technologies such as can be provided over 300 dpi. Organic Light Emitting Diodes (OLEDs), the current economical approach to making a largeformat, high-resolution display uses an array of projectors [9]. In our system, multiple projectors of the display are arranged in display array, producing an image of high resolution across a rear projection screen by utilizing long zoom lens projection technique, which will be described later in this paper. We wanted to maximize the number of pixels displayed, but image blending would sacrifice a large percent of pixels, especially around the four edges of each center row image. Digital camera
Fig.3. Projector with long zoom lens projection for creating high-density image
Spatial autostereoscopic image formation Linear stage (2 axis)
Fig.2. Image calibration device
The micro lens array comprises a plurality of convex lenses with lens pitch about 1mm, which is suitable to create 3-D image of various medical uses. By the nature if the IV principle described, the high-resolution and high-density image projected on screen must be free from distortion and reflection, so an antireflective antistatic coating [10] flat screen is used to display the image. This screen is placed in the rear of micro convex lens
1230
H. Liao, M. Iwahara, N. Hata, I. Sakuma, T. Dohi, T. Koike, Y. Momoi, T. Minakawa, M. Yamasaki, F. Tajima, H. Takeda: High-resolution integral videography autostereoscopic display using multi-projector; Proceedings of the Ninth International Display Workshops, IDW’02, pp.1229-1232
A
L
(a)
Fig.4. Devices of 9 projectors IV autostereoscopic display system. (a): parallel calculation and parallel display. (b): highresolution multi projection system, including 9 projectors and corresponding m irror combination.
array. When the rendered elemental IV image is projected on the screen, the autostereoscopic image will be formatted to a spatial image.
EQUIPMENTS AND EXPERIMENTS To evaluate the feasibility of this study, we applied this display system to image guide surgery. This kind of autostereoscopic image can enhance the surgeon’s capability to utilize medical 3-D imagery to decrease the invasiveness of surgical procedures and increase their accuracy and safety. In this study, we perform surgical planning and simulation by this newly developed IV autostereoscopic display.
Equipments We developed a multi-projector system (Fig.4) with 9 parallel rendering and display PCs (Pentium III 1GHz Dual CPU) and 9 XGA projectors (U21110, PLUS Vision Corp.), arranged in a 3X3 array, which producing a display of 2872X2150 pixels across a 241X181mm (302.38dpi) rear projection screen by utilizing long zoom lens projection technique. The pitch of each pixel projected on the screen is 0.084 mm. Micro lens array placed on the screen has hexagonal micro lenses with diameter of approximately 1.008mm, covers 12 pixels in the long zoom lens projection screen. The image-
t=0.19s
R
B
(b)
t=0.0s
F
t=0.38s
Fig.5. Motion parallax of IV autostereoscopic images (skull). The letter denotes the position of the observer to the IV image: A: above, L: left, F: front, R: right, B: below.
processing hardware (Model PA99, System Development Laboratory, Hitachi, Ltd.) was developed with the maximum pixel frequency of the 65MHz that corresponds to 60 frames of XGA image per second, enable arbitrary geometrical warping and modulation for individual pixels. We obtain precise alignment (or misalignment) information with a digital camera (Nikon D1X, 3008X1536 pixels).
Animated IV image and motion parallax In order to observe the 3-D image with correct motion parallax, one should observe at a distance of about 50 cm from the system. The condition wherein the projected 3-D image was continuously observed was examined (Fig.5). We evaluated the usefulness of the newly developed system in operative setting. In clinical feasibility study, we performed CT scanning to take photo of in-vivo human heart 5 times in one cardiac cycle. The volumetric CT images of human heart (512X512pixelsX80slices for one time, thickness of 1.0mm) were rendered 5 times separately in one heartbeat. The rendered elemental IV images were projected continually on the IV autostereoscopic display with the same heartbeat period of the patient (Fig.6). (Since the projected IV image was purely three-dimensional, it
t=0.57s
t=0.76s
Fig.6. IV CT autostereoscopic animated image: the patient has a rate of 63 beats per minute, cardiac cycle of 0.95s.
1231
H. Liao, M. Iwahara, N. Hata, I. Sakuma, T. Dohi, T. Koike, Y. Momoi, T. Minakawa, M. Yamasaki, F. Tajima, H. Takeda: High-resolution integral videography autostereoscopic display using multi-projector; Proceedings of the Ninth International Display Workshops, IDW’02, pp.1229-1232 was difficult to record in two-dimensional achieve a high-quality autostereoscopic image. We conventional photographs. The quality of the actual evaluated the feasibility of this display by image was much better than shown in this figure.) developing a 3-D CT autostereoscopic image and applying it to surgical planning. The feasibility DISCUSSIONS study indicated that the multi-projector and parallel rendering performance achieved with the proposed As accuracy is most important element in method is satisfactory and suitable in surgical medical imaging, this system has advantages over diagnosis and planning setting. The present works other 3-D image display method. Theoretically, relate to the three-dimensional display device also when the ideal lens is utilized, IV can provide a can be used for art, entertainment, information/ three-dimensional display that is free from any multi-media, simulation, design aid, educational aid discontinuous change of images that occur with and other fields. the observer’s movement and with the same resolution that conventional two-dimensional ACKNOWLEDGEMENTS displays feature [11]. The measurements of spatial resolution of IV autostereoscopic display show the This study was partly supported by the Grant-inquality of IV image can be improved. Actually, both Aid for the Development of Innovative Technology of the IV autostereoscopic display systems (12113) by the Ministry of Education, Culture, developed in this study have a pixel density of only Sport, Science and Technology in Japan. about 300dpi. The spatial resolution of the 3-D REFERENCES image projected is proportional to the ratio of the lens diameter in the lens array to the pixel pitch of 1. M.A.Guttman, et al., “Techniques for fast display. Thus, both the lens diameter and the stereoscopic MRI,” Magnetic Resonance in projected pixel pitch need to be made much Medicines, Vol.46, pp317-323, 2001. smaller. One of the possible solutions to realize 2. M.Blackwell, et al., “An image overlay system for higher pixel density is the use of more multimedical data visualization,” Medical Image projectors or projector with higher resolution Analysis, Vol.4 pp.67-72, 2000. (SXGA or more pixels) to create more high3. Boerner R. “Three stereoscopic 1.25 m diagonal resolution image. A more large computation power real projection systems with tracking features ,” is also needed to correspond to multi-projector Proceedings of IDW, pp.835¯838, 1997. system. 4. H.Liao, et al., “Intra-operative Real-Time 3-D The IV autostereoscopic display system Information Display System based on Integral developed in this study has three primary merits: Videography,” MICCAI2001, LNCS 2208, pp.392Superimposing the high-quality real, intuitive 3-D 400, 2001. image for medical use; Avoiding the need of extra 5. M.G.Lippmann, “Epreuves reversibles donnant la devices such as the wearing of special glasses, sensation du relief,” J. de Phys Vol.7, 4th series, and offering a geometrical accuracy image over pp821-825, 1908. the projected objects (esp. depth perspective); 6. Y.Masutani et al., “Development of integral Visibility of motion parallax over a wide area, photography-based enhanced reality visualization simultaneous observation by many people. system for surgical support,” Proc. of ISCAS’95, In the application test, we found that the display pp16-17, 1995. devices developed in this study is massive. The 7. S.Nakajima, et al., “Three-dimensional medical multi-projector system must be made more small display with computer-generated integral and simplicity to be use in operating room, photography,” Computerized Medical Imaging and especially for the navigation use or image-overlay. Graphics, 25, pp235-241, 2001. This can be resolved by altering the design of 8. Peter S.Pacheco, “Parallel Programming with MPI,” projector and using the technology of the Digital Morgan Kaufmann, 1996. Micromirror Device (DMD) directly. 9. K. Li et al., “Building and using a scalable display
CONCLUSION In conclusion, we have developed a high resolution with high pixel density image device for IV autostereoscopic display system using multiprojector with long zoom lens projection technique. The pixel number can be made as large as necessary, and the pixel density can be adjusted to the most suitable conditions of the lens array to
wall system,” IEEE Computer Graphics and Applications, Vol.20, No. 4, pp.29-37, 2000. 10. Y.Endo et al, “A study of antireflective and antistatic coating with ultrafine particles ,” Advances Powder Technol., Vol.7, No.2, pp.131-140, 1996. 11. H.Hoshino, et al., “Analysis of resolution limitation of integral photography,” Optical Society of America, Vol.15, No.8, pp2059-2065, 1998.
1232
