An Immersive Free-Viewpoint Video System Using Multiple Outer/Inner Cameras Hansung Kim1, Itaru Kitahara1,2, Ryuuki Sakamoto1 and Kiyoshi Kogure1 1
Media Information Science Lab, ATR Keihanna Science City, Kyoto, 619-0288, Japan {hskim, skmt, kogure }@atr.jp 2 Dept. of Intelligent Interaction Technologies, Univ. of Tsukuba Tsukuba Science City, Ibaraki, 305-8573, Japan
[email protected]
Abstract We propose a new free-view video system that generates immersive 3D video from arbitrary point of view, using outer cameras and an inner omni-directional camera. The system reconstructs 3D modes from the captured video streams and generates realistic free-view video of those objects from a virtual camera. In this paper, we propose a real-time omni-directional camera calibration method, and describe a shape-from-silhouette technique for 3D modeling and a micro-facet billboarding technique for rendering. Owing to the movability and high resolution of the inner omni-directional camera, the proposed system reconstructs more elaborate 3D models and generates natural and vivid video with an immersive sensation.
1. Introduction The ultimate goal in visual communication is to realize an interactive 3D imaging system that can provide the feeling of reality as seen in SF movies such as “Star Wars” or “Minority Report.” Although these film are still science fiction, many researchers have tried to develop imaging systems that give realistic impressions of a scene. The range of applications for such a system is obviously enormous. With recent progress in computer and video technologies, many computer vision-based 3D imaging systems have been developed [1][2]. The most important requirement of a 3D imaging system is to create realistic images of dynamically changing scenes. Kanade et al. proposed the concept of “Virtualized Reality” as a new visual medium for manipulating and rendering prerecorded scenes in a controlled environment [3]. The system consisted of a geodesic dome equipped with 49 synchronized environmental video cameras. Recently, Kitahara et al. introduced “Cinematized Reality,” the aim of which is to record unexpected moments and create movie-like footage
by a virtual camera generated from eight environmental cameras [4]. However, these systems have limitations in 3D reconstruction and rendering because they use only fixed environmental cameras. First, they cannot reconstruct an object composed of multiple, disjoint objects. If there is any region occluded from environmental cameras, the reconstructed model may exhibit undesirable artifacts such as phantom volumes. Second, the environmental camera systems suffer a trade-off between the size of modeling space and the resolution of textures due to the limited resolution of the cameras. In the case of the “Virtualized Reality” system by Kanade et al., they used 49 cameras, but the modeling space was only about 6×6×3m. Third, it is difficult to generate fine immersive video. Usually, all environmental cameras are oriented inward from the outside of the modeling space. Therefore, when the virtual camera shoots a movie outward from the inside of the space, the video quality deteriorates because of the difference of resolution between the environmental cameras and the virtual camera. On the other hand, Yamazawa et al. and Ikeda et al. have proposed panoramic movie generation systems that use omni-directional cameras [5][6]. These systems provide adaptive scenes from the viewpoint of their users viewpoints. However, the systems can generate only outward scenes from the real camera position since they cannot overcome the occlusion problem. In this paper, we propose a new free-view video generation system that generates fine immersive 3D video from arbitrary viewpoints using multiple outer/inner cameras. We overcome the problems of previous systems by combining the advantages of the environmental camera system and the omni-directional camera system. In Section 2, we describe the outline of the proposed scheme and advantages of the system. Section 3 describes camera calibration, and Section4 4 explains the 3D reconstruction process. In Section 5 Virtual View Rendering is addressed and the simulation results are presented in Section 6. We conclude the paper in Section 7.
Environmental multiple cameras
3D Modeling
Figure 2. LadybugTM system Omni-directional camera
be calibrated in real-time if any one of the known markers is detected by one of sub-cameras. In this project we are using the Ladybug2TM system produced by Point Grey® shown in Fig. 2 as the omni-directional multi-camera system [7]. The Ladybug camera unit consists of six 1024×768 color CCDs, with five CCDs positioned in a horizontal ring and one pointing straight up. The Ladybug covers approximately 75% of a full sphere and provides video streams at 30 fps for each camera. The following sections describe in detail the algorithms and realization of the system.
Virtual scene from arbitrary viewpoint
Rendering
Figure 1. Proposed 3D video system
2. Immersive Free-Viewpoint System Figure 1 shows the configuration of the proposed system. We set up multiple environmental cameras on the wall and the ceiling to surround the target object and an omni-directional multi-camera inside the working space. All environmental cameras were oriented toward the center of the space to capture almost the same area. When target objects are captured by cameras, each capturing PC segments the objects and transmits the masks and color textures to a 3D modeling server via the UDP (User Datagram Protocol). The modeling server then generates 3D models of each object from the gathered masks. Finally, the server generates a video at the designated point of view with the 3D model and texture information. We can obtain many advantages by inserting an omni-directional multi-camera into the modeling space with the environmental camera system. The greatest advantage of the omni-directional camera is movablity. The camera can be moved to any place in the space to augment the quality of the video with high resolution, or, on other hand, to get rid of interference with working. For example, in a round-table meeting, the system can provide the best scenes of all participants by the omni-directional camera being placed at the center of the table. Second, it is easy to calibrate the omni-directional multi-camera in real-time. Since the omni-directional multi-camera covers a very wide FOV (field of view), all sub-cameras in the camera unit can
3. Camera Calibration Camera calibration refers to determining the values of the camera’s extrinsic and intrinsic parameters. The key idea behind calibration is to write projection equations linking the known coordinates of a set of 3D points and their projections, and to determine the camera parameters. The following camera parameters are extracted for our system. Projection matrix
⎡ p11 P = ⎢⎢ p 21 ⎢⎣ p 31
p12 p 22 p 32
p13 p 23 p 33
p14 ⎤ p 24 ⎥⎥ p 34 ⎥⎦
Extrinsic parameters 3x3 rotation matrix
⎡ r11 R = ⎢⎢r21 ⎢⎣r31
[
3D translation vector T = t x
r12 r22 r32 ty
r13 ⎤ r23 ⎥⎥ r33 ⎥⎦ tz
]
T
Intrinsic parameters Lengths of effective pixel size units: sx and sy Image center coordinates: cx and cy Distortion parameters: k1, k2, p1 and p2
Generally, there are two approaches to camera calibration. The first method is to directly recover the intrinsic and extrinsic parameters, and the second (introduced by E. Trucco and A. Verri [8]) is to estimate the projection matrix first, without solving explicitly for the
y R i , Ti
x
Ci
z M
z R△i, T△i
W
z
y x
x
CD y
RD, TD
Figure 4. Changing coordinate systems in 3D space Figure 3. Camera calibration equipments
various parameters, which are then computed as closed-form functions of the entries of the projection matrix. For environmental camera calibration, we use the second method because it is simpler than the first one, and a projection matrix is used directly to reconstruct the 3D model. For the omni-directional multi-camera, we use the first method because the extrinsic parameters of each sub-camera are updated by the physical relationship between sub-cameras.
3.1. Environmental camera calibration One of the most difficult problems with camera calibration in a daily living area is setting up an accurate 3D world-coordinate system and effectively arranging a series of landmark points, because there are many variations to space size and there may also be many obstacles in that space. If we cover the space with a single scale or a single calibration board, accurate camera calibration is difficult because the scale may be too large, or there may be obstacles occluding some landmarks in the captured image. Our solution is to combine mobile calibration markers and a 3D laser-surveying instrument, which are usually used in civil engineering. Figure 3 shows the equipment we have used in the experiment. To accurately calibrate the camera, it is necessary to obtain many pairs of 3D coordinates (X,Y,Z) in the scene and 2D coordinates (u,v) on an image, thus by using the 3D laser-surveying instrument, we accurately obtain the location of the calibration board in the 3D space. In fact, the measurement error is less than 0.1 mm. By moving the calibration markers to cover the entire 3D space and by measuring its 3D location accurately, it is possible to virtually realize a calibration scale that possesses high shape freedom. Furthermore, our camera calibration method features the two following advantages. First, it is easy to increase the quantity of 3D coordinate data (X,Y,Z) with linear interpolation of the actual measurement values. Second, it is possible to almost automatically detect the landmark
point in the captured images by painting it with a discriminative color.
3.2. Omni-directional camera calibration Once we have calibrated all the camera parameters of the environmental cameras we can use them permanently: they never change since the cameras are fixed to the wall. On the other hand, the extrinsic parameters of the omni-directional camera can be changed by moving it inside the modeling space. Therefore, we need to extract intrinsic and extrinsic parameters independently. However, calibrating all extrinsic parameters of six sub-cameras in real-time is almost impossible because feature points for calibration can be occluded, and detection and calibration for all sub-cameras are time-consuming processes. We propose extracting the extrinsic parameters of the other cameras from that of one sub-camera by using the geometrical relationship between sub-cameras. Changing from one coordinate system to the other system can be described as follows, where R1,2 and T1,2 are a rotation matrix and a translation vector from the coordinate 1 to coordinate 2, respectively. M 2= R1, 2 M 1 + T1, 2
(1)
Figure 4 shows the relationship between the global coordinates and those of each camera. When CD is a camera coordinate of the reference sub-camera and Ci is that of the other camera, the following relationship can be derived from Eq (1) and Fig. 4. ⎧⎪M Ci = RD M W + TM ⎨ ⎪⎩M Ci = RΔ (RD M W + TM ) + TΔ ⎧ Ri = RΔ RD ⇒⎨ ⎩Ti = RΔ TM + TΔ
(2) (3)
In Eq. (3), R△ and T △ are stationary values for each sub-camera because all sub-cameras in Ladybug have a fixed geometrical relationship. Therefore, once we extract
Captured Image
Voxel V(X,Y,Z)
I1
Camera C3 Camera C1
Estimated 3D Shape M(X,Y,Z)
P2
P1
I2
Pv
Captured Image
Camera C2
Iv Virtual Camera
Figure 5. 3D reconstruction by shape-from-silhouette
Figure 7. Synthesis of a novel view with an estimated 3D shape Voxel
Micro-facet Billboarding micro-facet billboard
voxel
3D Object
3D Object
Figure 6. Octree structure
Input Image1
the parameters R△ and T△ for each camera by Eq. (4) in advance, we can calculate all the other extrinsic parameters from the parameters of one camera. ⎧⎪ RΔ = Ri RD ⎨ ⎪⎩TΔ = Ti − RΔTM −1
(4)
In our system, we set up feature points on the ceiling of working space. The vertical sub-camera in Ladybug is used for real-time extrinsic camera calibration.
4. 3D Reconstruction Since the 1980s, many works on computational models for computing surface shape from different image properties have been produced [9].In order to reconstruct the 3D shape of the captured object, we employ a shape-from-silhouette method. The shape-from-silhouette method is a very common way of converting silhouette contours into 3D objects [10]. Silhouettes are readily and easily obtainable and the implementation of the shape-from-silhouette method is generally straightforward. Figure 5 illustrates the relation of the multiple cameras and the voxels that are set up in the 3D space. Each camera is labeled as Cn, a captured image with the camera as In (n=1,…,N), and each voxel as V(X,Y,Z). Now, for example, let us assume that V(X,Y,Z) is a voxel inside the modeling space and Ip is a subset of In that include the projected point of V in their imaging area. If the 3D position of V is inside the 3D object, V must be projected onto the foreground regions of all the images Ip. Therefore, if there is any single projected point located in
Input Image2
Reverse Sequence
Virtual Image
Input Image2
Normal Ascending Order
Virtual Image
(a) Texture mapping on voxels (b) Micro-facet billboarding Figure 8. Texture mapping process
the background region in the images Ip, the voxel V is carved out from the 3D shape model. As a result, we can estimate the entire 3D object’s shape by examining every possible position of the voxels V(X,Y,Z). Testing all the points in a modeling space is, however, a very time-consuming process and results in excessive data. Therefore, we use an octree data structure for modeling [11]. For each voxel of a given level, 27 points (i.e., each corner and the centers of the edges, faces and a cube) are tested. If all checking points are either included in or excluded from an object, the voxel is assigned as a full or empty voxel, respectively. Otherwise, the voxel is split into eight sub-voxels and is tested again at the next refinement level. Figure 6 shows the structure of the octree. This structure dramatically reduces the modeling speed and the amount of data.
5. Virtual View Rendering Figure 7 illustrates the geometric relation between a scene point M(X,Y,Z), captured images In, and the image plane Iv of a virtual camera. We consider that the 3D model represents only the correspondence among the input multiple videos concerning a scene point M. The model
2.5m
5.5m 5.5m
Figure 9: Layout and captured images by the propose system
tells us where a point M exists in each captured image. Furthermore, when the position and orientation of the virtual camera are specified, the projective relation between a scene point M and a point on the image Iv is defined with a projective transformation matrix Pv. Here, we address the rendering of a scene point M(X,Y,Z) on the image of virtual camera Iv. Suppose a scene point M is projected to a point on the image Iv with the projective transformation matrix Pv, and M is observed in several captured images In. To select the most suitable texture to be used for rendering, we investigate occlusion among objects, the orientation of objects’ surfaces, and the distances from the surface to the real cameras. Fundamentally, a voxel-based structure is not a very good approximation for the object surface since the mesh obtained from the voxel is too dense, and the orientations of the mesh faces suffer from severe quantization as they are strictly aligned with the coordinate axes. A micro-facet billboarding technique is employed to solve this problem [12]. This technique was originally developed in photometric rendering research to express a 3D object that has fine texture, like a flowing coat or a jagged shape. Nevertheless, though the advanced 3D laser range finder can accurately measure 3D shape, the measurable spatial resolution is still lower than that of a high-resolution digital camera. Therefore, if the 3D object is described at a lower resolution, appearance information on the captured images will be wasted. On the other hand, the billboarding technique approximates a 3D object’s shape to a single plane and expresses the object’s appearance by mapping its captured image. This technique has the advantage of not wasting the resolution of captured images, even if an accurate 3D shape is not available. The micro-facet billboarding technique, which implements the advantage of the billboarding technique on 3D modelling,
can express more complicated shapes by encrusting micro-facet billboards onto the surface of the estimated 3D shape, while still applying all the advantages of the billboarding technique. In Fig. 8, we illustrate the difference of the texture mapping process between the voxel volume and the micro-facet billboarding technique. Since the normal vector of each surface of a voxel differs greatly from the correct normal vector of the 3D object’s surface, when we generate a virtual image with mapping captured images onto the estimated voxel volume, the order of the appearance of the 3D object is not often retained. As a result, many dots and cracks are observed on the surface of the 3D object. On the other hand, as shown in Fig. 8(b), the micro-facet technique absorbs the difference by controlling the orientation of each billboard. Consequently, the order of the appearance of the 3D object is always maintained.
6. Simulation Results We have implemented a distributed system using nine PCs, seven calibrated USB2.0 color cameras and one Ladybug2. The size of capturing space was about 5.5×5.5×2.5 m, and seven environmental cameras were set on the wall to surround the space. All cameras were oriented to observe the center of the space. Seven Pentium IV 2.8-GHz PCs for environmental cameras and one Pentium IV 3.6-GHz PC for Ladybug were used in order to capture a video stream from each camera and segment objects at 20 fps (frames per second). An intensity-based background subtraction method was used to segment the foreground and background regions in the input multiple images [13]. The segmentation masks and texture information are sent via UDP over a 100-Mbps (bits per second) network to the modeling and rendering PC. The
Figure 10: Evaluation of calibrated parameters
modeling and rendering PC features a Pentium IV 3.6-GHz CPU and a FireGL V3100 graphic accelerator. Figure 9 shows a set of snapshots from videos recorded with this layout. However, modeling and rendering speeds with the PC were not enough to perform them in real time so that the modeling and rendering process were done off-line in this experiment.
6.1. Real-time Ladybug calibration The Ladybug can be moved in the space according to efficiency of capturing or inconvenience to working people. Therefore, all extrinsic parameters of sub-cameras should be updated in real-time. The system calibrates one of the sub-cameras using a chessboard pattern attached on ceiling, and calculates all parameters of the other sub-cameras using Eq. (3) in real time. Because each lens of Ladybug has a viewing angle exceeding 80 degrees, the pattern can be captured at any place in the capturing space. We tested the accuracy of the extracted parameters by projecting global points measured by the 3D laser-surveying instrument to the image plane (up, vp). The error between projected points (up, vp) and detected image point (uc, vc) by corner detection was calculated. We tested 48 points in a chessboard pattern for each sub-camera. In Fig. 10, red circles represent 3D points projected onto the image planes, and Table 1 shows the mean and standard deviation (SD) of errors in pixels for each camera. The average errors were less than 0.5 pixels in the horizontal (u) and vertical (v) directions, and the distance error (d) was less than 0.8 pixels. TABLE 1. PERFORMANCE EVALUATION OF THE ESTIMATED PARAMETERS
Mean (pixels)
SD (pixels)
Subcamera
u
v
d
d
Cam1 Cam2
0.4611 0.4454
0.3091 0.4713
0.6295 0.6566
0.3401 0.3425
Cam3
0.4822
0.4598
0.7346
0.4130
Cam4
0.4633
0.4362
0.6928
0.3694
Cam5
0.4381
0.3510
0.6415
0.3409
(a) Without Ladybug (b) With Ladybug Figure 11: Results of 3D modeling
6.2. 3D reconstruction To verify the performance of the proposed system, we compared reconstructed models without/with the omni-directional camera. All images from the environmental cameras were captured at a resolution of 640×480 and the Ladybug at 1024×768 for each sub-camera. The 3D space was modeled at a resolution of 300×300×200 on a 1×1×1-cm voxel grid. Segmentation in this experiment was performed in a semi-manual way in order to avoid errors in the 3D model caused by segmentation error and to confirm the contribution of the omni-directional camera to modeling. Figure 11 shows 3D models generated by the shape-from-silhouette method without/with the Ladybug, respectively. We can see that the final models generated without Ladybug are coarse and failed in carving occluded areas from environmental cameras. On the other hand, the models generated using both environmental cameras and the Ladybug look more natural because the redundancies in the models are carved out by the inner cameras.
6.3. Virtual view rendering By using the algorithm described above, the proposed system synthesizes a 3D video from the position of the virtual camera. Because the system is installed in an environment that might have unfavorable conditions for 3D
Figure 12: Generated micro-facet billboards and selected cameras for texturing
(a) Without Ladybug (b) With Ladybug Figure 14: The effectiveness of the inner camera
(a) Texturing on voxels
(b) Texturing with micro-facet billboarding Figure 13: The effectiveness of the micro-facet billboarding
modeling such as a typical office environment, the reconstructed 3D shape includes many computational errors. Thus, if we just map the captured image as texture information, the quality of the generated 3D video will suffer seriously. As a solution, we employ the micro-facet billboarding technique to curb the improper influence of such errors. Each micro-facet billboard corresponds to a voxel; in other words, the center of both corresponding components of the 3D model should lie at the same 3D position. The orientation of the micro-facet billboards is set to ensure that the normal vector of the billboard and a virtual camera’s line of sight are parallel, just as with the ordinary billboard technique. To reduce the computational complexity, the system generates micro-facet billboards only on the surface facing the virtual camera, and selects textures of the closest camera by calculating distance and angles from the virtual camera to real cameras for each micro-facet billboard. In the proposed system, we gave priority to the Ladybug in camera selection because the Ladybug has a higher resolution than the environmental
cameras in addition to a very wide FOV. Figure 12 shows generated micro-facet billboards and selected cameras for each facet. The color of the surface corresponds to the color of each camera. As a result, the micro-facet billboarding technique covers the appearance deficit generated by errors in 3D shape estimation as shown in Fig. 13. We also tested the effectiveness of the omni-directional camera inside the modeling space on texturing. Figure 14 shows the same model textured without/with the Ladybug. When we use the Ladybug, the virtual camera can capture much more distinct textures because the Ladybug has a higher resolution and is placed nearer to the objects. The other advantage of using the omni-directional camera is that it can be moved to the optimal position to augment the video quality. On the second image set in Fig. 14, the Ladybug placed in a low position provides very good textures in looking up camera angles. Finally, Fig. 15 shows snapshots of the generated 3D free-viewpoint videos. It is clear that the system generally renders natural scenes from any point of view.
7. Conclusion We have presented a complete 3D imaging system using multiple outer/inner cameras. The system reconstructs 3D models from the captured video streams and finally displays realistic images of those objects from arbitrary viewpoints. We proposed a real-time omni-directional multi-camera calibration method, and used the shape-from-silhouette technique for 3D modeling and the micro-facet billboarding technique for rendering to generate fine free-view video. Cooperation between
Figure 15. Snapshots of the generated 3D free-viewpoint videos
environmental cameras and the omni-directional camera solved the limitation of previous systems and provides more natural and vivid videos. Future work will include improving segmentation performance, boosting the processing speed, and developing a texture-blending algorithm from multiple images to generate more natural surface texture.
[5]
[6]
Acknowledgement This research was supported by the National Institute of Information and Communications Technology.
References [1] M. Gross, “Blue-c: A spatially immersive display and 3D video portal for telepresence,” SIGGRAPH03, pp. 819-827, 2003. [2] X. Wu, T. Matsuyama, “Real-time active 3d shape reconstruction for 3d video,” Proc. International Symposium on Image and Signal Processing and Analysis, pp. 186-191, 2003. [3] T. Kanade, P. W. Rander, and P. J. Narayanan, “Virtualized Reality: Constructing Virtual Worlds from Real Scenes,” IEEE Multimedia, Vol. 4, No. 1, pp. 34-47, 1997 [4] I. Kitahara, R. Sakamoto, M. Satomi, K. Tanaka, and K. Kogure, “Cinematized Reality: Cinematographic Camera Controlling 3D Free-Viewpoint Video,” 2nd IEE European
[7] [8] [9] [10] [11]
[12]
[13]
Conference on Visual Media Production (CVMP2005), pp. 154-161, (2005) K. Yamazawa, H. Takemura, and N. Yokoya, “Telepresence system with an omnidirectional HD camera,” Proc. 5th Asian Conf. on Computer Vision (ACCV2002), Vol. II, pp. 533-538, Jan. 2002. S. Ikeda, T. Sato, and N. Yokoya, “Panoramic movie generation using an omnidirectional multi-camera system for telepresence,” Proc. 13th Scandinavian Conf. on Image Analysis (SCIA2003), pp. 1074-1081, July 2003. http://www.ptgrey.com/ E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision, New Jersey: Prentice Hall, 1998. L. G. Shapiro and G. C. Stockman, Computer Vision, chap. 12, New Jersey: Prentice Hall, 2001. W. Matusik, Image-Based Visual Hulls, MS Thesis, MIT, 1997. R. Szeliski, “Rapid octree construction from image sequences,” CVGIP: Image Understanding, vol.58, pp.23-32, 1993. S. Yamazaki, R. Sagawa, H. Kawasaki, K. Ikeuchi, and M. Sakauchi, “Microfacet billboarding,” Proc. 13th Eurographics Workshop on Rendering, 2002, pp. 175–186, 2002 H. Kim, I. Kitahara, K. Kogure, N. Hagita, and K. Sohn, “Sat-Cam: ‘Personal Satellite Virtual Camera,’” Proc. PCM 2004, LNCS. vol. 3333, pp. 87-94, Nov. 2004.
