An Improved Virtual View Rendering Method Based on Depth Image Mingsheng Li, Hui Chen, Ranran Li and Xin Chang School of Information Science and Engineering, Shandong University, Jinan 250100, China
[email protected],
[email protected] Abstract - Depth image based rendering (DIBR) is an important technology for virtual view synthesis in 3DTV. This paper presents a bi-directional depth image based rendering system without preprocessing depth image. An algorithm for three-dimension (3D) warping which declines exposed area is also proposed, and so is an improved merging algorithm for eliminating image ghosting. The proposed DIBR scheme can reduce computation cost compared to previous techniques. Experimental results show that edge ghosting can be completely eliminated in our method. We also demonstrate that both objective and subjective effects of the paper are better than the previous ones.
demanding. Pre-processing depth-image is another effective mean, which could reduce the size of exposed areas in the target image. Different filters are used to pre-processing depth-image, such as average filter [4], symmetric Gaussian filter [2], asymmetric Gaussian filter [5], edge dependent depth filter [6] et al. These measures all zoom out the exposed areas and reduce the number of small holes. But the depth perception is weakened and distortions appear because of the depth image smoothing. Furthermore, filling the holes is another difficult task, it is easy to cause some artifacts that are not easy to eliminate. In addition, the quality of created image has serious correlation with the baseline distance. Bidirectional DIBR using two sides color images and their associated depth-images of the object image can render arbitrary virtual view. The left (or right) warping image can acquire the disocclusions information from the right (or left) warping image, so the holes are filled more rational and the result is better. In this paper, we present an improved bi-directional DIBR to render arbitrary virtual view. The number of cracks is greatly reduced because of using the “one to many” algorithm in warping stage. An improved algorithm for image blending can effectively take away the edge ghosting; meantime, it can effectively reduce the computational burden. The experiments show that both the subjective and objective results are better than the previous work. The remainder of this paper is organized as follows. In section II, a brief introduction of the DIBR system and its own problems is given. In section III, the improved bidirection DIBR system is presented in details. We give the experiment results and discussions in section IV. Finally, the conclusion and further work are showed in section V.
Keywords - 3DTV, DIBR, 3D Warping, View Synthesis.
I.
INTRODUCTION
Virtual view synthesis can be classified into two different technologies: model-based rendering (MBR) and image-based rendering (IBR) [1]. The former one is based on building a 3D model to render virtual view. However, constructing the model of an object is complex, especially when an unexpected object model is required. Contrast to MBR, the IBR technology has an advantage that it doesn’t require 3D details of the scene objects to generate virtual view image, since it uses adjacent images of the scene to render the virtual views. In doing so, naturalness of the virtual view image is preserved. Another advantage is that it can render virtual image in real time. Considering the contents of traditional 2DTV and High Definition TV (HDTV) can be converted to 3DTV ones with the IBR technology, it is particular advantageous to meet the requirement of backwards-compatibility in today’s broadcast television system. So, IBR has broad prospects for development in 3DTV. Depth Image Based Rendering (DIBR) is the most important and latest method in many IBR methods which has several attractive features [2]. DIBR can theoretically render arbitrary view using a reference color image and its associated depth map which gives the per-pixel depth information. However, DIBR has its own limitations; for example, the disocclusion is one of the difficult issues, which means that the occlusion area in the original image becomes visible in virtual new view owing to the change of view point and depth discontinuous. To deal with it, many different methods are presented. One possible solution termed the layered-depthimage (LDI) [3], which uses more several pairs of associated color images and depth image, can get an acceptable result. However, considering the practicality, the LDI is not appropriate in 3DTV system, because of more producing complexity, transmission bandwidth and computationally
II. PROBLEMS IN DIBR SYSTEM A. DIBR System In many previous works, the basic block diagram for DIBR model is given. The three steps of DIBR system is showed in figure 1: pre-processing depth image, 3D image warping, and holes filling. Virtual image in target view can be rendered after using the three steps in terms of theory. Pre-processing the depth image mainly has two purposes: one primarily aims at improving the precision of the depth map, especially the
Figure 1. Basic diagram of DIBR system
___________________________________ 978-1-61284-307-0/11/$26.00 ©2011 IEEE
accuracy of edge area, the other purpose to reduce the disocclusions by diminishing the depth difference using different smoothing filters. 3D image warping generally can be divided into two steps, the points in reference image coordinate system projecting to the world coordinate system in the first step and in the second step, the points in world coordinate system reprojecting to the object image coordinate system. The 3D warping function is as below: W Pr 1 X r , (1)
(a)
X o PoW . (2) The two formulas are corresponding to the two steps. Where Pr 1 is the reference camera projection inverse matrix, Po is the objective camera projection matrix. Besides, W is the pixel coordinate in the world coordinate, X r is the pixel coordinate in reference image and X o is the coordinate in objective image. Holes filling, the last step is not necessary if no holes exist in the objective image. However, using the current methods, it always produces some holes more or less, so it is necessary to fill the holes in the virtual view image. Next, we will introduce some own problems in the DIBR and some methods to resolve the problems. B. Visibility In the 3D warping, two or more pixels in reference image may be projected to the same position in the objective view. Actually, the front pixels should be seen rather than the back, that is to say, the point with small depth value should be presented. But sometimes the situation is the opposite, the foreground points which should be seen are blocked up by the background points, Figure 2 illustrates the issue. The visibility problem can be eliminated adding the depth value comparison in the warping, such as the classic z-buffer algorithm [6]. C. Disocclusions and Cracks After 3D warping, some disocclusions and cracks emerge in the virtual view image. The disocclusions are caused by that some occlusion areas in the reference image expose in the depth discontinuity area in the target view. Especially, for the left reference image, disocclusions will appear in the low to high sharp depth transition, and for the right reference image, disocclusions will appear in the high to low sharp depth transition. The defect is show in figure 3.
(a)
(b)
Figure 3. (a) disocclusions and cracks problems. (b) original image
In order to diminish the disocclusions, Fehn presented a creative method, smoothing the depth map with a symmetric Gaussian filter [2], which decreases the image quality and causes some distortion in vertical direction although it reduces the holes number. Liang Zhang et al gave an improved method with an asymmetric Gaussian filter to smooth the depth image [5], which improves the distortion in vertical line. Wan-Yu Chen et al [6] proposed an interesting method smoothing the depth image using an edge dependent depth filter and acquired a better result. Luo process the depth map using morphology algorithm [7]. The above methods decrease the exposed areas and reduce the number of small holes, but it inevitably caused some problems at the same time, such as loss of depth perception, geometric distortion etc. After smoothing the depth image, complex and time-consuming inpainting algorithms are still needed to fill the holes, such as Criminisi’s algorithm [8]. As to cracks, which is caused by the rounding and nearby interpolation when the reference pixels are mapped onto noninteger coordinates in the objective view (see figure 3). A simple interpolation can fill cracks because the cracks are small. D. Edge Ghosting Edge ghosting is an intractable problem in DIBR system. It is because of the inaccurate camera parameters or the error edge matching of color image and associated depth image, so the edge of the foreground objects are projected onto the pixels in the background, forming a vague outline of the edge of the foreground object in the background. Figure 4 illustrate the edge ghosting. Removing the ghosting is a complex and time-consuming work, and there are general two ways to solve it, ignoring the pixels causing ghosting in the warping and expansion the edge area of holes.
(b)
Figure 2. (a) visibility problem. (b) original image (a)
(b)
Figure 4. (a) edge ghosting problem (b)original image
III. IMPROVED DIBR SYSTEM
Edge ghosting usually appears after the blending using traditional ways. To diminish the edge ghosting, boundary dilation method was proposed in [10], before the blending, the holes that will bring edge ghosts are found in the left and right reference image, and the edge of holes are expanded to eliminate the image ghosting. The ghosting are removed fully using this method, meantime, the computation is increased, too. Besides, another method eliminates the ghosting well by ignoring the pixels which will produce edge ghosting in the warping step [11], but this method need to determine the pixels which causes the edge ghosting. We present an improved fast blending method to synthesis the left and right image, meantime eliminates the edge ghosting. The improved method need not to expand the holes or find the points causing artifacts, thereby it greatly reduce the computation. The classic image blending method can be illustrated using the formula as below: f v (u, v) D f L (u, v) (1 D ) f R (u, v) (3) f v is final synthesis image, f L is the objective view image created by the left reference image, and f R is the objective view image created by right reference image, D is the weighting factor, it is calculated by the following formula usually: tV t R , (4) D tV t R tV t L
In order to quickly and efficiently to solve the above problems, the improved DIBR system is presented to synthesis arbitrary virtual view (see figure 5). In traditional bi-directional DIBR system, it consists of the following four steps: pre-processing depth image, 3D image warping, blending and holes filling. Compared with the one side DIBR system, the bi-directional DIBR system appends the blending step. The left (right) image disocclusions can get information from the right (left) image, so the arbitrary view rendering become possible in the large baseline. The improved DIBR system can be used to reduce computation, improve the computing speed, and generate a good final image. The improved method reduces computation mainly owing to three parts: first, it does not need to pre-process the depth image. Second, the improved blending method can synthesis the two images and eliminate the edge ghosting with less computation. Third, in the last stage, the holes are so small in the final image that it can be filled without complex inpainting algorithm. In the warping, Feng et al [9] present a new warping method that one point projects to many points with thinking over the depth and distance of pixels comprehensive. This new method can effectively take away the cracks but the computation increases much because of considering the depth and the distance. So we simplify the algorithm without considering the depth and distance, the simplified algorithm can reduce the computation and get an acceptable result. Figure 6 give the result using the simplified method.
where t is the camera translation vector. The improved blending method can be expressed by the following formula: , )(1D)fR(uv , ) others, DfL(uv ° , )z0, fR(uv , )z0, (5) fR(ut,v) 0, fL(uv , ) ®fL(u,v) fv(uv °f (u,v) fL(ut,v) 0, fL(uv , )z0, fR(uv , )z0. ¯R where t is the threshold which represents pixel distance and we set t = 5. In our experiments, D is the weighting factor calculated by formula (4) and the pixel value is zero where the points are in the disocclusions and cracks area. Using the improved synthesis algorithm eliminates the edge ghosting effectively, and the subjective effect is well (see figure 7).
Figure 5. Improved bi-directional DIBR
(a)
(b)
Figure 7. (a) traditional blending method with edge ghosting (b) our improved blending method without edge ghosting.
(a)
At last, the synthesis final image has few tiny holes, but filling the small holes is an easy work, it can be interpolated using the background texture.
(b)
Figure 6. (a) simplified one to many method (b) traditional warping method.
IV. EXPERIMENTAL RESULTS
Our algorithm is tested using the breakdancers video sequences supplied by interactive visual media group of microsoft research [12]. The eight different video sequences consist of color image and its corresponding depth image, and
In this paper, an improved 3D warping algorithm for decreasing exposed area and cracks is proposed, and a merging algorithm for eliminating image ghosting is presented too. Besides, the improved system reduces the computation cost, so both the computation space and the operation time are saved. Our proposed algorithm can obtain perfect results in the objective and the subjective. As further work, we will improve our algorithm in the holes filling, and implement on the GPU to further improve our system in quality and speed. ACKNOWLEDGMENT
(a)
This work is supported by the Natural Science Found of Shandong NSFSD under No.2009ZRB01675 and the NSF of China under No.60872119.
(b)
REFERENCES [1].
[2]. (c)
(d)
Figure 8. Results of different algorithm without holes filling (a) real image (b) proposed algorithm (c) traditional bi-directional DIBR algorithm (d) classic Gaussian filter TABLE I THE COMPARISON OF PSNR AND SSIM Algorithm classic Gaussian filter traditional bi-directional DIBR proposed algorithm
PSNR 23.683 30.425 31.725
[3]. [4].
SSIM 0.6147 0.8253 0.8567
[5]. [6].
the camera parameters of each view are offered too. In our experiments, the view 3 is our objective view, and the both sides view are the left and right reference view. The comparison uses the traditional bi-directional DIBR, the classic Gaussian filter, and our improved bi-directional method. The result showed in Figure 8. The object view using the traditional bi-directional DIBR method still has some cracks and edge ghosting, although it is better than the classic Gaussian filter method. Using the presented method in this paper, the computation cost is reduced and the edge ghosting is removed too. The proposed method achieves a better result in subjective compare with other methods. At the same time, the Peak Signal to Noise Ratio (PSNR) and the Structural SIMilarity (SSIM) [13] of the present method are illustrated comparing with other methods in the Table I.
[7]. [8]. [9].
[10].
[11]. [12]. [13].
V. CONCLUSIONS
H. Saito, S. Baba, M. Kimura, S. Vedula, and T. Kanade. Appearancebased virtual view generation of temporally-varying events from multicamera images in the 3D room. Technical Report CMUCS99127, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, Apr.1999. C. Fehn, “Depth-Image-Based Rendering(DIBR),the compression and Transmission for a New Approach on 3DTV,”In Proceedings of SPIE Steroscopic Displays and Virtual Reality Systems , 2004,pp.93-104. J. Shade, S. Gortler, L.-W. He, and R. Szeliski, “Layered Depth Images," in Proc. of ACM SIGGRAPH '98, pp. 231-242, (Orlando, FL, USA), July 1998. Chung J. Kuo, Ching Liao, and Ching C. Jin, “STEREOSCOPIC IMAGE GENERATION BASED ON DEPTH IMAGES”, ICIP, Singapore, October. 2004, pp. 2993-2996 L. Zhang, J. Tam, and D. Wang, “Stereoscopic image generation based on depth images,” in Proc. IEEE Conf. Image Processing, Singapore, pp. 2993–2996 Oct. 2004. Wan-Yu Chen, Yu-Lin Chang, Shyh-Feng Lin, Li-Fu Ding, and LiangGee Chen, Ā Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation.ā Multimedia and Expo, 2005. ICME 2005. IEEE International Conference, 1314- 1317ˈJuly 2005. LUO Kai, LI Dongxiao, FENG Yamei, “Arbitrary View Generation Based on DIBR and Depth-Aided Image Inpainting,” Journal of Image and Graphics,vol.15,pp.444-449,2010. $ &ULPLQLVL 3 3HUH] DQG . 7R\DPD ³5HJLRQ ¿OOLQJ DQG REMHFW removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200–1212, Sep. 2004. Y. M. Feng; D. X. Li, K. Luo, and M. Zhang, “Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video,” IEEE Trans. Consumer Electronics, vol. 55, no. 4, pp.2349 – 2355, Nov. 2009. K.-J. Oh, S. Yea, and Y.-S. Ho, “Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video,” in Proc. Picture Coding Symp., Chicago, IL, May 2009, DOI: 10.1109/PCS. 2009.5167450. L Do, S Zinger and P H.N. de, “Quality improving techniques for freeviewpoint DIBR” - Proceedings of SPIE, 2010. MSR 3D video Sequences. Available at: http://www.research. microsoft.com/vision/ImageBasedRealities/3DVideoDownload/. Z. Wang, A.C. Bovik, H.R. Sheik, “Image quality assessment: From error visibility to structural similarity,” Proc. of IEEE Transaction on Image Processing ,vol.13,pp.600–612,2004
