Vector field fitting for real-time environment matting of transparent ...

VECTOR FIELD FITTING FOR REAL-TIME ENVIRONMENT MATTING OF TRANSPARENT OBJECTS Qi Duan

Jianfei Cai

Jianmin Zheng

School of Computer Engineering Nanyang Technological University, Singapore 639798 Email: {duan0013, asjfcai, asjmzheng}@ntu.edu.sg ABSTRACT In this research, we focus on real-time environment matting of transparent objects. The major drawback of real-time environment matting is that the extracted matte data often contains significant amount of noise and errors. Although some filtering process has been employed in the existing real-time environment matting methods to remove the noise to obtain acceptable composition results, it fails in the existence of errors. Here, we propose a method to simultaneously remove both noise and errors in the extracted matte data. In particular, we establish a light motion field to describe the environmental matting of transparent objects. We formulate the artifact removal task as an energy minimization problem and propose a vector fitting algorithm to remove noise and rectify errors. Experimental results show that, compared with the existing real-time environment matting approaches, our method is less sensitive to noise and can achieve perceptually better composition results. Index Terms— Environment matting, Light motion field, Vector field fitting, transparent objects. 1. INTRODUCTION Digital image matting and composition are very important image editing techniques, which can generate interesting and fantastic effects for various application such as film making. The core aspect of image matting is to extract the foreground object and its related information from a source image. In a large sense, image segmentation is one type of image matting techniques. Specifically, when the foreground object is opaque and its boundary is sharp, image matting becomes the same as image segmentation. However, when handling other types of objects, image matting technique becomes more suitable since we need to extract not only the object boundary but also other related information. Smith and Blinn [1] proposed a blue screen matting method and formally formulated the matting problem as C = αF + (1 − α)B

(1)

where C, F, B are color of the captured image, the foreground object and the background scene, respectively, and α is a parameter to represent the opacity information of the object. The value of α is between [0, 1] while it only takes one of the two values {0,1} in image segmentation. According to Eq. (1), the matting problem can be summarized as: given the color of the captured image C, how to derive F , B and α? Clearly, image matting is an ill-posed problem since it has more variables than equations (e.g., for a color image there are 7 variables with only 3 equations). Many approaches have been proposed to solve this image matting problem including Bayesian statistic method [2], Poisson method [3] and iterative optimization approach [4]. For transparent objects, the matting problem becomes even more complicated because it involves refraction and reflection information extraction. Zongker et al.[5] first proposed the concept of environment matting, which refers to the matting with the addition consideration of refraction and reflection effects. They computed the environmental matte data through utilizing many specially designed projection patterns with multiple images being captured. Later on, Chuang et al. [6] proposed two extensions to extend the environmental matting work in [5] in order to enhance the usability of the environment matting technique. The first extension is targeted to improve the matting accuracy through using more complex oriented elliptical patterns. The second extension aims to achieve real-time computation by making assumptions and simplifying the matting process with only one shot picture against a special backdrop. There are some other solutions for environment matting. Wexler et al. [7] assumed that every background pixel has a contribution to a foreground pixel and used a probabilistic model to calculate the contribution to extract the matte. Peers and Dutr´e [8] connected the environment matting algorithm with image-based relighting techniques and used various wavelet pattern to extract matte. Zhu and Yang [9] proposed a frequency-based method to find a more accurate mapping than those in [5, 6], and calculated the frequency response of foreground pixels to the time-sequence backdrop through Fourier analysis.

Most of the above methods require huge number of image samples, especially the wavelet method [8] and the frequencybased method [9], which make the capture process very timeconsuming. On the other hand, the real-time environment matting algorithm [6] is more practical and has been widely used in many applications, although it is very sensitive to noise and requires some filtering process to obtain acceptable results. In this paper, we are interested in real-time environmental matting of transparent objects. We introduce a new viewpoint, i.e. light motion field, to interpret the environment matting in a more intuitive way. In particular, based on this viewpoint, we first apply the real-time environment matting algorithm [6] to extract light motion vectors. Then, we propose a light motion field fitting algorithm to remove both noise and errors in the initial light motion vectors. We formulate the artifact removal task as an energy minimization problem. Experimental results show that, compared with the existing real-time environment matting approach, our method is less sensitive to noise and can achieve perceptually better composition results. 2. SYSTEM SETUP AND PROBLEM STATEMENT 2.1. A New Viewpoint on Environment Matting According to the optical principle, after complicated light refraction and transportation, the light intersecting a transparent object will change its original route and hit at a different point or region of background. In a captured image, it can be considered that each pixel belonging to the foreground object has a corresponding mapping center in the background. We call the vector starting from the foreground point and ending at its corresponding mapping center as light motion vector, which describes the route change of light transportation. We propose to establish such a light motion field to describe the refraction and reflection properties of the entire transparent object. It is clear that the shape and material properties of the transparent object determine the property of the light motion field since they are the key factors affecting the refraction effect. On the other hand, the property of such light motion field reflects the physical properties of the object such as smoothness, symmetry, etc. One important property of the light motion field is the local consistency. In particular, considering a foreground point (x, y) has a corresponding light motion vector (mx , my ), any of its neighboring point will have a similar motion vector as long as the distance between the neighboring point and (x, y) is small enough. In a discrete image, this means the light motion vectors are consistent in a local region. Moreover, following the work in [10], which gives an approximate relationship between the mapping center and the shape and refractive properties of the object, we can obtain a similar property for the light motion field, i.e. a light motion vector satisfies an approximate linear (logarithm) relationship to the object prop-

erties for solid (hollow) objects [10]. 2.2. Light Motion Extraction Now the question is how to extract the light motion field that reflects the refraction and reflection properties of the transparent object. We adopt the real-time environment matting approach proposed in [6] to extract the light motion field. In particular, by assuming that the transparent object itself is colorless, the environment matting model can be simplified as [6] C = ρT (c)

(2)

where c is the mapping center on the background, and ρ and T (·) are reflectance coefficient and background texture respectively. In our implementation, firstly we capture only two images, i.e. with and without transparent object in front of a special background [6]. Through background subtraction, the foreground region of the transparent object is identified. Assuming ρ is a value close to one, C and T (c) should be almost identical. Thus, for each foreground pixel, we search the closest background matching point. After identifying a pair of C and T (c) values, the corresponding ρ value is calculated according to Eq. (2). In this way, we establish the entire light motion field and the corresponding ρ values associated with each light motion vector. 2.3. Problem Statement Ideally, with the established light motion field, we should be able to composite the transparent object into any new background image to generate the refraction and reflection effects. However, directly applying the pre-generated light motion vectors cannot achieve satisfactory performance. This is mainly because the obtain light motion vectors contain quite significant amount of noise, which is resulted from the assumptions and approximation made in the process of light motion extraction. Fig. 1(a) shows the results of directly reusing the pregenerated light motion vectors to composite the transparent object, the cup, into a new background. In the figure, some additional marks are added to show the light motion vectors, where the red points belonging to the transparent object have corresponding mapping centers in blue. Comparing with the ground truth (the captured image), we can clearly see the artifact of noise, which can be considered as high frequency noise signals added to the original smooth matte data that make the result irregular. Some approaches have been proposed to combat the noise in real-time environmental matting. For example, in [6], Chuang et al. used anisotropic filter to remove the noise to obtain a visually acceptable result. According to partial differential equation theory, anisotropic filter is a kind of heat equation that can remove noise after several iterations to make the result smooth. Another common filter to remove

noise is bilateral filter, which belongs to the same category, i.e. adaptive smoothing, as anisotropic filter [11]. Although these filters are effective in removing noise, they are not able to remove errors, which is another type of artifacts existing in environmental matting. Fig. 1(b) gives an example to illustrate errors using the same composition result as Fig. 1(a). Errors in the composition results can be regarded as significant or low-frequency noise signals, which are impossible to be removed through the aforementioned filters. To the best of our knowledge, how to remove errors in real-time environmental matting has not been mentioned in the previous studies. 3. PROPOSED METHOD TO REMOVE ARTIFACTS Here we propose a light motion field fitting algorithm to remove both noise and errors in the extracted matte data. In particular, considering the particular transparent object studied in this research, it is natural to believe that there exists certain consistence among the light motion vectors of all the points at the same row. Based on this idea, for each row, we formulate the artifact removal task into an energy minimization problem, i.e. min

N X

ωi kV˜i − Vi k2

(3)

i=1

where Vi = (mxi , myi ) is the pre-generated light motion vector for point (xi , yi ) by the approach given in Section 2.2, V˜i = (m ˜ xi , m ˜ yi ) is the vector fitting result that we want to compute, N is the number of light motion vectors for a particular row, and ωi is the weight for each vector. For each expected fitting vector in one row, its components m ˜ xi and m ˜ yi are modeled as polynomial functions of the horizontal position xi . Least square fitting algorithm is used to obtain the model parameters. Note that in order to obtain a good result, high order polynomial fitting functions are desired at the cost of computational complexity. To trade off between performance and complexity, we√empirically choose the order of fitting function to be max{ N /2, 3}, i.e. proportional to the number of vectors in one row. The weight ωi is crucial to the final result since it reflects the importance for different types of light motion vectors including regular vectors, noise vectors and error vectors. Obviously, we want to give highest weight, lower weight and lowest weight to regular vectors, noise vectors and error vectors, respectively. In this way, through the energy minimization, the corrupted light motion field can be pulled towards the regular one. In this paper, we define the weight as

shown in Fig. 1(b), error data changes its direction and length dramatically compared with the rest. Thus, based on Eq. (4), error vectors will be assigned very small weight values. In this way, the influence of error vectors can be almost neglected in the fitting. On the other hand, Eq. (4) gives approximately the same weight for regular vectors and noise vectors. This is not a problem since noise effects can be automatically reduced by the least square fitting algorithm. 4. EXPERIMENTAL RESULTS Fig. 1(d) - 1(f) show the results of compositing the transparent cup into a new background with different artifactremoving approaches. For the result of using anisotropic filter, severe artifacts can still be seen, e.g. the region near to the top of the cup. The result of using bilateral filter is even worse, which generates some black rectangles. On the contrary, our motion field fitting method achieves the most visually acceptable result. Fig. 2 gives the close view results of the lower part of the cup. Compared with the captured image, the anisotropic filter result still has artifacts in some region. Although the bilateral filter result is quite smooth, it is too smooth to maintain all the features and some noise can still be seen near the boundary of the cup. Our proposed approach generates the most similar image to the real captured one. In addition, the filter based methods require many iterations and need to adjust several coefficients to generate an acceptable result, while our method only needs to fit the initial data once to generate a good result. 5. CONCLUSION In this paper, we focused on real-time environmental matting of transparent objects. The concept of light motion field was first introduced to interpret environment matting. From the viewpoint of light motion field, the use of anisotropic filter or other filters in the existing real-time environmental matting algorithms is to smooth the light motion field and make synthesized results visually acceptable. By noticing that the filter-based approaches cannot handle possible errors in the extracted matte data, we designed a vector field fitting algorithm that can remove noise and rectify errors simultaneously. Experimental results demonstrated that compared with the existing real-time environmental matting algorithms, our proposed approach can generate more accurate composition results. 6. REFERENCES

ωi = exp(−βkVi − V¯ k2 )

(4)

where β is a constant and V¯ is the average light motion vector for one row. We find that β = 0.1 gives good results. As

[1] A. R. Smith and J. F. Blinn, “Blue screen matting,” in SIGGRAPH ’96, New York, NY, USA, 1996, pp. 259– 268, ACM.

[2] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A bayesian approach to digital matting,” CVPR, vol. 02, pp. 264, 2001. [3] J. Sun, J. Jia, C.-K. Tang, and H.-Y. Shum, “Poisson matting,” in SIGGRAPH ’04, New York, NY, USA, 2004, pp. 315–321, ACM.

(a) Noise

(b) Error

[4] J. Wang and M. F. Cohen, “An iterative optimization approach for unified image segmentation and matting,” ICCV 2005., vol. 2, pp. 936–943 Vol. 2, 17-21 Oct. 2005. [5] D. E. Zongker, D. M. Werner, B. Curless, and D. H. Salesin, “Environment matting and compositing,” in SIGGRAPH ’99, New York, NY, USA, 1999, pp. 205– 214, ACM Press.

(c) Captured image

(d) Anisotropic filter

[6] Y.-Y. Chuang, D. E. Zongker, J. Hindorff, B. Curless, D. H. Salesin, and R. Szeliski, “Environment matting extensions: towards higher accuracy and real-time capture,” in SIGGRAPH ’00, New York, NY, USA, 2000, pp. 121–130, ACM Press. [7] Y. Wexler, A. W. Fitzgibbon, and A. Zisserman, “Imagebased environment matting,” in EGRW ’02, Aire-laVille, Switzerland, Switzerland, 2002, pp. 279–290, Eurographics Association. [8] P. Peers and P. Dutr´e, “Wavelet environment matting,” in EGRW ’03, Aire-la-Ville, Switzerland, Switzerland, 2003, pp. 157–166, Eurographics Association.

(e) Bilateral filter

(f) Proposed vector fitting

Fig. 1. The image composition results.

[9] J. Zhu and Y.-H. Yang, “Frequency-based environment matting,” in PG ’04, Washington, DC, USA, 2004, pp. 402–410, IEEE Computer Society. [10] M. Zhao, S. Xiao, X. Yang, and L. Ma, “Variable refractive index in environment matte,” in Journal of Zhejiang University. 2006, vol. 7, no. 7, pp. 1160–1167, Zhejiang University Press. [11] D. Barash, “A fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 6, pp. 844–847, 2002.

(a) Captured image

(b) Anisotropic filter

(c) Bilateral filter

(d) Proposed vector fitting

Fig. 2. Close view of the image composition results.