STOCHASTIC APPROACH TO SEPARATE DIFFUSE AND SPECULAR REFLECTIONS Sang Hwa Lee∗ , Hyung Il Koo∗ , Nam Ik Cho∗ , and Jong-Il Park∗∗ ∗
School of Electrical Eng. and Computer Science, INMC, Seoul National Univ., Seoul, Korea. ∗∗ Division of Electrical and Computer Eng., Hanyang Univ., Seoul, Korea. Emails:
[email protected],
[email protected],
[email protected],
[email protected]
ABSTRACT This paper presents separation of specular and diffuse reflection components from an image pair. The proposed approach is based on the dichromatic reflectance model and Markov random field models. The proposed method estimates specular and diffuse components by minimizing observation color noise and prior potential of specular reflectance. The specular reflection component is modelled as an MRF and estimated in maximum a posteriori framework. This paper proposes likelihood term of specular component and the prior model of specular reflectance based on Phong’s shading. Some experiments show that the proposed approach separates the specular and diffuse reflection components effectively. The separated specular components can be utilized in image-based lighting which renders a scene with virtual lighting sources. Index Terms— color, reflection, Markov process, rendering 1. INTRODUCTION Most of natural objects have non-Lambertian surfaces. The colors on the same surface point look different with respect to the viewing directions since the amount of reflected light at the surface point becomes varied. Thus, when we obtain multiple images of the same scene in the different viewpoints and different lighting directions, the observed colors of the same point are different in each image. Especially, the surface looks much brighter when the viewing direction coincides with that of reflected light. This is called specular reflection. While the diffuse reflection usually has little variation in color from the different viewpoints, the specular reflection changes significantly with respect to the different viewing directions and the normal components of the 3D surfaces. The specular reflection makes computer vision problems more difficult because it eliminates the color consistency in the images of same scene. The color consistency is a main cue for correspondence estimation, image-based modelling, segmentation, and so on. Thus, when we deals with multiple images of same scene, we have to eliminate the specular reflection component from the images so that color consistency is well restored. This fact results in recent researches on separation of specular and diffuse reflection components and radiometric calibration [1, 3, 4, 5, 13, 14]. The performance of correspondence estimation can be increased by eliminating the specular reflectance and restoring the diffuse components in image pairs [1]. The segmentation of color region is performed on the only diffuse component [2]. And virtual illumination and lighting environment are modelled by analyzing the specular reflection in the images. Images with novel lighting conditions are generated by the modelled specular reflectance.
Most of the previous works related to separation of specular and diffuse reflections are based on the dichromatic reflectance model and color histogram of images. The dichromatic reflectance model states that the color space consists of two reflections, diffuse and specular components. In addition, the specular and diffuse components of dielectrics have different spectral distributions in color space. Thus, the specular components are detected and separated by analyzing the color histogram of images. The color histogram was analyzed in the whole image, image patch, or on the epipolar line of stereo images [1, 3, 6]. These works required multiple observations of a surface to obtain the histogram. Polarization was another good cue to separate the reflections [9, 10]. The previous works showed good separation results, but most of them required specific color information and many images to obtain color histograms. Color segmentation or manipulations to get a color consistency region was needed to find the histogram. And the polarization is not the general tool in image acquisition systems. This paper proposes a stochastic approach to separate the specular and diffuse reflections from an image pair. This paper is organized as follows. Section 2 introduces color observation model in the dichromatic reflectance. Section 3 describes the proposed stochastic models of specular reflection. Section 4 shows some experimental results and an example of imagebased lighting using the separated specular component. Finally, this paper is concluded in Section 5. 2. COLOR OBSERVATION MODEL The methods to separate specular and diffuse components are based on the dichromatic reflectance model. The dichromatic reflectance model means that the spectral distribution of specular reflections is different from that of diffuse ones. The spectral distribution of specular reflections is similar to that of lighting source. However, the spectral distribution of diffuse component is a product of the lighting source and surface pigments. This fact means that each RGB component of the diffuse reflection varies with the same rate such that the direction of the color vector does not change for the same lighting source [7, 8]. Finally, the colors of observations can be expressed as a linear combination of a vector for surface reflectance color (diffuse component) and a vector for lighting source color (specular component). Fig. 1 shows the dichromatic reflectance model in 3D color space. The color vector of specular component (Is ) is parallel with the lighting source (E). And, the color vector of diffuse component (Id ) is proportional to the pure diffuse reflection (ID ). Thus, all image pixels on a uniform-colored surface lie on a dichromatic plane
In Eq. (3), the first term is called likelihood model, which measures how well the specular field matches the image pair. And, the second term is the prior model to describe the known information of specular reflection.
E
B
3.2. Likelihood model The likelihood model is a potential function to measure how well the estimated specular field matches the image pair. It is an matching error given the image pair and specular components. Thus, the likelihood model is a kind of the potential function based on Eq. (1). The likelihood model is defined such that the diffuse reflection components of corresponding image pair are well matched,
G I
ID n
U (Il |S, Ir ) = ρi (Il − (βE + αIl )),
Is
R
Fig. 1. Color observation in dichromatic reflectance model. which can be spanned by the vectors of pure diffuse and lighting source components. In this paper, the dichromatic reflectance model for an observation color vector I is modelled as I = Id + Is + n = αID + βE + n,
(1)
where n is the noise of the observation. The coefficients α and β are the proportional constants of the pure diffuse component ID and the lighting source E, respectively. β determines the amount of specular reflection component of an observed color. 3. STOCHASTIC APPROACH The proposed approach in the paper is based on the maximum a posteriori (MAP) framework and Markov random field (MRF). The specular reflection is considered as an MRF and estimated in the MAP framework. We define the likelihood model between an image pair, and design the prior model to present the known information of specular reflection. Since the diffuse and specular components are dependent each other in the dichromatic plane, we only model and estimate the specular reflection from image pairs. 3.1. MAP framework Denote the specular reflection component vector S and the image pair Il , Ir . We consider the specular reflection as a continuous MRF. Given an image pair, we have to find the random field S which maximizes a conditional probability p(S|Il , Ir ). The probability is decomposed by Bayesian rule, p(S|Il , Ir ) =
p(Il |S, Ir ) · p(S|Ir ) , p(Il |Ir )
(2)
which defines the relation of specular field and images. And, we can transform the probability space into potential (or energy) space by the equivalence of MRF and GRF. Thus, maximization of probability function is changed into minimization of potential function, min U (S|Il , Ir ) ∝ min[U (Il |S, Ir ) + U (S|Ir )].
(3)
(4)
where the right image has specular component and the left image has only the pure diffuse one. The potential function ρi (·) is a robust functional to adapt non-linear and various noise distributions. In Eq. (4), the noise of image pair are compared after separating the specular reflection. Note that the diffuse component α is directly determined from the estimated specular reflectance since they are dependent each other to span a dichromatic plane. 3.3. Prior model The prior model exploits the known property of random field. It is a kind of constraint such that the estimated random field preserves the known property. The role of prior model is to regularize the estimated random field to preserve the property. The prior model in the MRFs defines the interaction between neighboring fields to preserve the prior information. The prior model of specular reflection is based on Phong’s shading. It says that the specular reflection is proportional to cosn θ, where θ is the angle between reflected light ray on the surface and viewing direction. We can express the Phong’s shading function by Taylor series, cosn θ = 1 −
n 2 n(3n − 2) 4 θ + θ + · · ·. 2 24
(5)
In general, since the specularity is observed within a small range of angles, we assume that the angle between reflected light ray and viewing direction (or camera optical axis) is likely to be small. Then, we can approximate the Phong’s shading model as a second-order polynomial of the angle. This approximation enables us to design the surfaces of specular field as quadratic curves. Furthermore, since the quadratic functions have constant second derivative, we can model the specular field to have a constant Laplacian value. We use only first-order MRF window which have four neighboring sites, (i, j − 1), (i, j + 1), (i + 1, j), and (i − 1, j) for a site (i, j) in 2D discrete lattice. The potential function of specular field is implemented to preserve the Laplacian of specular field, U (si,j |Ir )
=
ρs (L(sn ))
n∈N
=
ρs (si,j−2 − 2si,j−1 + si,j )
+
ρs (si,j+2 − 2si,j+1 + si,j )
+
ρs (si−2,j − 2si−1,j + si,j )
+
ρs (si+2,j − 2si+2,j + si,j ).
(6)
where L(si,j ) is the second-order derivative in 2D discrete lattice (Laplacian operation) at a site (i, j), and ρs (·) is a potential function of specular field. By minimizing the above potential, we preserve the estimated specular field to be quadratic curves. 3.4. Iterative minimization The specular field is estimated in the process of energy minimization as is usual in the MAP framework. The energy function to be minimized is the summation of likelihood model, Eq. (4) and prior model, Eq. (6). The energy minimization methods in MAP framework have been interesting subjects, but are out of scope in the paper. We adopt simulated annealing [11, 12] to minimize the potential function. By iterative minimization process in the manner of simulated annealing, the optimal specular field is estimated. For fast convergence, we initialize the specular field by the deterministic likelihood model.
Fig. 2. Separation of specular and diffuse components for a synthetic image pair. (left) Test image pair. (middle) diffuse component, and (right) specular component.
3.5. Initialization The initialization is based on the noise minimization in the dichromatic reflectance model as shown in Fig.1. It also considers the variation of diffuse component when the lighting condition changes. If we minimize the noise of Eq. (1) in the color space, we can estimate the diffuse and specular components from the color observations, min ||n||2 = min ||I − βE − αID ||2 . α, β
(7)
By differentiating the noise energy with respect to the parameters, α and β, and solving the system equations, we can estimate the diffuse and specular components of the observations, α=
|E|2 (I · ID ) − (ID · E)(I · E) , |E|2 |ID |2 − (ID · E)2
(8)
and β=
I · ID − α|ID |2 |ID |2 (I · E) − (ID · I)(ID · E) . = ID · E |E|2 |ID |2 − (ID · E)2
(9)
Note that the parameters, α for diffuse reflection and β for specular one, are dependent each other as shown in Eq. (9). The specular field is used as an initial states in the iterative process. This approach is a deterministic manner and different from the histogram-based methods [1, 3, 5]. Another main difference is that the proposed method deals with the variation of diffuse component under the different lighting conditions. The parameter α for the same surface pigment changes with respect to the lighting condition in the image pair. And we should note that the derived formulation to minimize the noise in color space is a kind of likelihood model of specular reflectance since the likelihood model is related to the noise of the observation. 4. EXPERIMENTS The experiments are performed using the image pairs illuminated by two lighting sources. An image pair is obtained under two lighting conditions such that at least one of two corresponding pixels in the image pair should have only the diffuse component. Our test images are generated with a camera and two lighting sources. A camera is fixed at the center and the lighting sources are located in left and
right viewing directions, respectively. And two light sources illuminate the scene alternatively to obtain the image pair. We assume that there is no color saturation which violates the color observation model in the dichromatic reflectance. This image acquisition makes the problem easier because the correspondences between image pair are well defined. When we use the stereoscopic images, then we also have to find the correspondences before separating the reflection components. Fig.2 and Fig.3 show the separation results by the proposed stochastic method. As shown in the figures, the specular and diffuse components are effectively estimated with respect to the lighting conditions. The specular fields are estimated as gray-level images since the illumination of test images is the white lighting source. The estimated specular field is smooth and continuous, which results from the prior model approximated to quadratic surface. As we can see in the figures, the diffuse reflectance is effectively estimated despite of the large variation of diffuse components due to different lighting conditions. In other words, the diffuse components with strong specularity have the brighter diffuse colors even though the specular components are eliminated. These results coincide with the observation model which stated that brightness of diffuse component is proportional to the illumination strength. According to the results in various test images, the proposed methods proved to be appropriate to separate the diffuse and specular reflectance components from an image pair. In the future, it is necessary to study the proposed framework with stereoscopic images where the correspondence estimation is also required. The specular reflection extracted from different lighting conditions is useful for image-based lighting. When the specular reflectance is separated, the image has only the diffuse component. We add the specular component separated from a region to the diffuse component in another image region. Then, the synthesized image seems to be captured with different lighting source. Fig.4 shows a re-illuminated images where a virtual lighting source is moving from left to right direction.
(a)
(b)
(d)
(c)
Fig. 4. Re-lighted image sequence. The virtual light source is moved from left to right in the sequence. Fig. 3. Separation of specular and diffuse components for a real image pair. (top) Test image pair. (middle) diffuse component, and (bottom) specular component.
[3] D. N. Bhat and S. K. Nayar, “Stereo in the presence of specular reflection,” ICCV-1995, pp. 1086-1092, 1995.
5. CONCLUSIONS
[5] R. Tan and K. Ikeuchi, “Separating reflection components of textured surfaces using a single image”, ICCV-03, 2003.
This paper has proposed a stochastic approach to separate diffuse and specular reflection components from an image pair. The specular component has been modelled as an MRF and has been estimated in MAP framework. In the likelihood model, the linear relation of specular and diffuse components under different lighting conditions has been analyzed in the dichromatic reflectance model. And the prior information of specular field has been modelled as quadratic surfaces which result from approximated Phong’s shading. According to the experiments, the specular and diffuse components have been successfully separated with respect to the lighting conditions. The separated specular component will be useful for image-based lighting with varying lighting conditions. Some future works consider multi-views condition and simultaneous stereo matching and separation of specular/diffuse reflections.
[6] S. W. Lee, and R. Bajcsy, “Detection of specularity using color and multiple views,” Image and Vision Computing, no. 10, pp. 643-653, 1992.
6. ACKNOWLEDGMENT
[4] S. Lin and S. W. Lee, “Estimation of diffuse and specular appearance,” ICCV-1999, 1999.
[7] S. Shafer, “Using color to separate refelctance components,” Color Research and Applications, 10, pp. 210-218, 1985. [8] S. Lin and S. W. Lee, “A representation of specular appearance,” ICCV-1999, 1999. [9] L.B. Wolff and T. E. Boult, “Constraining object features using a polarization reflectance model,” IEEE Trans. on PAMI, vol. 13, no. 7, pp.635-657, July 1991. [10] S. Nayar. X. Pang, and T. E. Boult, “Removal of specularities using color and polarization,” CVPR-1993, 1993. [11] J. Besag, “Spatial interaction and the statistical analysis of lattice system,” J. Royal Statis. Soc., Series B, no. 2, pp. 192-236, 1974.
This work was supported by Korea Research Foundation Grant funded by Korea Government(MOEHRD, Basic Research Promotion Fund) (KRF-2005-003-D00360).
[12] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images,” IEEE Trans. PAMI , vol. 6, no. 6, pp. 721-741, 1984.
7. REFERENCES
[13] S. P. Mallick, T. Zickler, P. N. Belhumeur, and D. J. Kriegman, “Specularity removal in mages and videos: A PDE approach,” Proc. of ECCV-2006, 2006.
[1] S. Lin, Y. Li, S. B. Kang, X. Tong, and H. Y. Shum “Diffuse-specular separation and depth recovery from image sequences,” ECCV-2002 , LNCS 2352, pp. 210-224, 2002. [2] I. Omer and M. Werman, “Color lines: Image specific color representation,” CVPR-2004, 2004.
[14] S. P. Mallick, T. Zickler, P. N. Belhumeur, and D. J. Kriegman, Beyond Lambert: Reconstructing specular surfaces using color,” Proc. of CVPR-2005, 2005.
