AN IMPROVED METHOD FOR EXTRACTION OF INTRINSIC IMAGES FROM A SINGLE IMAGE WITH INTEGRATED MEASURES Yun-Chung Chung1, Shyang-Lih Chang2, Jung-Ming Wang1, Sei-Wang Chen1 Graduate Institute of Computer Science and Information Engineering1 Department of Electronic Engineering2 National Taiwan Normal University St. John's & St. Mary's Institute of Technology Taipei, Taiwan, Republic of China Taipei, Taiwan, Republic of China
[email protected] ABSTRACT Intrinsic images, including reflectance and illumination images, are desirable to many vision applications. An improved method for extracting intrinsic images from a single color image with integrated measures is presented. To start with, the input image convolves with a predefined set of derivative filters. The pixels of filtered images are then classified into reflectance-related or illumination-related using a criterion measure comprising three measures of filtered pixels calculated from the input image. The three measures are denoted as chromatic measure, blur measure, and intensity measure. Finally, the intrinsic images of the input image can be computed from the classification results of the filtered images. Both synthetic and real images have been utilized in our experiments. The results demonstrated that the proposed technique can effectively extract the intrinsic images from a single image. KEY WORDS Intrinsic images, reflectance, chromatic measure, blur measure, intensity measure
1. Introduction For many applications, such as illumination assessment [6], shadow removal [5], vehicle detection and traffic monitoring [4], and security surveillance [6], eliminating illumination changing interferences can greatly improve the effectiveness of the algorithms. In all the cases, intrinsic characteristics (reflectance and illumination information) are constructive for various objectives, i.e., intrinsic characteristics can provide useful information for removing shadows and illumination effects. However, the extraction of intrinsic images from input images is a nontrivial task. The intrinsic image model begins from an early work by Barrow and Tenenbaum [2], who modeled an input image I as the product of a reflectance R and an illumination L, i.e., I = R × L. To decompose both R and L from single I is an ill-posed problem because two unknowns (R, L) have to be derived from a single I. It has been assumed that the convolutions of images and derivative filters tend to be sparse, Weiss [7] solved partially the intrinsic problem from a series of various illumination images. Consider a fixed viewpoint scene with the constant reflectance R. Let Ii (i = 1,..., n) be a collection of images acquired from the scene with different illuminations, from which a set of n equations, Ii = R × Li (where i = 1,..., n), can be constructed. However, this set of 453-278
equations is still not adequate for solving the n+1 unknowns of R and Li. Additional constraints are required to be included; Weiss introduced the sparseness assumption and recovered R and Li successfully from n input images using the Maximum-Likelihood (ML) estimation framework. Since fixed viewpoint and multiple images were used by Weiss, the applicability of his method is surely restricted. Our previous work [9] using physical chromatic information had remarkable success in extracting intrinsic images from a single image. The previous technique relies on a set of chromatic characteristics defined by a photometric reflectance model. This set of chromatic characteristics is effective in classifying the pixels of filtered images as being reflectance-related or illumination-related. However, chromatic properties alone are somehow not enough to describe the lighting properties. In order to improve the classification results, two more measures, blur measure and intensity measure, are employed to design our new classifier. The blur measure provides important information about the scene illumination because edges of shadows trend to be blurred compared with object edges, thus it is employed to reflect the fact that shadow edges are often not as sharp as object edges. On the other hand, intensity measure can describe strengths of shadows. In this paper, an improved technique for extracting intrinsic images quickly from a single image using integrated measures is presented. Our basic assumptions include that the observed materials (objects) are thick, i.e., not transparent / translucent, and the effects of radial photometric distortions of perspective projection and the camera ϒ-nonlinearity and gain control effect are omitted and not considered. Except for the above basic assumptions, information including shape and position of the objects, shadows and light sources, the intensity of ambient illumination, camera geometry and parameters is not required. The proposed approach is described in Sec. 2. Major steps involved in the approach are detailed in Sec. 3 to 5. Both synthetic and real image results are shown in Sec. 6; followed by concluding remarks and future work in Sec. 7.
2. Improved Intrinsic Image Extraction Algorithm The flowchart of our proposed improved technique for intrinsic image extraction is depicted in Fig. 1. Let I denote the input color image, which is generally modeled as I = R × L, where R (reflectance) and L (illumination) are the intrinsic images of I to be extracted.
356
To begin, the input image I is transformed into the log domain, i.e., log I = log (R × L) = log R + log L. Let I’, R’ and L’ represent log I, log R, and log L, respectively, and I’ = R’ + L’. The transformed image I’ then convolves with a prescribed set of n derivative filters, fj (where j = 1, …, n), giving rise to a set of n filtered images I’j = I’ * fj. Combining I with each of the set of filtered images I’j, we calculate three classification criteria information, namely chromatic measure, blur measure, and intensity measure. Three maps, chromatic measure map MCj, blur measure map MBj, and intensity measure map MIj, are obtained respectively. Afterwards, these three maps, MCj, MBj, and MIj are fused together to gain the criterion map Mj. With the criterion information provided by Mj, most pixels in I’j can be classified into the classified filtered images I”j. However, some pixels may not yet be determined in I”j. For those pixel properties that can not be decided, a fast evidence flowing procedure is applied. The results are utilized to separate I’j into reflectance-related or ˆ and Lˆ ′j , according to I”j. illumination-related image, i.e., R′ j From the calculated sets of reflectance-related and ˆ , the reflectance illumination-related images, R′ ˆ and L′ j j image Rˆ ′ and the illumination image Lˆ′ of the transformed image I' are estimated by the pseudo-inverse transformation processes. Finally, Rˆ ′ and Lˆ′ are transformed back to the spatial domain to obtain the estimated Rˆ and Lˆ .
3. The Measures of Filtered Pixels In order to extract robustly the required information, three classification criteria are designed in this paper. They are chromatic measure, blur measure, and intensity measure. 3.1 Chromatic Measure In this section, physical color properties of an image from an RGB-camera are analyzed. According to the image signal model, the general case of camera received image luminance s (λ , xG ) is defined as: G G G (1) s (λ , x ) = I (λ , x ) R∞ (λ , x ) ,
G
where x is the location of any pixel, λ denotes the wavelength, I (λ , xG ) is the radiance and R∞ (λ , xG ) is the material surface reflectivity. The radiance I (λ , xG ) combines the amount of light power received per unit object surface area, which is a function combining all light sources with ambient lights. G According to Lamberts cosine law, radiance I (λ , x ) is defined: G G 1 G I (λ , x ) = ∫ I Ak (λ )dk + ∫ ψ ( x ) I (λ )( N ( x ) ⋅ L j )dj , (2) k j σ d 2j Dj where Lj is the unit vector of direction of light source j, G N ( x ) is the unit vector of object surface normal, and G G N ( x ) ⋅ L j denotes dot product of N ( x ) and Lj. IDj(λ) is the irradiancy of light source j, and IAk(λ) denotes the received radiance of ambient light k. Note that the object receiving radiant strength of IDj(λ) is inversely proportional to square of the distance dj between the object and light source j, and σ is the property parameter related to the atmosphere. The
shading factor ψ ( xG ) describes the possibility of transition between shadow and non-shadow area, and its range is G defined between 0 and 1, i.e., 0 ≤ ψ ( x ) ≤ 1 . While ψ ( xG ) = 1 , objects are illuminated by the light sources and non-shadows G in this area. If ψ ( x ) = 0 , there is no direct irradiance and it is totally un-illuminated and called umbra. When 0 < ψ ( xG ) < 1 , it is called penumbra and objects are partially illuminated. To abbreviate Eq. (2), we denote the light radiance reaching the object as i (λ ) = 1 I (λ ) , since it is almost Dj σ d 2j Dj impossible to distinguish ambient light sources. The total radiance of ambient light ∫ I Ak (λ )dk is denoted as iA(λ), k
and Eq. (2) is rewriten in discrete format as:
G G n G I (λ , x ) = iA (λ ) +ψ ( x )∑ iDj (λ )( N ( x ) ⋅ L j ) ,
(3)
j =1
in which n light sources are assumed in the scene. Substituting the radiance I (λ , xG ) into Eq. (1), and denoting G G iDj (λ )( N ( x ) ⋅ L) as i j (λ , x ) , we rewrite Eq. (1) as:
G G G G n G s (λ , x ) = iA (λ ) R∞ (λ , x ) + ψ ( x ) R∞ (λ , x )∑ i j (λ , x ) . (4) j =1
The strength of light source can be described by the photometric reflectance model which depicts the optical properties of mediums regarding the absorption and scattering of light. Chromatic characteristics can be defined by a photometric reflectance model [3] developed from the Kubelka-Munk theory. In addition, the reflectance of smooth surfaces can be described by Fresnel reflectance property which depends exclusively on the refractive and incident angles, i.e., the light radiance is depicted as: G n G R∞ (λ , x )∑ i j (λ , x ) = . (5) j =1 G 2 G n G G n G (1 − ρ f ( x )) R∞ (λ , x )∑ e j (λ , x ) + ρ f ( x )∑ e j (λ , x ) j =1
j =1
where e j (λ , xG ) is the illumination spectrum of light source j and ρ f ( xG ) is the Fresnel reflectance. In reality, it is almost impossible to estimate ambient light condition. However, because the ambient light term G i A (λ ) R∞ (λ , x ) is generally very low in intensity compared with light sources, the wavelength λ is approximated to be G constant. In addition, for a nearby area of x , the object surface roughly has the same property around the location G x , thus the ambient light term is assumed to be constant of a G nearby area of x , denoted as ξ. Substituting Eq. (5) into Eq. G (4), we can depict the image luminance s(λ , x ) as: G G s (λ , x ) = ξ + ψ ( x ) ⋅ . (6) ⎧ G 2 G n G G n G ⎫ − + ρ x R λ x e λ x ρ x e λ x (1 ( )) ( , ) ( , ) ( ) ( , ) ⎨ ⎬ ∑ j ∑ j f ∞ f ⎩
j =1
j =1
⎭
Since there are many factors influencing the imaging condition, such as equal or arbitrary energy illumination, matte or dull surfaces, and uniformly stained or generally colored objects, different chromatic characteristics can be
357
G
G
The above equation shows that Cλm also depends only on the object reflectivity parameter R∞(λ, x). In addition, if the object surface is planar (i.e., a Mondrian world); the intensity components of the source are assumed to have constant location, Eq. (10) becomes:
derived. Let x denote x and hereafter the arrow above x is dropped whenever there is no confusion. First, under the condition of equal energy illumination, in which the spectral components of the source have constant wavelength, Eq. (6) then becomes n
n
s (λ , x) = ξ + ψ ( x) ⎡⎣ (1 − ρ f ( x)) 2 R∞ (λ , x) + ρ f ( x ) ⎤⎦ ∑ i j ( x) , (7)
s (λ , x ) = ξ + ψ ( x )∑ i j R∞ (λ , x) .
j =1
(14)
j =1
where ij(x) is the spatial component of the intensity variation The object reflectance property, Wλm, is specified by originating from the object geometry of light source j. sm (15) Wλ m = λ x . Furthermore, if an object has its reflectance property, Hλm, s ( λ , x ) independent of surface orientation, viewpoint, illumination Specifically, m∈ℵ, and direction and density, the reflectance property of the object can be described by ∂ m ∂s(λ , x) n ∂ m ⎡ψ ( x)∂R∞ (λ , x) R∞ (λ , x)∂ψ ( x) ⎤ . sλ m x = = ∑ij m ⎢ + m m −1 ⎧ ⎥ ∂λ ∂x ∂λ ⎣ ∂x ∂x ∂ ⎪ −1 ⎛ sλ ⎞ ⎫⎪ , ⎦ j =1 (8) H λ m = m −1 ⎨ tan ⎜ ⎟⎬ From Eq. (14), ∂λ ⎩⎪ ⎝ sλλ ⎠ ⎭⎪ n ∂ m ⎡ψ ( x)∂R∞ (λ , x) R∞ (λ , x)∂ψ ( x) ⎤ where m∈ℵ, ij m ⎢ + ∑ ⎥⎦ . n ∂λ ⎣ ∂x ∂x (16) j =1 ∂s (λ , x) 2 ∂R∞ (λ , x ) , and Wλ m = sλ = = ψ ( x)∑ i j ( x )(1 − ρ f ( x)) n ∂λ ∂λ j =1 ξ + ψ ( x ) i R (λ , x )
sλλ
2 n ∂ 2 s (λ , x ) 2 ∂ R∞ (λ , x ) . = = ψ ( x ) i ( x )(1 − ρ ( x )) ∑ j f ∂λ 2 ∂λ 2 j =1 Therefore,
Hλm
⎧ ⎛ ∂R∞ (λ , x) ⎞ ⎫ ⎟ ⎪⎪ ∂ m −1 ⎪⎪ −1 ⎜ = m −1 ⎨ tan ⎜ 2 ∂λ ⎟⎬. λ R ( , x ) ∂ ∂λ ⎪ ⎜ ⎟⎪ ∞ ⎜ ⎟ ⎪⎩ ∂λ 2 ⎝ ⎠ ⎪⎭
(9)
The above equation states that Hλm depends only on the object reflectivity parameter R∞(λ, x). Moreover, if the constraint of matte and dull object surfaces is also included, and Fresnel reflectance property is very small, i.e., let ρf(x) ≈ 0, Eq. (7) can be further simplified as n
s (λ , x) = ξ + ψ ( x )∑ i j ( x)R∞ (λ , x) .
(10)
This type of object reflectance property, denoted by Cλ , becomes m
sλ m
,
s (λ , x ) for m∈ℵ, and n ∂ m R∞ (λ , x) , thus ∂ m s (λ , x ) sλ = = ψ ( x ) i ( x ) ∑ j ∂λ m ∂λ m j =1
(11)
m
n
Cλ m =
ψ ( x )∑ i j ( x ) j =1
∂ m R∞ (λ , x) . ∂λ m
(12)
n
ξ +ψ ( x)∑ i j ( x)R∞ (λ , x) j =1
If light source is assumed to be much stronger than n ambient light, i ( x) ξ , the ambient light constant ξ is
∑ j =1
j
relatively very small, i.e., let ξ ≈ 0, Eq. (12) becomes ∂ m R∞ (λ , x) ∂λ m . (13) Cλ ≈ R∞ (λ , x) m
j
j =1
∞
If the light source is a point light source, then partially illuminated penumbra areas almost disappear. The shading factor ψ(x) can be reduced to be a binary function. When it is illuminated by the light sources and non-shadows, i.e., let ψ(x) = 1, and by shadow areas, i.e., let ψ(x) = 0. Equation (16) can be modified as: n ∂ m ⎡ ∂R (λ , x) ⎤ ij m ⎢ ∞ ∑ ⎥⎦ , for non-shadow areas, and ∂λ ⎣ ∂x j =1 Wλ m =
n
ξ + ∑ i j R∞ (λ , x) j =1
Wλ m = 0 , for shadow areas.
(17)
For non-shadow areas, omit ambient light as previously assumed, i.e., n i ξ , let ξ ≈ 0,
∑ j =1
j =1
Cλ m =
∑
j
∂ m ⎡ ∂R∞ (λ , x) ⎤ ⎥⎦ ∂λ m ⎢⎣ ∂x . (18) Wλ m ≈ R∞ (λ , x) From Eq.(17) and Eq.(18), for non-shadow areas, Wλm is a geometry-dependent intensity term and is viewed as an edge detector in some spectral distribution, else Wλm = 0. Next, when only one major light source is considered, i.e., let n = 1, then the radiance Eq. (6) is modified as: G G s (λ , x ) = ξ + ψ ( x ) ⋅ , (19) {e(λ , xG)(1 − ρ f ( xG))2 R∞ (λ , xG) + e(λ , xG ) ρ f ( xG )}
under the conditions of colored non-uniform illumination and objects with matte and dull surfaces, the illumination e(λ, x) can be decomposed into a spectral component e(λ) and a spatial component i(x). Fresnel reflectance property is very small, i.e., ρf(x) ≈ 0 for matte and dull surfaces. Equation (6) can be rewritten as: (20) s (λ , x ) = ξ + ψ ( x )e(λ )i ( x ) R∞ (λ , x ) , where e(λ) represents the illumination color. In this case, if an object has its reflectance property independent of surface
358
orientation, viewpoint, illumination direction, density and color, the change Nλm in object reflectance is ∂ m −1 ∂ ⎧ sλ ⎫ , (21) N λ m = m −1 ⎨ ⎬ ∂λ ∂x ⎩ s (λ , x) ⎭ where m∈ℵ, ⎧ ∂R (λ , x) ⎫ ⎫ ∂e(λ ) ⎧ + e(λ ) ∞ ψ ( x)i ( x) ⎨ R∞ (λ , x) ⎬ ⎪ . (22) ∂ m −1 ∂ ⎪⎪ ∂ ∂λ λ ⎩ ⎭⎪ N λ = m −1 ⎨ ⎬ ∂λ ∂x ⎪ ξ + ψ ( x)e(λ )i ( x) R∞ (λ , x) ⎪ m
⎪⎩
⎪⎭
Omitting ambient light as previously assumed, i.e.,
e(λ )i ( x) ξ , let ξ ≈ 0, Eq. (22) can be rewritten as ⎧ ∂R∞ (λ , x) ⎫ ⎪. ∂ m −1 ∂ ⎪ ∂λ N λ = m −1 ⎨ ⎬ ∂λ ∂x ⎪ R∞ (λ , x) ⎪
2
(23)
m
⎩ ⎭ Equation (23) demonstrates that Nλm also depends only on the object reflectivity parameter R∞(λ, x). Furthermore, if an object with uniformly colored planar surface is considered, i.e., its reflectance properties, R∞(λ) and ρf, are spatially constant, Eq. (6) is rewritten as s (λ , x) = ξ + ψ ( x)e(λ , x) (1 − ρ f ) 2 R∞ (λ ) + ρ f . (24)
{
}
The object spectral reflectance, Uλ, is formulated as ∂ ⎧ s ⎫ s s (λ , x) − sλ sx . (25) Uλ = ⎨ λ ⎬ = λx 2 s (λ , x ) ∂x ⎩ s (λ , x) ⎭ After substituting Eq. (24) into Eq. (25), ∂R (λ ) ⎫ ∂e(λ , x) ⎧ ψ ( x) (1 − ρ f ) 2 R∞ (λ ) + ρ f + ψ ( x)e(λ , x)(1 − ρ f ) 2 ∞ ∂ ⎪ λ ∂ ∂λ ⎪ Uλ = ⎨ ⎬ ∂x ⎪ ξ + ψ ( x)e(λ , x) (1 − ρ f ) 2 R∞ (λ ) + ρ f ⎪ ⎩ ⎭
{
}
{
}
.
(26) Omitting ambient light as previously assumed, i.e., e(λ , x) ξ , let ξ ≈ 0, Eq. (26) can be rewriten as ⎧ ∂e(λ , x ) ⎫ ∂ ⎪ ∂λ ⎪ . Uλ = ⎨ ⎬ ∂x ⎪ e(λ , x) ⎪ ⎩ ⎭
(27)
Equation (27) shows that for non-shadow areas, Uλ also depends only on illumination parameter e(λ, x). 3.2 Blur Measure Our recently proposed non-parametric blur measure [8] is employed. The blur measure mB(p′) for an edge point p′ is obtained by computing the weighted average of the standard deviation σ and the edge magnitude ∇I ( p′) as
mB ( p′) = η β
∇I ( p′) , σ ( p′) + (1 − η β ) σ max ∇I ( p′) max
where B is the number of pixels in image I(x, y) and C is the number of color bands (C = 3 as RGB images used). Ii(x, y) is the ith color band of image I(x, y), and RMSRi is th the i color component of the output of the Multiscale Retinex (MSR) method [1]. The MSR results reveal the quantity of enhancement required for edge contrast. The ith component of the MSR is given by 1 N RMSRi ( x, y) = ∑ [ log( I i ( x, y)) − log( Fn ( x, y )* Ii ( x, y ))] ,(30) N n =1 where the symbol * indicates convolution. Fn(x, y) is a Gaussian smoothing filter and is calculated as Fn ( x, y ) = Ke− ( x + y ) / c . The parameter cn is the scale constant for the nth scale, and K is a normalizing constant such that Fn(x,y) sums to 1. Values of cn are determined from the experimental result. Jobson et al. [1] recommended that N = 3, and from our previous study [8], we verified that this value gave good results. 3.3 Intensity Measure The intensity of object pixels will be obviously lower than the original if they are covered by shadows. The idea of comparing of the intensity values across both sides of an edge point is utilized to design the intensity measure. Although a dark object may have the same properties as shadows in this aspect, this intensity measure serves to eliminate possible errors caused by colors with similar intensity values of object boundaries. In addition, for a shadow edge point, the difference in intensity values of both edge sides reveals the strength of the shadow. For image I(x, y), x and y are the row and column coordinates, respectively. At any edge pixel location p(x, y); select two points (p1(x1, y1), p2(x2, y2)) along the direction of the gradient from the different sides of p(x, y), the intensity measure of p(x, y) is defined as
(28)
where σmax and ∇I ( p′) are normalization terms denoting max the maximum values for all standard deviations and for all edge gradient magnitudes. Additionally, the weight ηβ is related to the global image contrast information, and it is defined as RMSRi ( x, y ) 1 , (29) ηβ = ∑ ∑ BC ( x , y ) i log I i ( x, y )
2
mI ( x, y ) =
2 n
p1 − p2 p1 + p2
(31)
where p1, p2 are gray values, and 0 ≤ mI(x, y) ≤ 1. Since the computation time of this intensity measure is very fast, those pixels which have low difference of intensity values of both sides can be directly classified in advance as non-shadow edge pixel (reflectance edge) without calculating the chromatic and blur measures, i.e., in the classification step, let (32) p ( x, y ) ∈ Rˆ ′j , if mI (x, y) ≤ If , where Rˆ ′j is the reflectance-related image, and If is a threshold which can be determined by experiments. 3.4 Measures Fusion Three measure maps, chromatic measure map MCj, blur measure map MBj, and intensity measure map MIj, are calculated from each filter image, I’j. Afterwards, these three maps, MCj, MBj, and MIj are fused together giving the final criterion map Mj. There are many possible ways to fuse the measure maps. However, since the characteristics of the three measures represent different properties of the image; we design the fusion formulation according to their properties to obtain the best results.
359
Given ms(x, y) ∈ Msj, s ∈ {C, B, I}, the criterion value m(x, y) ∈ Mj is calculated from the following equation:
m( x, y ) = mI ( x, y ) [κ mc ( x, y ) + (1 − κ )mB ( x, y ) ]
, (33) where κ is a weight value which can be determined by experiments, and the control term mI (x, y) = 0, if mI (x, y) ≤ If . In addition, in order to prevent the control term mI (x, y) from dominating Eq. (33), let mI (x, y) = 1, when mI (x, y) > Id; both Id and If are thresholds which are not critical and can be determined by experiments. 4. Classification of Filtered Image Pixels According to the information obtained from the criterion map Mj, the pixels of filtered images are classified into reflectance-related or illumination-related. Let I’j denote a filtered image. For each pixel p ∈ I’j, there is a criterion value m(x, y) associated with p which contains the values of classification characteristics computed from the input image. Define Op as a window centered at p. To begin with, search for a line passing through p within the window Op. The line is formed from selecting the pixels whose sum of filtered values is maximum. Next, this line will divide the window Op into two separate regions, say O1 and O2. For every pixel p1i ∈ O1 and p2k ∈ O2, the correlation cik of their classification vectors is calculated. Define cmax = max{cik } , and cmin = min{cik } , and according to i ,k
i,k
the results of cmax and cmin, the decision value rp for pixel p is defined as follows: if cmax > tmax and cmin > tmin, rp = true, rp = false, if cmax < tmax and cmin < tmin, (34) rp = unknown, otherwise, where tmax and tmin are thresholds. With the criterion information provided by Mj, most pixels in I’j can be classified into the classified filtered images I”j. A reflectance-related image Rˆ ' j is composed of the pixels {p∈I’j | rp = true} collected, while an illuminationrelated filtered image Lˆ ' j is composed of the pixels {p∈I’j | rp = false} collected. However, some pixels may not be determined in I”j when rp = unknown. For those pixel properties that can not be determined, the fast evidence flowing procedure is applied. Pixels whose rp’s are unknown are quickly classified as follows. Suppose that pixel q has not yet been classified. Among its neighbors that have been classified, search the one with the largest filtered value and regard q belonging to the same class of that neighbor. The results are utilized to separate I’j into reflectance-related or illumination-related image, i.e., Rˆ ′j and Lˆ′j , according to I”j.
5. Estimation of Intrinsic Characteristics ˆ for each filtered image I’j ˆ and L′ Having determined R′ j (j=1,…,n), we can estimate the reflectance image Rˆ and illumination image Lˆ . According to Weiss [7], n
n
j =1
j =1
Rˆ ' = g *(∑ f jr * Rˆ ' j ) , and Lˆ ' = g *(∑ f jr * Lˆ ' j ) ,
(35)
where fjr is the reversed filter function of fj, defined as fjr(x, y) = fj(-x, -y), and x, y are pixel coordinates. The symbol * is a discrete convolution operation, and g denotes a
normalization function, which is the solution to the following equation (δ is the Kronecker delta function): n (36) g *( f r * f ) = δ
∑
j
j
. Let F (F–1) denote the (inverse) Fourier transform, from the convolution theorem, j =1
n
F ( Rˆ ') =
F (∑ f jr * Rˆ j ) j =1 n
F (∑ f jr * f j ) j =1
n
, Rˆ ' = F −1 (
F (∑ f jr * Rˆ j ) j =1 n
F (∑ f jr * f j )
)
.
(37)
j =1
Then Rˆ = exp( Rˆ '), and Lˆ can be found by Lˆ = I / Rˆ .
6. Experimental Results In our experiments, each input RGB image is first transformed into the log domain. The transformed image is filtered by two given derivative filters. The pixels of the filtered images are then classified using the criterion map Mj which is combined with MCj, MBj, and MIj. Figure 2 shows synthetic example images with blur shadows. Figure 2(a) and (e) contain a pseudo blue disk object with three blur shadows of different size, shape, and intensity. Shadows in Fig. 2(e) are intentionally painted more blur than those in Fig. 2(a). Figure 2(b) and (f) show the edge magnitude maps of Fig. 2(a) and (e), respectively, in which blue color indicates the vertical edge magnitude, and green color is the horizontal edge magnitude. In the decomposed reflectance image 2(c, g), all shadows have been successfully eliminated. In the estimated illumination image 2(d, h), the shadows are clearly shown. For comparison with our previous work, Table 1 shows the Mean Square Error (MSE) of the estimated reflectance images and the ground truth - original synthetic object image, Fig. 2(i). In Table 1, the first row shows improved results obtained by the new method with integrated measures and the second row shows the results of the previous method with only chromatic measure. Note that the new method is rarely affected by blur shadows, and the previous one increases its error as the shadows get more blur. For real image experiments, Fig. 3 demonstrates an outdoor example. The input image Fig. 3(a) contain many shadows cast by objects due to sunshine. Figure 3(b) shows the edge magnitude map of the input image, in which blue / green color indicates the vertical / horizontal edge magnitude. In the decomposed reflectance image Fig. 3(c), both shadows and sunshine have been successfully eliminated. Note that some faint object shapes can still be seen in the illumination image Fig. 3(d). Some researchers, such as Matsushita et al. [6], claimed in their work that objects should be totally invisible in illumination images. However, we do not fully agree to this idea because the illumination close to object surfaces is certainly influenced by their own colors. The appearance of faint object shapes in the illumination images can be acceptable in most situations. To demonstrate the improvement of our previous work, Fig. 3(e) and (f) show the results of reflectance and illumination images of Fig. 3(a) from our previous work [9], respectively. Comparing the figures reveal that the details of the recovered reflectance image (Fig. 3(c)) extracted by the
360
new method is obviously better than the previous one (Fig. 3(e)). For example, the utility pole in the upper left corner is better recovered in Fig. 3(c) than in Fig. 3(e). 7. Concluding Remarks and Future Work An improved integrated measures approach using the Weiss’ ML framework for extracting intrinsic images from a single image was presented. The proposed technique relies on a criterion measure composing three measures of filtered pixels calculated from the input image. The three measures are chromatic measure, blur measure, and intensity measure. This set of criterion measures has played an important role in classifying effectively the pixels of filtered images as being reflectance-related or illumination-related. Both synthetic and real images have been utilized in our experiments, and the results reveal the effectiveness of the proposed technique. For different image conditions, the selection of the set of chromatic measures merits more detailed exploration. Adaptive selection of chromatic measures to fit various images can also be studied. In addition, more investigations into the various meanings of measures and the methods using combined measures will be necessary in order to improve the classification results of filtered image pixels.
Lˆ ' j
Rˆ ' j n
Rˆ ' = g *(∑ f jr * Rˆ ' j ) j =1
Rˆ = Exp( Rˆ ')
n
Lˆ ' = g *(∑ f jr * Lˆ ' j ) j =1
Lˆ = Exp( Lˆ ') Lˆ
Rˆ
Figure 1. Flowchart of the proposed method
References: [1] D.J. Jobson, Z. Rahman, and G.A. Woodell, “A multiscale retinex for bridging the gap between color images and the human observation of scenes,” IEEE Transactions on Image Processing, Vol.6 , Issue:7, July 1997, pp. 965-976. [2] H.G. Barrow, and J.M. Tenenbaum, “Recovering intrinsic scene characteristics from images”, Computer Vision Systems, Academic Press, pp. 3–26, 1978. [3] J.M. Geusebroek, R. van den Boomgaard, A.W.M. Smeulders, and H. Geerts, “Color invariance”, IEEE Trans. on PAMI, Vol. 23, Issue. 12, pp.1338-1350, 2001. [4] J.M. Wang, Y.C. Chung, S.C. Lin, S.L. Chang, S. Chern, and S. W. Chen, “Vision-Based Traffic Measurement System,” IEEE 17th International Conference on Pattern Recognition, Cambridge, United Kingdom, Vol. 4, pp. 360-363, Aug 2004. [5] J.M. Wang, Y.C. Chung, S.L. Chang, and S. W. Chen, “Shadow Detection and Removal for Traffic images,” IEEE International Conf. on NSC, Taipei, Taiwan, Mar 2004. [6] Y. Matsushita, K. Nishino, K. Ikeuchi, and M. Sakauchi, “Illumination normalization with time-dependent intrinsic images for video surveillance”, IEEE C.S. Conf. on Computer Vision and Pattern Recognition, Vol. 1, pp.I-3-10, 2003. [7] Y. Weiss, “Deriving intrinsic images from image sequences”, IEEE Intel. Conf. on Computer Vision, Vol. 2, pp.68-75, 2001. [8] Y.C. Chung, J.M. Wang, R.R. Bailey, S.W. Chen, and S.L. Chang, "A Non-Parametric Blur Measure Based on Edge Analysis for Image Processing Applications," 2004 IEEE Conf. on CIS, Singapore, Dec 2004, in press. [9] Y.C. Chung, J.M. Wang, R.R. Bailey, S.W. Chen, S.L. Chang, and S. Cherng, “Physics-based Extraction of Intrinsic Images from a Single Image,” IEEE 17th ICPR, Cambridge, United Kingdom, Vol. 4, pp. 693-696, Aug 2004.
(a)
(b)
(c)
(d)
(e)
(f)
(g) (h) (i) Figure 2. Synthetic examples – blur shadows, (a, e) input I, (b, f) edge magnitude map of I, (c, g) reflectance Rˆ , (d, h) illumination Lˆ , (i) original synthetic object image.
(a)
(b)
(c)
Table1. MSE of integrated and chromatic measures MSE
Integrated Measures Chromatic Measure
non-blur shadows
blur shadows
more blur shadows
0.01209
0.01207
0.01208
0.01238
0.01251
0.01303
(d) (e) (f) Figure 3. An outdoor example, (a) input I, (b) edge map, (c) reflectance Rˆ , (d) illumination Lˆ , (e) previous Rˆ , (f) previous Lˆ .
361
