Lie Methods in Color Signal Processing: Illumination ... - IEEE Xplore

Lie Methods in Color Signal Processing: Illumination Effects Reiner Lenz, Martin Solli Dept. of Science and Technology, Link¨oping University SE-60174 Norrk¨oping, Sweden [email protected], [email protected]

Abstract In this paper we describe illumination changes with the help of elements in the Lorentz group SU(1,1). We show how Lie-theoretical methods can be applied to solve problems related to illumination changes. We derive partial differential equations that describe the changes in the space of color signals. We show how these changes effect the induced variations in the space of RGB vectors. We illustrate the application of these methods with two examples: In the first example we derive a simple linear equation system that links the pointwise pixel changes to the parameters of the illumination change. In the second example we construct operators in the RGB space that either compensate illumination changes or predict the effects of illumination changes.

1. Introduction and Overview Understanding the processes involved in the formation of color images is of fundamental importance for many applications. As examples we mention: • Color imaging itself where signal processing algorithms form an essential part of all digital cameras [1] • Investigation of biological vision systems where topics like natural scene statistics have gained a lot of interest recently [12] • The investigation of invariants has a long history in computer vision and pattern recognition and color constancy is one example [3, 2, 14, 5] In this paper we will use a model based on the theory of Lorentz groups to derive linear operators that describe the changes of the RGB-vectors as a function of illumination changes. We will mainly derive the theoretical framework to construct these operators and we will illustrate the results with some examples. A comparison with similar methods is out of the scope of this paper.

The 18th International Conference on Pattern Recognition (ICPR'06) 0-7695-2521-0/06 $20.00 © 2006

In the following Section 2 we will introduce the notations and the basic model used in the following. In Section 3 we describe the differential equations related to illumination changes and in Section 4 the RGB-operator. A simulation illustrating the results is given in Section 5.

2. The Model We will use a multi-spectral model of color where the key components of the model are functions of the wavelength variable λ located in some interval. In the theoretical derivations we will often use functions of λ and in the numerical model we use the interval from 410nm to 710nm sampled in 10nm intervals (since we use the images described in [12]). We denote the functions describing the Illumination Source, the properties of the Objects Points and the Sensor by L(λ), R(x, λ) and S(λ) respectively. We think of L(λ) as the number of photons of wavelength λ emitted during a certain period of time, R(x, λ) as the probability of a photon of wavelength λ to be re-emitted from point x and S(λ) is the output signal generated by the sensor after interacting with a photon of wavelength λ. Here we only consider cameras with three sensors with sensitivity functions S k , (k = 1, 2, 3) and the measured pixel values at point x form therefore a three-dimensional vector P (x). After sampling we obtain the following vector-matrix form of the model: P (x) = CLR(x)

(1)

where L is the diagonal matrix with the vector elements of L along its diagonal. From the definitions we have L(λ) ≥ 0, S(λ) ≥ 0, 0 ≤ R(x, λ) ≤ 1. We will consider spectrally varying illumination sources and analyze the relation between illumination and pixel changes. The basic idea is the following: operators M (in the space of illumination spectra) map illumination spectra L to modified illumination spectra: M : L → M L = LM . The adaptation operators A (in pixel-space) map pixel vectors to other pixel vectors: A : P (x) → A (P (x)) and we will

in the following investigate the relation between the spectral operators M and RGB-operators A. In a typical example the camera C and the statistics of the reflectance spectra R(x, λ) are given and for a given operator M we want to find an operator A = AM such that the original pixels are similar to compensated pixels in some sense: P (x) = CLR(x) ≈ AM CLM R(x)

(2)

The operator AM stabilizes the sensor output and we can think of it as a color constancy operator. We start by characterizing the operators M operating on the space of illumination spectra. The illumination spectra are non-negative functions and they are therefore located in a cone-like function space. It can be shown that the elements in this space can be approximately described by coordinates ν, ξ, η where ν ∈ R and ζ = ξ + iη is an element of the unit disk. They are obtained by the following construction: Choose a basis B (L) with first three basis vec(L) (L) (L) tors b0 , b1 , b2 so that the first coefficient in this basis is greater than zero for all non-black spectra. In the following we will often suppress superscripts like (L) if it is clear from the context. This gives L ≈ v0 b0 + v1 b1 + v2 b2 ≈ eν (b0 + ξb1 + ηb2 )

(3)

with v0 = eν , ξ = v1 /v0 , η = v2 /v0 . We investigate operators in the spectral space that are of the form (ρ, M ) with:  (e , ζ) → ν

ρ+ν

e

aζ + b , cζ + d

 = (eρ v0 , M ζ)

(4)

a b

The matrix M = c d is in the group SU(1,1) and has the exponential representation: t(α1 J 1 +α2 J 2 +α3 J 3 )

Mt = e  J1 =

i 0

0 −i



 ; J2 =

0 1

1 0

with

(5)





 ; J3 =

0 i −i 0

For fixed parameters (α1 , α2 , α3 ) the points ζ(t) = M t ζ0  , t ∈ R define a curve through ζ0 and the matrices M t , t ∈ R define a one-parameter subgroup of SU(1,1). The parameters (αk ) describe the shape and t the position on the curve. Applying the elements of a one-parameter subgroup to a fixed point ζ0 describes the chromatic changes of the illumination and investigations of different light sources show that a large class of illumination sources can be modeled in this framework. Examples are black-body radiators (where the group parameter t is closely related to the inverse temperature of the radiator) and sequences of daylight spectra. For matrices M t and a point ζ0 we obtain

The 18th International Conference on Pattern Recognition (ICPR'06) 0-7695-2521-0/06 $20.00 © 2006

curves ζ(t) = ξ(t) + iη(t) = M t ζ0 ) and we compute the derivatives: dξ(t) |t=0 = −2η0 α1 + α2 − ξ02 α2 − 2ξ0 α3 η0 + η02 α2 dt dη(t) |t=0 = 2α1 ξ0 − 2ξ0 η0 α2 − η02 α3 + ξ02 α3 + α3 (6) dt More background information on these models can be found in [7, 6, 8, 9, 10, 11].

3. Description of Pixel Changes We will now describe the relation between the parameters of the illumination operator and the corresponding pixel vector changes. We consider first the problem of estimating the intensity change from a pair of images. We have pixel vectors P (x) = (p1 (x), p2 (x), p3 (x)) with pk (x) ≥ 0 and intensity changes given by L → eρ L. Therefore P (x) → eρ P (x) and for the intensity σ(x) = p1 (x)+p2 (x)+p3 (x) we get σ(x) → eρ σ(x). For non-black points we can pass to logarithms which gives ln(σ(x)) and ρ+ln (σ(x)) . The unknown intensity change parameter ρ can be estimated as a global shift. Now consider the problem of estimating the intensity change from a continuous sequence of images. We assume an exponential model with constants γ > 0 and β such that the intensity change at time t is given by γeρt . Denote the intensity at point x and time t as σ (x, t). Then we get γ = 1 from the condition at time t = 0 and by differentiating at t = 0: ∂eρt σ (x, 0) ∂σ (x, t) |t=0 = |t=0 = ρσ (x, 0) ∂t ∂t We can estimate the illumination change from the instant change ∂σ(x,t) |t=0 and the current value σ (x, 0). ∂t We now consider the SU(1,1) part of the illumination (L) (L) (L) change. We have L ≈ b0 + ξb1 + ηb2 and illumination changes can be described as L → LM ≈ (L) (L) (L) b0 + ξM b1 + ηM b2 . In the space of reflection spectra we introduce the basis B (R) and approximate the reflection spectra by R ≈ B (R) ρ. For the pixel vectors we find: P = CLR ≈

2 

(vk Gk ) ρ = h0 + ξh1 + ηh2

(7)

k=0

  (L) with Gk = Cdiag bk B (R) . We will now use this relation to compute a linear approximation of the pixel changes in terms of the curve parameters for the illumination change. The complete derivation of the illumination change is summarized in the following steps:

1. Compute Gk from the camera and bases B (R) , B (L) . 2. Multiply with reflection spectrum coordinate vector ρ to get vectors hk = Gk ρ 3. Use Eqs. (6) to approximate the ζ curve ξM ≈ ξ0 − 2η0 α1 + α2 − ξ02 α2 + η02 α2 − 2ξ0 η0 α3 ηM ≈ η0 + 2ξ0 α1 − 2ξ0 η0 α2 − η02 α3 + ξ02 α3 + α3

4. Insert into Eq. (7): P M ≈ h0 + ξ0 h1 + η0 h2 + c1 α1 + c2 α2 + c3 α3 where:

c1 = −2η0 h1 + 2ξ0 h2 c2 = (1 − ξ02 + 2ξ0 η02 )h1 − 2ξ0 η0 h2 c3 = −2ξ0 η0 h1 + (1 + ξ02 − η02 )h2

5. Collect the ck in the matrix C = (c1 , c2 , c3 ) and the αk in the vector α to obtain P M ≈ h0 + ξ0 h1 + η0 h2 + Cα 6. For the pixel difference this gives the linear approximation: (8) P M − P 0 ≈ Cα The matrix C is singular since every point ζ on the unit disc is a fixed point of a whole subgroup of SU(1,1). Algebraically we can find this subgroup by solving the equadη(t) tions 0 = dξ(t) dt |t=0 = dt |t=0 for αk (given the values of ξ0 , η0 ) : α1 , α2 = α3 = 0 ( if η0 = ξ0 = 0) (1 + η02 ) α1 = α2 , α3 = 0 ( if ξ0 = 0) 2η0 (1 + η02 + ξ02 ) η0 α3 ; α2 = − α3 ( if ξ0 = 0) (9) α1 = − 2ξ0 ξ0 In the experiments we compute a particular solution of the equation (8) by QR-decomposition C = QU where Q is orthogonal and U an upper triangular matrix. This gives Q−1 (P M − P 0 ) ≈ U α from which we can compute a solution of the form (α1 , α2 , 0). Applying Eqs. (9) gives all solutions.

4. Compensation of Pixel Changes In the last section we derived a linear relation between the illumination parameters and the change of the measured pixel vectors. We will now use this and construct a transformation in the pixel-space that stabilizes the resulting image sequence.

The 18th International Conference on Pattern Recognition (ICPR'06) 0-7695-2521-0/06 $20.00 © 2006

2 We can interpret P ≈ k=0 (vk Gk ) ρ from Eq. (7) as an operator, parameterized by the coordinates of the illumination spectrum and operating on the vector ρ. Dynamical 2illumination changes result in an operator series P t = k=0 (vk (t)Gk ) and we will now construct an operator sequence At such that P 0 − At P t is minimal. We restrict us to exponential operators At = etB , repeat the derivation leading to steps 3 and 4 in the previous section and approximate the operator P t by P 0 + tD : P t ≈ G0 + ξ0 G1 + η0 G2 + t (D 1 α1 + D 2 α2 + D 3 α3 ) where the matrices D k are computed as the same combinations as the vectors ck in step 4. If we use the exponential form of At , and compute the first order approximation of At P t we get: d (At P t ) |t=0 = BP 0 + D dt

(10)

We require that the average norm of the change-vectors is minimal:   ! 2 Q(B) = E (BP 0 + D) ρ = min (11)   Q(B) = tr (BP 0 + D) Σ (BP 0 + D) = tr (BP 0 ΣP 0 B 



+ BP 0 ΣD +

DΣP 0 B 

(12) + DΣD  )

where ρ are the coefficient vectors of the reflection spectra with correlation matrix Σ. . Computing derivatives (see appendix in [4]) we obtain B as solution of BP 0 ΣP 0 + DΣP 0 = 0

(13)

We can also include intensity changes and consider more

tB with general operators of the form At = γeβt Ae

scalars γ, β and a constant matrix A and derive a similar solution as in (13). This is the form we used in the following experiment.

5. An Illustration We illustrate the results obtained with two examples. We use the operators At as described above, i.e. as stabilizers of the operators P t and we use At to predict the effect of the operators P t operating on a given pixel vector. This is not an evaluation of the method but simply an illustration. We show results based on the following datasets and parameters: the multispectral image consists of 31 bands, we use the estimated spectral characteristics of a Canon 10D camera ([12],[15]). The basis for the illuminants is computed by PCA from Planck spectra in the range 3000K to 14000K using mired (inverse temperature) sampling. The basis in the space of reflection spectra is computed from the

Munsell reflectance spectra (see [13]) using PCA. The matrix B and constant β are computed from the image pair corresponding to 5562K, 5856K. From these we compute operators At corresponding to equal increments in the miredscale. We use 30 images in the mired-scale from 3000K to 14000K (5562K and 5856K correspond to number 18 and 19 in this sequence). We then use At to predict the color image generated with the corresponding illumination change. The images in Figure 1 show the results obtained. The images in the upper row are computed from the true Planck spectra. In the lower row we start from the image in the middle and apply the operators At to predict the images at temperatures 3000K and 14000K. In the Figure 2 we see the same images in the top row, but now we start from the images at 3000K and 14000K and compensate them with the operators At so that they correspond to the start image at 5562K.

Figure 1. Prediction of color changes.

Figure 2. Compensation of color changes. Our experiments showed good prediction and compensation properties for a number of hyperspectral images and databases such as the Munsell color atlas. Another observation we made was that the matrices etB were clearly dominated by the coefficients in the diagonals, a property known as von-Kries adaptation. A typical example is the matrix for the transition A3K : 5562K → 3000K given by ⎛ ⎞ 0.69 0.18 −0.11 0.27 ⎠ A3K = ⎝ −0.03 1.10 0.05 −0.22 2.09

The 18th International Conference on Pattern Recognition (ICPR'06) 0-7695-2521-0/06 $20.00 © 2006

6. Conclusions We showed that Lie-theory provides a framework to investigate problems related to illumination changes. We studied two problems: estimation of illumination changes from sequences of measured RGB images and compensation/prediction of RGB-changes related to illumination changes. We also mention that the same methodology is not only applicable for the study of illumination changes. It can also be used in the study of other types of non-negative signals such as probability distributions.

References [1] IEEE-Signal Processing Magazine - Issue on Color Image Processing, January 2005. [2] G. Finlayson, M. Drew, and B. Funt. Spectral sharpening: sensor transformations for improved color constancy. J. Opt. Soc. Am. A, 11(5):1553–63, May 1994. [3] D. Forsyth. A novel algorithm for color constancy. Int. J. Comp. Vision, 5(1):5–36, Aug. 1990. [4] K. Fukunaga. Introduction to Statistical Pattern Recognition. Academic Press, Boston, 1990. [5] J.-M. Geusebroek, R. van den Boomgaard, A. W. Smeulders, and T. Gevers. Color constancy from physical principles. Pattern Recognition Letters, 24(11):1653–1662, 2003. [6] R. Lenz. Estimation of illumination characteristics. IEEE Trans. IP, 10(7):1031–1038, July 2001. [7] R. Lenz. Two stage principal component analysis of color. IEEE Trans. IP, 11(6):630–635, June 2002. [8] R. Lenz. Spectral Color Spaces: Their Structure and Transformations. In: Advances in imaging and electron physics, volume 138, pages 1–67. Elsevier, 2005. [9] R. Lenz and T. H. Bui. Recognition of non-negative patterns. In Proc 17th Int. Conf. Pattern Recognition, volume III, pages 498–501, 2004. Cambridge, UK. [10] R. Lenz and T. H. Bui. Statistical properties of color signal spaces. J. Opt. Soc. Am. A, 22(5):820–827, 2005. [11] R. Lenz, T. H. Bui, and J. Hern´andez-Andr´es. Group theoretical structure of spectral spaces. J. Math. Im. and Vision, 23:297–313, 2005. [12] S. M. C. Nascimento, F. P. Ferreira, and D. H. Foster. Statistics of spatial cone-excitation ratios in natural scenes. J. Opt. Soc. Am. A, 19(8):1484–1490, 2002. [13] J. Parkkinen, J. Hallikainen, and T. Jaaskelainen. Characteristic spectra of Munsell colors. J. Opt. Soc. Am. A, 6(2):318– 22, Feb. 1989. [14] D. Slater and G. Healey. The illumination-invariant matching of deterministic local structure in color images. IEEE Trans. PAMI, 19(10):1146–51, Oct. 1997. [15] M. Solli, M. Andersson, R. Lenz, and B. Kruse. Color measurements with a consumer digital camera using spectral estimation techniques. Proc. SCIA 2005, Vol. 3540 Springer LNCS, pages 105–114. Springer Verlag, 2005.