Recognizing Color Patterns Irrespective of Viewpoint and ... - CiteSeerX

Recognizing Color Patterns Irrespective of Viewpoint and Illumination F. Mindru, T. Moons and L. Van Gool

Katholieke Universiteit Leuven, ESAT / PSI, 3000 Leuven, Belgium

Abstract

ments beyond the point where they remain stable in order to create sucient discriminant power for distinguishing between dierent patterns. These problems are remedied in this paper by introducing powers of the intensities in the individual color bands and combinations thereof in the expressions for the moments. The resulting generalized color moments implicitly characterize the shape, the intensity and the color distribution of the pattern in a uniform manner; and, a broad set of moment invariants can be extracted that never call upon high powers of either intensities or spatial coordinates. Section 2 gives a classi cation of all moment invariants for ane transformations of the pattern and linear photometric changes. The discriminant power and classi cation performance of these invariants for color pattern recognition is tested in section 3 on a data set of images of real, outdoors advertising panels taken from dierent viewing angles and under dierent weather conditions. Physically dierent panels representing the same pattern and partially occluded ones are included as well. The performance of these new invariants is also compared on the same data set to other moment invariants presented in literature.

New invariant features are presented that can be used for the recognition of planar color patterns such as labels, logos, signs, pictograms, etc., irrespective of the viewpoint or the illumination conditions, and without the need for error prone contour extraction. The new features are based on moments of powers of the intensities in the individual color bands and combinations thereof. These moments implicitly characterize the shape, the intensity and the color distribution of the pattern in a uniform manner. The paper gives a classi cation of all functions of such moments which are invariant under both ane deformations of the pattern (thus achieving viewpoint invariance) as well as linear changes of the intensity values of the color bands (hence, coping with changes in the irradiance pattern due to dierent lighting conditions and/or viewpoints). The discriminant power and classi cation performance of the new invariants for color pattern recognition is tested on a data set of images of real, outdoors advertising panels. A comparison to moment invariants presented in literature is included as well.

1 Introduction

2 Moment invariants for color images

This paper contributes to the viewpoint and illumination independent recognition of planar color patterns such as labels, logos, signs, pictograms, etc.. Much research has been put into invariants for planar shapes under geometrical deformations (see [12, 13] for an overview). Most of this work has focused on invariants for the shapes' contours. For the patterns considered here, the pictorial content usually is too complicated to robustly extract object contours from it. Color information has proven very useful in pattern recognition as well. Much attention has been paid to the illumination independent characterization of the color distribution of the pattern [3, 5, 6, 7, 17, 18]. Color histograms, however, do not exploit the spatial distribution of the colors within the pattern. Another strand of research has focussed on moment invariants for greyvalue intensity patterns under dierent types of geometric and / or photometric changes [1, 4, 8, 10, 15, 16, 19]. A limitation of this approach is that one may have to let grow the order of the mo-

2.1 Generalized color moments

A color pattern can mathematically be represented as a vector-valued function I de ned on a region in the (image) plane and assigning to each image point (x; y) 2 the 3-vector I (x; y) = ( R(x; y) ; G(x; y) ; B (x; y) ) containing the RGB-values of the corresponding pixel. We de ne the generalized color moments Mpqabc by

Mpqabc =

ZZ



xp yq [R(x; y)]a [G(x; y)]b [B (x; y)]c dxdy :

Mpqabc is said to be a (generalized color) moment of order p + q and degree a + b + c. Observe that generalized color moments Mpq000 of degree 0 in fact are the (p; q)-shape moments of the image region ; and, that the generalized color moments of degree 1, viz. Mpq100 , Mpq010 , Mpq001 , are just the (p; q)-intensity moments of respectively the R-, G- and B-color band. On the

1

other hand, the generalized color moments M00abc of order 0 are the non-central (a; b; c)-moments of the (multivariate) color distribution of the RGB-values of the pattern. Hence, these generalized color moments generalize shape moments of planar shapes, intensity moments of greylevel images, and non-central moments of the color distribution in the image. A large number of generalized color moments can be generated with only small values for the order and the degree. In this paper, only generalized color moments up to the rst order and the second degree are considered. In the sequel they will be called \moments" for short.

strategy typically is what has happened in the literature. The photometric oset can e.g. be eliminated through the use of intensity minus average intensity and the photometric scale parameters can be eliminated by normalizing the resulting intensity's variance [16]. After these normalizations one has to deal with ane deformations exclusively. The resulting ane invariants may look simpler than the ones given here, but the inherent complexity is at least comparable. In this paper, only a normalization against photometric oset is performed. The reason being that it is a low complexity operation which reduces the photometric transformation to only scaling and yields invariants with a high degree of symmetry in the order and the degree of the moments involved. As these invariants have the 0th order 1st degree moments M00100 , M00010 , M00001 in their denominators, the normalization is performed by taking the absolute value of the intensities obtained after subtraction of the average intensity in each color band, thus keeping the 0th order 1st degree moments positive.

2.2 Geometric and photometric changes

The geometrical deformations of the pattern caused by changes in viewpoint are modelled by ane transformations :

 x   a a  x   b  + 1 = 11 12 0

b2 y a21 a22 with jAj = a11 a22 ? a12 a21 6= 0. This implies that the y

0

camera is assumed to be relatively far from the object. This assumption together with the fact that the object is planar can greatly simplify the analysis of the photometric changes. Typically, light sources are far from the objects as well. The geometry of light re ection is the same for all points in that case, i.e. they share the same angles of light incidence and camera viewing direction. Also for the more sophisticated models of diuse re ection the change in camera or light position will in that case result in an overall scaling of the intensity [14]. Furthermore, an oset allows to better model the combined eect of diuse and specular re ection [20] and has been found to give better performance [16]. Thus, the photometric changes to be considered comprise scaling combined with an oset for each color band: i.e. K (x; y) = sK K (x; y) + uK with K 2 f R; G; B g and 0 6= sK ; uK 2 IR. The combined eect of the ane and photometric transformations of the moments Mpqabc is given by

2.3 Classi cation of the color invariants

As only moments up to the 1st order and the 2nd degree are considered here, the resulting invariants are functions of the generalized color moments M00abc, M10abc and M01abc with (a; b; c) 2 f (0; 0; 0) ; (1; 0; 0) ; (0; 1; 0) ; (0; 0; 1) ; (2; 0; 0) ; (0; 2; 0); (0; 0; 2) ; (1; 1; 0) ; (1; 0; 1) ; (0; 1; 1) g. By normalization against oset, the photometric changes only involve scaling of intensities in the dierent color bands. The combined eect of ane and photometric transformations on these moments thus is:

0? abc
1 0a a b 10M abc1 M 10
? B@ M01abc CA = saR sbG scB jAj @a1121 a1222 b12A@M1001abcA : ?
abc 0

0

M00abc

0

0

0

0

1

M00

As can be seen from the way they transform, the 0thorder moments M00abc can be considered in isolation; and a similar remark holds for individual color bands and for pairs of color bands. Dierent powers of the intensity of a single color band can be taken in isolation as well. Hence, the invariants can be classi ed according to 3 degrees of freedom: the order, the degree and the number of color bands of the moments involved. To allow maximal exibility for the user in the choice of the color bands and to ensure the highest possible robustness of the invariants in the classi cation, the following 2 criteria were used to construct a basis for the moment invariants: (1) Keep the number of color bands as low as possible; and (2) include as much as possible low-order invariants. This results in the following classi cation, which is obtained by the techniques explained in [11].

?M abc
= ZZ [ a x + a y + b ]p [ a x + a y + b ]q 11 12 1 21 22 2 pq 0



[ sR R(x; y) + uR ]a [ sG G(x; y) + uG ]b [ sB B (x; y) + uB ]c abs(jAj) dxdy which, after expansion, yields a linear combination of moments of order  p + q and degree  a + b + c. The actions of the ane and photometric changes on the moments come out to commute. Hence, one might rst normalize against one type of transformation and then against the other. Alternatively, one may normalize against one and switch to the use of invariants for the other. To some extent, the latter

2

Theorem 1

(1-band invariants) All geometric / photometric invariants involving generalized color moments up to the 1st order and 2nd degree in only 1 color band are functions of the following 2 basis invariants: 2 0 0th order, 2nd degree : B02 = M(M00001M)002 1st order, 2nd degree : 2 0 1 1 2 0 0 1 2 10 M01 M00 +M10 M01 M00 B12 = M10 M01 M00 +M 1 M0 M002 M00 00

0th order, 2nd degree : (R) , B (G) , B (B ) , C (RG) , C (RB ) , C (GB ) ; B02 02 02 02 02 02 (GB ) ; ( RB ) ( RG ) 1st order, 1st degree : C11 , C11 , C11 1st order, 2nd degree: (R) , B (G) , B (B ) , C 1(RG) , C 1(RB ) , C 1(GB ) , B12 12 12 12 12 12 3( 3( RG ) 2( GB ) 2( RB ) 2( RG ) C12 , C12 , C12 , C12 , C12GB) , C124(RB) i(KL) are the 2-band invariants dewhere Bij(K ) and Cpq ned in Theorem 2 and evaluated in the color bands K and L. In particular, every invariant involving all 3 color bands is a function of invariants which involve only 2 of the 3 bands.

Theorem 2

3 Recognition performance of the new invariants

? M10 M01 M00 +MM00102 MM00011 MM00000 +M10 M01 M00 where Mpqi stands for either Mpqi00 , Mpq0i0 or Mpq00i . 2

1

0

1

0

2

0

2

1

(2-band invariants) All geometric / photometric invariants involving generalized color moments up to the 1st order and 2nd degree in 2 of the 3 color bands are functions of the following 10 basis invariants: 11 M 00 1 , B 2 , C02 = M00 00 0th order, 2nd degree : B02 10 M 01 ; 02 M00 00 1st order, 1st degree : 10 01 00 M 01 M 00 M 10 +M 00 M 10 M 01 10 01 00 10 01 00 C11 = M10 M01 M00 +M 10 01 00 00 M00 M00

The usefulness of these new invariants for viewpoint and illumination independent recognition of planar color patterns has been tested in a number of experiments on data sets of real images. The tests involved both synthetic transformations of real images as well as real changes in viewpoint and illumination. Recognition of a pattern is performed by means of a nearest neighbour classi cation scheme based on feature vectors consisting of a number of moment invariants. Since our image data set is medium size, the recognition performance of the system is estimated by means of a jackkni ng procedure [9].

? M10 M01 M00 +MM100010 MM010001 MM000000 +M10 M01 M00 ; 1 , B2 , 1st order, 2nd degree : B12 12 11 M 00 M 10 +M 10 M 11 M 00 +M 00 M 10 M 11 M C121 = 10 01 00 M001011 M000110 M000000 10 01 00 11 10 00 10 00 11 00 11 10 ? M10 M01 M00 +MM100011 MM010010 MM000000 +M10 M01 M00 ; 11 00 01 M 01 M 11 M 00 +M 00 M 01 M 11 10 01 00 10 01 00 C122 = M10 M01 M00 +M 11 01 00 00 M00 M00 11 01 00 01 00 11 00 11 01 ? M10 M01 M00 +MM100011 MM010001 MM000000 +M10 M01 M00 ; 02 00 10 M 10 M 02 M 00 +M 00 M 10 M 02 10 01 00 10 01 00 C123 = M10 M01 M00 +M 02 10 00 00 M00 M00 02 10 00 10 00 02 00 02 10 ? M10 M01 M00 +MM100002 MM010010 MM000000 +M10 M01 M00 ; 10

00

01

01

10

00

00

01

10

3.1 Discriminant power under ideal model conditions

A rst test aimed at assessing the discriminant power of the new invariants under ideal model conditions. To this end, parameterized geometric and photometric transformations were applied to a set of 15 real color images, some of which are shown in Figure 1. The images were selected on the basis of their color / pattern characteristics (e.g. predominance of a certain color band, high intensities but dierent spatial distribution in all 3 bands, highly correlated bands, ne grained patterns, etc.). The geometric transformations applied to the images are combinations of scalings in the x- and y-direction ranging from 0:7 to 1 with 0:1 step size, skewing in the x-direction (i.e. x = x + s y) ranging from 0 to 0:3 with 0:1 step size, and rotations ranging from 0o to 45o with 15o intervals. The photometric transformations of the images comprise scalings of the intensity values ranging from 0:65 to 1 with 0:05 step size and osets ranging from 0 to 30 with 5 unit intervals, both applied independently on each of the color bands. For these transformed images, the average recognition performance obtained by a single invariant is between 85:2% and 92% correct

M10 M01 M00 +M10 M01 M00 C124 = M10 M01 M00 +M 02 10 00 00 M00 M00 20 M 00 M 01 +M 01 M 20 M 00 +M 00 M 01 M 20 M ? 10 01 00 M100020 M010001 M000000 10 01 00 ; i is the 1-band invariant Bpq de ned in Theowhere Bpq rem 1 and evaluated in the ith (of the 2) color band(s), and where Mpqij stands for either Mpqij0 , Mpqi0j or Mpq0ij , 20

01

00

01

00

20

00

20

01

0

depending on which of the 2 color bands are used.

Theorem 3

(3-band invariants) All geometric / photometric invariants involving generalized color moments up to the 1st order and 2nd degree in all 3 color bands are functions of the following 21 basis invariants:

3

Figure 1: Examples of color images used for the discriminant power assessment.

Figure 2: Examples of color images in the data set of advertising panels. assignments. When using feature vectors containing 2 A = f B12(R) ; B12(G) ; B12(B) g ; invariants 100% correct recognition rates are achieved, but not all couples of invariants yield total discrimiB = f C111(RB) ; C111(RG) ; C111(GB) g ; nation between the patterns. In particular, feature C = f C~02(RB) ; C~02(RG) ; C~02(GB) g ; vectors involving all 3 color bands and a 2nd degree D = f C121(GB) ; C122(GB) ; C123(GB) g ; invariant(s) yield clear and well-separated clusters. E = f C~02(GB) ; C~02(RG) ; C111(GB) g ; F = f C~02(GB) ; C~02(RG) ; C111(GB) ; C123(GB) g ; 3.2 Experiments with real data G = f C123(GB) ; C124(RB) ; C~02(GB) ; C~02(RG) ; C02(RB) g : The rows in Table 1 give for each pattern the numTo test the discriminant power of the new invariants ber (in % ) of corresponding images in the data set under real world conditions, a series of experiments that is correctly recognized by the given feature vecwas done on a data set containing color images of diftor. The average performance of each vector over all ferent outdoors advertising panels. The images were the 11 dierent patterns is indicated in the last coltaken from various viewpoints at dierent times of the umn. The outcome of the experiments is discussed in day and under dierent weather conditions. Physimore detail in the subsequent sections. cally dierent panels representing the same pattern were included as well. Figure 2 shows some examples of such images. For the tests, 11 classes of dif3.2.1 Recognition performance on real data ferent patterns were used, each containing 10 to 18 First, the recognition rates of the 2-band invariants images. The panels were manually cut out from the in Theorem 2 were investigated by considering the images. Theoretically, 21 (basis) invariants are availfeature vectors formed by applying every invariant to able for a complete distinction between the patterns each (combination of the) color band(s) (cf. A, B20 and (Theorem 3). However, due to the limited amount 02 1 B2 B 02 ~ C in Table 1). The invariant C02 = (C02 )022 = M(M000011M)002 of samples and classes, statistically signi cant concluwas also included in the test because of its simple, sions can only be drawn for feature vectors composed symmetrical form. The individual performance of of 5 invariants at most [2]. Therefore, more than 100 the dierent triplets is quite similar and their averfeature vectors combining 3 to 5 invariants were seage recognition rate ranges from 63% to 69% corlected and tested. The feature vectors are selected rect matches. From the 0th order 2nd degree (i.e. according to the types of the constituting invariants; histogram-based) invariants, C~02 showed the best perand, the best performing combinations of invariants formance (viz. 65:4%). Using 1st order moments betare selected by a Principal Component Analysis. As ter recognition rates can be obtained, although the inan example, Table 1 shows the recognition rates of 6 crease in performance is not signi cant. Furthermore, of these feature vectors which are representative for it was observed that all the invariants perform better the dierent types of feature vectors used in the test: 4

A B C D E F G

1 40 33 73 46 93 93 93

2 3 4 5 6 7 8 9 10 11 41 70 66 33 78 66 77 50 100 75 33 50 88 93 42 44 100 50 100 100 100 30 100 60 7 100 11 38 100 100 42 60 83 53 93 89 78 75 100 87 100 80 78 100 100 78 89 100 100 62 100 90 83 87 100 89 100 100 100 87 100 100 100 100 93 100 89 87 100 100

mean 64 66 65 73 89 93 96

Table 1: Recognition rates (in % ) of dienent types of feature vectors on the data set of advertising panels. on patterns with higher variations in the intensities. Dark patterns or patterns with low variations in the intensities in each bands tend to be misclassi ed by invariants only involving 1st degrees in the intensities (especially by C11 ). The 2nd degree moment invariants clearly cope better with patterns that have a low variation in their intensity content. Taking combinations of 3 dierent invariants but applying it to only 1 pair of color bands (cf. D in Table 1) improves the recognition rate with approximately 8%. This proves that the invariants capture dierent characteristics of a color pattern. Combinations of 3 dierent invariants applied to dierent combinations of the color bands (cf. E in Table 1) clearly yield the best results: Their average recognition rates range from 78:4% to 89:1%. One reason for having a better performance when using all 3 bands is that these feature vectors cope better with correlations between the bands. Therefore, the best way of capturing the information contained in all 3 bands in an uniform manner is to use \symmetric" groups of invariants, which are evaluated on the 3 bands and combining dierent types (in order and degree) of invariants. Finally, increasing the length of the feature vectors improves the recognition rate with another 4:5% (cf. F and G in Table 1). The best recognition performance for the given data set with a feature vector of length less than 6 is obtained by the invariant vector G in Table 1. Its average recognition rate is 96:3%.

images of the advertising panels was taken in which the panels are partially occluded (to dierent degrees) by passing cars or trac signs. Non-occluded panels having a strong (partial) shadow cast on them or panels that are illuminated by strong light are included as well. And, last but not least, the test set also contains images with a signi cant amount of perspective deformations or blurring. Figure 3 shows a few examples of these images. The \distorted" images were all

Figure 3: Examples of partially occluded and geometrically and photometrically distorted images. classi ed by the feature vector G which was found to give the best average recognition performance on the training data set (cf. section 3.2.1). Depending on the amount of occlusion or distortion, each image was either correctly classi ed or the correct class was found to be the second best match. A determining factor here is the color distribution of the occluding object and its location in the image. If the color histogram of the occluding object is similar to that of the pattern or if the variations in the intensities in the 3 color bands of the occluding object is relatively small, then the invariants can cope with fair amounts of partial occlusions. But, if the color histogram of the occluding object is complementary to that of the pattern, then even a relatively small amount of occlusion may lead to misclassi cation. Furthermore, the new invariants are evenly sensitive to perspective distortions as their greylevel counterparts. This is not surprising, of course, since only invariance to ane transformations was considered. On the other hand, it turns out that the new invariants can cope with relatively

3.2.2 Sensitivity to occlusions and severe geometric and photometric distortions

It is well known that intensity moments are quite sensitive to partial occlusions of the pattern. Color histograms, on the other hand, are rather robust for partial occlusion, at least if the occlusion is not too severe. The new moment invariants presented above combine both approaches. Thus, it is important to investigate how they are aected by occlusions and by geometric and photometric distortions violating the assumptions underlying the invariants. To this end, a collection of 5

high degrees of photometric distortions (strong partial shadowing, high intensity lighting, etc.) and with fair amounts of image blur.

[2] R. O. Duda and P. E. Hart, Pattern classi cation and scene analysis, John Wiley, 1973. [3] G. Finlayson, B. Schiele and J. Crowley, Comprehensive colour image normalisation, Proc. ECCV '98, LNCS 1406, Springer, 1998, pp 475 { 490. [4] J. Flusser and T. Suk, Pattern recognition by ane moment invariants, Pattern Recognition, Vol. 26 (1993), pp. 167 { 174. [5] D. Forsyth, A novel algorithm for color constancy, Int. Journal of Computer Vision, Vol. 5 (1990), pp. 5 { 36. [6] B. Funt and G. Finlayson, Color constant color indexing, IEEE Trans. PAMI, Vol. 17 (1995), pp. 522 { 529. [7] T. Gevers, and A. W. M. Smeulders, A comparative study of several color models for color image invariant retrieval, Proc. Intern. Workshop on Image Database and Multimedia Search, 1996, pp. 17 { 23. [8] M. Hu, Visual pattern recognition by moment invariants, IRE Trans. Information Theory, Vol. 8 (1962), pp.179-187. [9] R. A. Johnson and D. W. Wichern, Applied multivariate statistical analysis, Prentice-Hall, 1992. [10] S. Maitra, Moment invariants, Proc. CVPR '79, IEEE Comp. Soc. Press, 1979, pp. 697 { 699. [11] T. Moons, E. Pauwels, L. Van Gool, and A. Oosterlinck, Foundations of semi-dierential invariants, Int. Journ. Computer Vision, Vol. 14 (1995), pp. 25 { 47. [12] J. L. Mundy, and A. Zisserman A (eds.), Geometric invariance in computer vision, MIT Press, 1992. [13] J. L. Mundy, A. Zisserman, and D. Forsyth (eds.). Applications of invariance in computer vision, LNCS 825, Springer, 1994. [14] M. Oren and S. Nayar, Seeing beyond Lambert's law, Proc. ECCV '94, LNCS 800, Springer, pp. 269 { 280. [15] R. Prokop and A. Reeves, A survey of moment-based techniques for unoccluded object representation and recognition, CVGIP: Models and Image Processing, Vol. 54 (1992), pp. 438 { 460. [16] T. Reiss, Recognizing planar objects using invariant image features, LNCS 676, Springer, 1993. [17] D. Slater and G. Healey, The illumination-invariant recognition of 3D objects using local color invariants, IEEE Trans. PAMI, Vol. 18 (1996), pp. 206 { 210. [18] M. Swain and D. Ballard, Color indexing, Int. Journal of Computer Vision, Vol. 7 (1991), pp. 11{32. [19] L. Van Gool, T. Moons, and D. Ungureanu, Ane / photometric invariants for planar intensity patterns, Proc. ECCV '96, LNCS 1065, Springer, 1996, pp. 642 { 651. [20] L. Wol, On the relative brightness of specular and diuse re ection, Proc. CVPR '94, IEEE Comp. Soc. Press, 1994, pp. 369 { 376.

3.3 Comparison to the literature

The recognition rates of the new invariants are also compared on the same data set of advertising panel images to similar moment invariants reported in the literature. In [16] functions of the central intensity moments up to the 4th order of greyvalue intensity patterns are presented, which are invariant under ane transformations of the image. Photometric changes are dealt with by normalization against both intensity scaling and oset. Among the 10 invariants used, J1 , 5 and 6 are reported in [16] to provide the best performance (notations as in [16]). When applied to (the greylevel equivalent of) the images in our data set, the feature vector f J1 ; 5; 6 g has an average recognition performance of 23:4% correct matches. Extending the feature vector with the 4th and 5th best invariants in [16], viz. 1 and 2 , increases the average recognition rate to 34:2%. This weak performance most probably is caused by a combination of noise in the images and the high order of the moments involved. When applied to the individual color bands of the images in the data set, the recognition performance of these invariants improves with another 20%. So, color greatly adds to the recognition process, but the overal performance remains signi cantly less than that obtained by the new invariants de ned above. An approach which comes closest to what is reported here is found in [19]. The ane / photometric invariants in [19] involve shape and intensity moments up to the 2nd order of greyvalue intensity patterns. Some of these invariants also require an ane invariant area subdivision of the pattern for their computations (which makes them computationally more demanding). Evaluating such an invariant on all 3 color bands yields recognition performances ranging from 84:3% to 89:3%. However, combining 3 to 5 of these 2nd order invariants does not signi cantly improve the recognition rate. Furthermore, only 7 of these invariants have shown good individual performance. In comparison to the new invariants presented above, the 21 invariants in Theorem 3 are computationally simpler and have acceptable individual performance rates. Moreover, optimal combinations of 3 to 5 of these new invariants show slightly better performances than those in [19]. Using all 21 new invariants is expected to make them much better suited for use with large data sets.

References

[1] A. Abo-Zaid, O. Hinton, and E. Horne, About moment normalization and complex moment descriptors, Proc. ICPR '88, IEEE Comp. Press, pp. 399 { 407.

6