Scale Space Color Blob and Ridge Detection1 - Springer Link

REPRESENTATION, PROCESSING, ANALYSIS AND UNDERSTANDING OF IMAGES

ScaleSpace Color Blob and Ridge Detection1 N. A. Khanina, E. V. Semeikina, and D. V. Yurin Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Leninskie gory 1, build. 2, Moscow, 119991 Russia email: [email protected], [email protected], [email protected], http://imaging.cs.msu.ru Abstract—Feature detection in color images frequently consists in image conversion from color to grayscale and then one of grayscale detectors application. This approach has a few disadvantages: some features become indistinguishable in grayscale and features ordering based on response of grayscale detector do not accord with features order of importance from human’s perception point of view. There are two essential contribu tions in this paper. First, the method for direct detection of blobs and ridges in color images is proposed. Sec ond, for scalespace ridge detection we introduce a 3D non maxima suppression procedure (in two orthogo nal directions) which makes ridge detection simple and easy programmable in contrast to Lindeberg’s auto matic scale selection approach. The proposed algorithms also produce estimates of blobs sizes and ridges width. Keywords: color blob detection, color ridge detection, color line detection, ridge edge, feature points, scale space, non maxima suppression. DOI: 10.1134/S1054661812010221

1. INTRODUCTION Point and linear feature detection is a basic prob lem of image mosaicing, image registration, 3D recov ery, pattern recognition, and scene analysis. There exist a large number of corner [1–3], edge [4, 7], blob [5–6], and ridge [7] detectors for grayscale images. Feature detection in color images often consists of conversion from color to grayscale mode and one of grayscale detectors application. This approach has dis advantages described below. Most of feature detectors [1–6] consist of three steps. The first of them is application of some trans form to image in order to construct Feature Response Image (FRI). Typical examples of FRI are gradient absolute value (for Canny edge detection [4]), differ ence of Gaussians (grayscale blob detection [5–6]), Harris functional (corner detection [3]). The second step is extrema detection (or non maxima suppression [4]) in this image followed by the third step—thresh olding or hysteresis [4]: sufficiently large extrema are considered to be features. Using of thresholds gives rise to the first argument for color image analysis with out conversion to grayscale: a possibility of feature dis tinguishability reduction. Equal brightness of a feature and background (or different feature’s components) is a quite rare situation, but visibility decreasing of some features after conversion to grayscale is typical. This results in FRI value reduction. So these features can pass thresholding or not in dependence of slight 1The article was translated by the authors.

Received October 10, 2011

change in shooting conditions. This can lead to unsta ble feature matching. Moreover for a wide class of algorithm correct fea ture sorting in order of their importance is critically sig nificant. Many state of art algorithms [8] estimate model parameters (homography or essential matrix) using matched pairs of features from different views. Estimation is fulfilled in two stages: using pairs of “most important” features (in terms of FRI value) and refinement with all feature pairs. If feature ordering is not stable, sets of most important features from differ ent views can contain points corresponding to differ ent points of 3D scene and matching will be incorrect. Shadows present one more problem of feature detection. Shadows have close to background hue and saturation components and a lot differing lightness. In images converted to grayscale shadows frequently become more contrast. It does not accord with human vision properties, which interprets shadows as unim portant features. The corner [9] and edge [10, 11] detectors for color images are known and will be overviewed in Section 2. Unfortunately such approach is not applicable to fea tures which depend on second image derivatives. In this paper the method for color blobs and ridges direc tion is proposed. We know the only work [12] on color blobs detection, comparative discussion of the pro posed method and [12] will be given in Section 5.4. As far as we know no color ridge detectors have been pro posed yet. In our method sizes of blobs and width of ridges are defined adaptively in scalespace. Advantages of the proposed method in comparison with wellknown

ISSN 10546618, Pattern Recognition and Image Analysis, 2012, Vol. 22, No. 1, pp. 221–227. © Pleiades Publishing, Ltd., 2012.

222

KHANINA et al.

scalespace ridge detector [7] will be discussed in Sec tion 6. 2. FEATURE DETECTORS FOR COLOR IMAGES The main idea of the classical corner [9] and edge [10, 11] detectors for color images is substitution of operations on vector of image color components for scalar operations on image brightness, particularly substitution of inner products of vectors of image color components for squared brightness. These color detectors have an important “continu ity” property: small changes in RGB channel values results in small FRI changes (no any kind of threshold switching). Other one property is “compatibility” with grayscale detector: when image color components are equal: R ( x, y ) ≡ G ( x, y ) ≡ B ( x, y ),

(1)

formulae used to construct FRI of color image come to formulae used for grayscale images [3, 4]. Thus the results of detection using color and grayscale algo rithms are the same when (1) is true. We propose the method for blobs and ridge detection in this paper that satisfies “continuity” and “compatibility” properties too. 2.1. Corner Detection Kanade–Lucas corner detector [1, 2] and Harris corner detector [3] are most often used ones. Feature corners are local maxima of FRI. In Kanade–Lucas detector pixel values of FRI present minimal eigen ˆ calculated in every image value of structure tensor C point: ⎛ 2 〈 L x 〉 〈 L x L y〉 ˆ C ( x , y ) = 〈 C〉 = ⎜ ⎜ 2 ⎝ 〈 L x L y〉 〈 L y 〉

⎞ ⎟, ⎟ ⎠

(2)

where L is input grayscale image, Lx, Ly are image derivatives, and notation 〈 …〉 means averaging over a point neighbourhood. Extension to the color images [9] consist in substi tution of color components vector L(x, y) for scalar brightness L(x, y) in derivation of needed FRI: T

L ( x, y ) = ( R ( x, y ), G ( x, y ), B ( x, y ) ) .

(3)

Derivation analogous to [1, 2] with L(x, y) leads to color structure tensor, which looks like (2) but oper ates with vector value L(x, y) and with vector inner (dot) product instead of derivative multiplication.

2.2. Edge Detection Canny detector [4] is considered to be the standard method for edge detection. Edges are points where gradient absolute value is maximal in gradient direc tion. In other words, edges are maxima in FRI, where every pixel of FRI presents highest eigenvalue of a qua dratic form: ⎛ 2 T Lx Lx Ly C ( x, y ) = gg = ⎜ ⎜ 2 ⎝ Lx Ly Ly

⎞ ⎟. ⎟ ⎠

(4)

Di Zenzo–Cumani color extension [10, 11] consists of substitution of value (3) and also satisfies “continu ity” and “compatibility” properties: C = 1 C R + 1 C G + 1 C B 3 3 3 ⎛ L ⋅L L ⋅L = 1 ⎜ x x x y 3 ⎝ Lx ⋅ Ly Ly ⋅ Ly

(5)

⎞ ⎟. ⎠

3. SCALESPACE Differentiation of an image brightness function has been intensively used in Sections 2.1 and 2.2. How ever, the input image is given at discreet pixel mesh in the plane (x, y). Scalespace theory [13–15] proposes to use instead discreet image L(x, y) its’ version L(x, y, t) blurred with the Gaussian kernel G(x, y, t): L ( x, y, t ) = G ( x, y, t ) * L ( x, y ), 2

1 where G ( x, y, t ) = e 2πt

2

(x + y ) –  2t

(6) .

Blurred function L(x, y, t) is infinitely differentia ble. Using convolution properties the derivatives of L(x, y, t) can be calculated via convolution of discreet function L(x, y) with the corresponding derivatives of Gauss function. Thereby image derivatives depend on the blurring parameter t called scale [15]. In feature detection problem there is no a priory known scale, which should be used for derivatives cal culation. The scalespace theory [15] states that a fea ture should be detected at the scale where it is visible better. Usually this scale is proportional to the feature size. This leads to consideration of onedimensional family of images L(x, y, t) blurred with different parameter t. Let us notice that since Gauss function is used as the blurring kernel, and then function L(x, y, t) satis fies diffusion equation [15]: 1 L t =  ( L xx + L yy ). 2

PATTERN RECOGNITION AND IMAGE ANALYSIS

Vol. 22

(7) No. 1

2012

SCALESPACE COLOR BLOB AND RIDGE DETECTION

4. GRAYSCALE BLOB AND RIDGE DETECTION AT A FIXED SCALE

223

L

Image brightness at the point (x, y) can be expanded into Taylor series:

+ −

T

⎞ ⎛ ⎛ ⎞ 1⎛ ⎞ T dL ( x, y ) = g ( x, y ) ⎜ dx ⎟ +  ⎜ dx ⎟ H ⎜ dx ⎟ + …, ⎝ dy ⎠ 2 ⎝ dy ⎠ ⎝ dy ⎠ (8) ⎛ L xx L xy ⎞ where H ( x, y ) = ⎜ ⎟ is Hessian matrix. ⎝ L xy L yy ⎠ In the middle of a blob or a ridge, first derivatives (gradient) are small and the second term becomes the main. Image brightness L(x, y) defines the surface. Eigenvalues λ1, λ2 (|λ1| ≥ |λ2 |) of Hessian matrix characterize curvature of this surface in direction of eigenvectors v1, v2. If both curvature values have the same sign and similar magnitudes then the feature is blob. In case of ridge one of the eigenvalues is close to zero and other’s absolute value is large. The above dis cussion is a short explanation of popular ridge [7] and blob [5, 6] detection techniques basics. Frequently used LoG and determinant of Hessian detectors can be expressed correspondingly as sum and product of the Hessian eigenvalues. Thus grayscale blob detection procedure consists of FRI construction |λ2(x, y)| and local maxima detection via scanning with 3 × 3 frame. Ridge detection consists of FRI construction |λ1(x, y)| followed by finding of local maxima along corresponding eigenvector direc tion.

+ r0

Fig. 1. Second derivative of the onedimensional Gauss function.

Laplacian detector cannot distinguish between blobs and ridges. On the other hand, feature detection using Hessian matrix eigenvalues cannot be directly applied to color images. Usually in order to solve this problem grayscale image is constructed as a projection of color image to some direction in color space, for example, (0.299, 0.587, 0.114) [16]. In this work we propose to select color projection direction adaptively in every image point as a direction in color space of the fastest color change T 1C/ C . c = [ c R c G c B ] =  3

Derivatives calculation in the point (xi, yi) of this adaptively constructed grayscale image via convolu tion with Gaussian derivatives would require forming of separate image Li: L i ( x, y ) = c R ( x i, y i )R ( x, y ) + c G ( x i, y i )G ( x, y ) + c B ( x i, y i )B ( x, y ),

5. COLOR BLOB AND RIDGE DETECTION AT A FIXED SCALE 5.1. Color Variation Vector and Hessian Matrix for Color Image Let us apply Laplace operator to each of image color channels. And let us introduce vector C: T C = 1 [ R xx + R yy, G xx + G yy, B xx + B yy ] . 3

neighbourhood (Fig. 1). Here r0 = σ N G , where σ = t , NG is the dimensionality of Gaussian function used, in our case NG = 2. So we can say that C = C+ – C– is color variation vector in the feature neighbourhood. PATTERN RECOGNITION AND IMAGE ANALYSIS

(10)

where coefficients c(xi, yi) are fixed in all points of Li. Fortunately, we can avoid building of such huge image set {Li} and calculate derivatives in all points directly from color components derivatives using coefficients (10): ˆ ( x, y ) ) = c ( x, y )D ( R ( x, y ) ) D(L R + c G ( x, y )D ( G ( x, y ) ) + c B ( x, y )D ( B ( x, y ) ),

(9)

Application of the Laplace operator, i.e. convolution with second derivatives of Gauss function, means dif 1 ference between weighted mean C– =  [R–, G–, B–]T 3 over the point neighbourhood of radius r0 and weighted mean C+ = 1 [R+, G+, B+]T over outer ring 3

X

(11)

where D(…) is any differentiation operator. Since only adaptive image derivatives are needed for blobs and ridge detection, the proposed method does not include image conversion from color to grayscale. Hessian matrix can be formed using derivatives (12). Further blobs and ridges detection is analogous to detection in grayscale images using eigenvalues λ1, λ2 (|λ1| ≥ |λ2 |) of Hessian matrix (see Section 4). It is easy to see that if the condition (1) is satisfied, the derivatives (12) are equal to grayscale image deriv atives. Thus the proposed algorithm satisfies “compat ibility” property (see Section 2). Vol. 22

No. 1

2012

224

KHANINA et al.

(x , y − 1)

(x − 1, y − 1)

v1(x, y)

(x + 1, y − 1)

A

(x − 1, y )

(x − 1, y + 1)

(x + 1, y )

(x, y)

B

(x, y + 1)

(x + 1, y + 1)

Fig. 2. Points A and B are neighbours of point (x, y) in the direction of the eigenvector v1(x, y), |λ1(x, y)| values at these points obtained via linear interpolation.

5.2. Blob Detection at a Fixed Scale Let us chose threshold β characterizing maximum allowed oblongness of a blob. Local maxima of FRI |λ2(x, y)| (Fig. 3b), where 0 < λ1/λ2 < β, are detected as blobs. These local maxima can be detected via scan ning with 3 × 3 frame and comparing central pixel on FRI with its 8 neighbour pixels. 5.3. Ridge Detection at a Fixed Scale Ridge detection consists of the stages described below: a. Construction of FRI |λ1(x, y)|, obtained from matrix (8) with color derivatives (12) (see Fig. 3c). b. For non maxima suppression (NMS) in the direction of eigenvector v1(x, y) is fulfilled (Fig. 2). Example of NMS result can be seen in Fig. 3d. c. False response suppression. The described ridge detector has response in some nonridge points. In order to suppress false point an additional condition (a)

(b)

should be checked: in ridge points color gradient changes its sign. This condition can be checked during NMS (step B): in the points A and B (Fig. 2) neigh bouring to the current point in direction of eigenvector v1(x, y) gradient signs should be checked in addition to checking values of |λ1(x, y)|. Two variants of color gradient (Lx, Ly) calculation have been considered: using Di Zenzo–Cumani method [10, 11] and with formula (12). Di Zenzo– Cumani method allows calculation of gradient direc tion up to the sign. The only known way for sign ambi guity resolution [17] has artifacts in some cases. On the other hand, color gradient calculated with (12) has no artifacts and keep on the logic of derivatives calcula tion in the proposed algorithm. More over, such color gradients are required only at the extrema points of |λ1(x, y)| where color variation vector c is welldefined. d. Hysteresis. From the left points we chose that ones where detected feature is oblong enough: 0 < λ2/λ1 ≤ 1/β. Following ridge to its ends, relation λ2/λ1 increases over the threshold 0 < λ2/λ1 ≤ 1/β. In order to detect ridges including its ends hysteresis is fulfilled: after selection of points, where 0 < λ2/λ1 ≤ 1/β, neigh bour points, where λ2/λ1 < 1/β are also selected. 5.4. Comparison with Other Methods Adaptive projection (11) preserves feature distin guishability towards background unlike methods using a fixed direction or the adaptive method [12] proposed by Ming and Ma: R ( x, y ) G ( x, y ) L ( x, y ) =  D ( R ( x, y ) ) +  D ( G ( x, y ) ) S ( x, y ) S ( x, y ) B ( x, y ) +  D ( B ( x, y ) ), (12) S ( x, y ) where S ( x, y ) = R ( x, y ) + G ( x, y ) + B ( x, y ) . Unlike the proposed algorithm method [12] take into consideration only the color in the current point and do not use background color. Let us consider an example feature of color (255, 0, 0) against a back ground of color (255, 0, 5). In formula (13) red com ponent, which does not distinguish feature and back (c)

(d)

(e)

Fig. 3. (a) synthetic color image example, (b) |λ2(x, y)|, (c) |λ1(x, y)|, (d) points left in |λ1(x, y)| after non maxima suppression (NMS); (e) detected blobs and ridges. PATTERN RECOGNITION AND IMAGE ANALYSIS

Vol. 22

No. 1

2012

SCALESPACE COLOR BLOB AND RIDGE DETECTION (a)

225

matrix 2 × 2 (8), and detect local maxima via scanning with window 3 × 3 × 3. For scalespace ridge detection we propose to use Hessian matrix 3 × 3 (14) in extension to the approach described in Section 5. Matrix (14) characterizes image brightness change in 3D space (x, y, t).

(b)

H ( x, y , t ) ⎛ L L L ⎞ ⎜ xx xy xt ⎟ = ⎜ L xy L yy L yt ⎟ = { diffusion equation (7) } ⎜ ⎟ ⎝ L xt L yt L tt ⎠ Fig. 4. (b) blobs detected in color image (4a).

⎛ L xxx + L xyy ⎜ L xx L xy  2 ⎜ ⎜ L xxy + L yyy = ⎜ L yy  L xy ⎜ 2 ⎜ + L L + L L + 2L L xxx xyy xxy yyy xxxx xxyy + L yyyy ⎜       ⎝ 2 2 4

ground, has the highest contribution while the most important blue component is suppressed, weights vec tor is (0.98, 0, 0.02). In our method (12) a component has the higher weight the higher change between fea ture and background for this component. The weight vector is c = (0, 0, 1/ 3 ) for the above example.

For ridge detection in scalespace we construct a set of FRIs |λ1(x, y, t)| of 3D Hessian matrix (13) and fulfill non maxima suppression in every point of the obtained 3D space (x, y, t) in two perpendicular direc tions, defined as follows. Direction of the possible ridge in the point (x, y) is defined with eigenvector v2(x, y) of Hessian matrix (8). Quadric form (14) defines 3D ellipsoid. This ellipsoid section with the plane perpendicular to possible ridge direction in 2D is an ellipse. Axes of this ellipse are used as directions of NMS. It should be noted that two eigenvectors of (14) corresponding to two highest eigenvalues cannot be used for NMS in straightforward manner since they yield discontinuous ridges, especially in points where ridge width changes. In other words when ridge repre

6. SCALESPACE BLOBS AND RIDGES DETECTION Edge detection using incorrect scale frequently leads only to some shape change of the detected edges. Unlike edges blobs and ridges will be missed if incor rect scale is used for detection. In the current section the method for scalespace detection which allows to detect feature at the scale where it is better visible is proposed. For blob detection in scalespace we construct needed derivatives (12) using a set of sequential scales tj. Using these derivatives we construct a set of FRIs |λ2(x, y, t)|, where λ2 are second eigenvalues of Hessian (a)

⎞(13) ⎟ ⎟ ⎟ ⎟. ⎟ ⎟ ⎟ ⎠

(b)

(c)

Fig. 5. (b) the ridges detected in color image (5a); (c) detected ridges are drawn as lines with the width corresponding to scale where ridge has been detected and with color of input image in the corresponding point. PATTERN RECOGNITION AND IMAGE ANALYSIS

Vol. 22

No. 1

2012

226

KHANINA et al.

sentation in scalespace |λ1(x, y, t)| goes through scales. Suppression of the false detection via analyzing gradient absolute value is fulfilled for each scale. After ridge points accumulation from all the scales we check constraint λ1/λ2 ≥ β > 0 and apply hyster esis. For oblongness check we use values λ1 and λ2, calculated at the scale where the current point has been detected. The described scalespace ridge detection can be applied to grayscale images without modifications with the exception of direct derivatives calculation without using of formula (12). In our experiments threshold β = 8 and scale grid with the step σi + 1/σi = 4 2 have been used. Examples of the algorithm results are presented in Figs. 4, 5. The only previously proposed scalespace ridge detector [7] can be used only for grayscale images. Proposed in this paper algorithm can be applied both to grayscale and color images. Another one advantage is simplicity of 3D non maxima suppression procedure in comparison with construction of surfaces 3D space and construction of their sections needed for [7]. 7. CONCLUSIONS The scalespace algorithm for blobs and ridges detection in color images has been developed. Up today the feature detectors dependent on first deriva tives (edges and corners) for color images were known. In this paper we have proposed other approach of tak ing into account color information which applicable for features dependent on second derivatives—blobs and ridges. As far as we know, no color ridge detectors were proposed earlier. The only previously known color blob detector has serious disadvantages in com parison with our approach. These weaknesses were discussed in the paper and a failing example was given where Ming and Ma algorithm greatly reduces feature distinguishability. Additionally we introduce in this paper 3D non maxima suppression procedure. It per forms non maxima suppression in two orthogonal directions in scalespace. Application of this NMS procedure for ridge detection allows us make scale space ridge detection sufficiently fast and easy pro grammable for practical use in comparison with other known methods. ACKNOWLEDGMENTS This work was supported by the Federal Program “Scientific and Scientific Pedagogical Personnel of Innovative Russia” in 2009–2013 and the Russian Foundation for Basic Research, project no. 0907 92000HHC_a.

REFERENCES 1. C. Tomasi and T. Kanade, “Shape and Motion from Image Streams: a Factorization Method—Part 3. Detection and Tracking of Point Features,” Tech. Report CMUCS91132 (Computer Science Depart ment, Carnegie Mellon University, Apr. 1991). 2. J. Shi and C. Tomasi, “Good Features to Track,” in Proc. IEEE Conf. on Computer Vision and Pattern Rec ognition (CVPR’94) (Seattle, June 1994), pp. 593–600. 3. C. Harris and M. Stephens, “A Combined Corner and Edge Detector,” in Proc. 4th Alvey Vision Conf. (Manchester, 1988), Vol. 15, pp. 147–151. 4. J. Canny, “A Computational Approach to Edge Detec tion,” IEEE Trans. PAMI 8, 34–43 (1986). 5. D. G. Lowe, “Object Recognition from Local Scale– Invariant Features,” in Proc. ICCV (Kerkyra, Sept. 1999), pp. 1150–1157, Available from: http://www.cs. ubc.ca/~lowe/papers/iccv99.pdf 6. D. G. Lowe, “Distinctive Image Features from Scale– Invariant Keypoints,” Int. J. Comp. 60 (2), 91–110 (2004). 7. T. Lindeberg, “Edge Detection and Ridge Detection with Automatic Scale Selection,” Int. J. Comp. Vision 30 (2), 117–156 (1998). 8. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge Univ. Press, 2004). 9. P. Montesinos, V. Gouet, and R. Deriche, “Differential Invariants for Color Images,” in Proc.14th Int. Conf. on Pattern Recognition (Brisbane, 1998), pp. 838–840. 10. S. Di Zenzo, “A Note on the Gradient of Multi– Image,” Comput. Vision Graph. Image Processing 33, 116–125 (1986). 11. A. Cumani, “Edge Detection in Multispectral Images,” Comput. Vision, Graph. Image Processing 53 (1), 40–51 (1991). 12. A. Ming and H. Ma, “A Blob Detector in Color Images,” in Proc. 6th ACM Int. Conf. on Image and Video Retrieval (Amsterdam, 2007), pp. 364–370. 13. A. P. Witkin, “Scale–Space Filtering,” in Proc. 8th Int. Joint Conf. on Artificial Intelligence (Karlsruhe, Aug. 1983), pp. 1019–1022. 14. J. J. Koenderink and A. J. van Doorn, “The Structure of Images,” Biol. Cybern. 50, 363–370 (1984). 15. T. Lindeberg, ScaleSpace Theory in Computer Vision (Kluwer Acad. Publ., Dordrecht, 1994). 16. W. K. Pratt, Digital Image Processing: PIKS Scientific Inside, 4th ed. (WileyIntersci., 2007). 17. D. P. Nikolaev and S. M. Karpenko, “ColortoGray scale Image Transformation Preserving the Gradient Structure,” in Proc. 20th European Conf. on Modeling and Simulation (ECMS 2006) (Bonn, 2006), pp. 321–323. Natalia Alekseevna Khanina (1989), student, Chair of Mathematical Physics, Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State Univer sity, Russia.

PATTERN RECOGNITION AND IMAGE ANALYSIS

Vol. 22

No. 1

2012

SCALESPACE COLOR BLOB AND RIDGE DETECTION

Dmitry Vladimirovich Yurin (1965), PhD, senior researcher at laboratory of Mathematical Methods of Image Processing, Faculty of Computa tional Mathematics and Cybernetics, Lomonosov Moscow State Univer sity, Russia.

Ekaterina Victorovna Semeikina (1987), PhD student, Chair of Mathematical Physics, Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Russia.

PATTERN RECOGNITION AND IMAGE ANALYSIS

227

Vol. 22

No. 1

2012