A comparative study of different corner detection methods JunJie Liu∗ , Anthony Jakas∗ , Ala Al-Obaidi† , and Yonghuai Liu∗ ∗ Department
of Computer Science Aberystwyth University, Ceredigion SY23 3DB, UK {jul06,ajj08,yyl}@aber.ac.uk † Smart Light Devices, Ltd. Aberdeen, AB24 2YN, UK
[email protected]
Abstract— Interest points are widely used in computer vision applications such as camera calibration, robot localization and object tracking that require fast and efficient feature matching. A large number of techniques have been proposed in the literature. This paper evaluates the state of art techniques for interest point detection including excution time and suitability for real time applications. Such comparative study is crucial for specific applications, since it is always necessary to understand the advantages and disadvantages of the existing techniques so that best possible ones can be selected. The comparative study shows that: (1) the CSS method performs best in corner extraction. It is the fast and the most reliable and has the lowest noise sensitivity with the highest true corner detection rate, even though it still detects some false corners; (2) SUSAN detector would be the second choice and is acceptable and useful in applications requiring a computationally efficient detector and working on a restricted set of images.
I. I NTRODUCTION The use of interest points (corner detectors) to find corresponding points across multiple images and between an image and a real object is a key step in many image processing and computer vision applications. The most notable application includes image matching, real-time gesture recognition, mouth detection and motion tracking and robot navigation. In the last two decades a large number of interest point detectors have been developed. Some of them find points with high local symmetry, others find areas of highly varying texture, while others locate corner points. The comparative study is crucial for camera calibration, robot localization and many of the machine vision applications, since it is always necessary to understand the advantages and disadvantages of the existing techniques so that the best possible ones can be selected. Although some work on comparing and evaluating interest points detector have been published [1], [2], [3], [4], only [3] and [4] compares different cornerness measures. Most of them do not conclude or evaluate which corner detector is most appropriate to real time applications. Even though [4] compares four corner detectors [6], [7], [8], [5], most of them are the extensions of [5]. In this paper, we carry out a comparative study of corner detectors from different categories and also the latest ones with real time applications in mind. Section 2 outlines each of the corner detectors selected, while Section 3 presents the experimental results based on both synthetic and real data. Finally, some conclusion is drawn in Section 4.
II. C ORNER D ETECTION In this section, we outline four representative corner detectors [5], [7], [9], [10] which are especially suitable for real time applications such as camera calibration and 3D reconstruction and motion tracking: Harris, Noble, SUSAN, and CSS. A. Harris and Stephens / Plessey corner detection Harris and Stephens improved Moravec’s corner detector by considering the differential of the corner score with respect to direction directly, instead of using shifted patches. This corner detector is an alternative approach for feature detection and based on a measure H(x, y) of corner strength. For each pixel(x, y) in the image calculate the autocorrelation matrix M : A C M= (1) C B ∂I 2 ∂I ∂I 2 ∂I 2 ) ⊗ w, B = ( ∂y ) ⊗ w, C = ( ∂x , ∂y ) ⊗ w, where A = ( ∂x ⊗ is the convolution operator, w is the Gaussian window. The corner strength and H(x, y) is defined as:
H(x, y) = det(M ) − k(trace(M ))2
(2)
where det(M ) = λ1 λ2 = AB − H 2 and trace(M ) = λ1 + λ2 = A + B. If λ1 ≈ 0 and λ2 ≈ 0 then this pixel has no features of interest. If λ1 ≈ 0 and λ2 is a large positive value, then an edge is found. If λ1 andλ2 is two large positive values, then a corner is found. B. Noble corner detection This corner detector Noble [7] defines a new cornerness measure as a function of the autocorrelation matrix M in the Harris and Stephens corner detector. Considering that the variable k in the Harris and Stephens cornerness measure is likely to be small, he proposed a new cornerness measure H(x, y) as: det(M ) (3) H= trace(M ) + ε which removes parameter k, but introduces a small constant ε to avoid a singular denominator in case of a rank zero divisor. While the parameter k is usually data dependent, the constant ε is usually data independent and much easy to set up.
If c is the rectangular function, then n is the number of pixels in the mask which are within t of the nucleus. The response of the SUSAN operator is given by: g − n(M ) if n(M ) < g R(M ) = (5) 0 otherwise, (a) Four circular masks at different locations in a simple image
where g is named the ”geometric threshold”. In other words the SUSAN operator only has a positive score if the area is small enough. The locally smallest USAN can be found using non-maximal suppression. The value t determines how similar points have to be to the nucleus before they are considered to be part of the univalue segment. The value of g determines the minimum size of the univalue segment. If g is large enough, then it becomes an edge detector. For corner detection, two further steps are used. Firstly, use moment calculations applied to the USAN to find edge direction and then apply non-maximum suppression thinning and sub-pixel estimation, if required.
(b) USANs are shown as the white parts of the mask Fig. 1.
SUSAN and USANs
D. Corner detector based on global and local curvature properties
C. SUSAN corner detector The principle implemented by Smith and Brady [9] is a digital approximation of a circular mask. SUSAN places a circular mask over the pixel to be tested (the nucleus). As shown in Figure 1(a), a circular mask is shown at five image positions. When the brightness of each pixel within the mask is compared with that of its nucleus, an area of the mask which has the same (or similar) brightness as the nucleus can be defined. This area of the mask shall be known as the ”USAN”, an acronym standing for ”Univalue Segment Assimilating Nucleus”. In Figure 1(b) each mask from Figure 1(a) is depicted with its USAN shown in white. This concept that each image point is associated with a local area of similar brightness is the basis for the SUSAN principle. The local area or USAN contains much information about the structure of the image. It is essentially region finding on a small scale. From the size, centroid and second moments of the USAN two dimensional features and edges can be detected. The area of an USAN conveys the most important information about the structure of the image in the region around any point in question. As can be seen from Figures 1(a) and 1(b), the USAN area is at a maximum when the nucleus lies in a flat region of the image surface, it falls to half of this maximum very near a straight edge, and falls even further when inside a corner. It is this property of the USAN’s area that is used as the main determinant of the presence of edges and two dimensional features. To implement it as a corner detector, considering the mask M , a pixel in this mask is represented by m ∈ M . The nucleus is at m 0 . Every pixel is compared to the nucleus using the comparison function: (I(m)−I( m 0 ))6 t c(m) = e where t determines the radius, and the power of the exponent has been determined empirically. The area of the SUSAN is given by n(M ) =
m∈M
c(m)
(4)
Corner detector based on global and local curvature properties [12] is an extension of the CSS method [10], [11] which is a curvature-based detector that detects both fine and coarse features accurately at low computational cost. This corner detector marks both images’ edges and corners available for later processing and then restricted to find out which are the interest points. The traditional singlescale algorithms that detect corners by considering local properties may either miss fine features or detect noisy points as false corners. This corner detector utilizes both global and local curvature properties, and balance their influence when extracting corners. After initial list of corner candidates, this detector then optimizes the final result: remove the roundcorner, false-corner and end-points. 1) Detect the edges using Canny edge detector to obtain a binary edge map. 2) Extract contours from the edge map. When an edge reaches an end point, fill the gap and continue the extraction if the end point is nearly connected to another end point, or mark this point as T-junction corner if the end point is nearly connected to an edge contour, but not to another end point. 3) Compute the curvature at a fixed low scale, and the pixels with local maxima of the absolute curvatures as corner candidates. 4) Compute a threshold adaptively according to the mean curvature within a region of support. Round corners are removed by comparing the curvature of corner candidates with the adaptive threshold. 5) Remove the FC based on a dynamically recalculated region of support, evaluate the angles of the remaining corners. 6) Finally, consider the end points of open contours, and mark them as corners unless they are very close to others.
(a) Harris
(b) Noble
(c) SUSAN
(d) CSS
(e) Harris
(f) Noble
(g) SUSAN
(h) CSS
(i) Harris
(m) blocks(59)
(j) Noble
Ref.Artificial
(k) SUSAN
(n) Ref.Blocks(60)
(l) CSS
(o) Ref.House(76)
Fig. 2. Corner detection results of images free from noise. Top row: Artificial blocks; second row: Blocks; third row: House; last row: Reference corners.
III. C OMPARATIVE E VALUATION In this section, detection results of all the selected methods are presented. An ideal corner detector satisfies the following main criteria: (1) All ”true” corners should be detected and ”false” corners should not be detected; (2) Corner points should be well localized; (3) Detector should be robust with respect to noise; and (4) Detector should be computationally efficient. Thus, the corner detector will be evaluated from these four aspects using both synthetic artificial blocks and real blocks and house image data. To require the as much as true corners, the best region size will be chosen during the tests. A. Comparative evaluation with noise free data In this section, we report the experimental results based on images free from noise. All the images were converted to gray ones before applying any corner detector. During
the experiments, we adjust all the parameters to obtain best results in different detectors. In addition, ground truth corners are identified manually. Since it is often difficult to decide whether or not a point should be defined as a corner, only the trivial ones are shown as the ground truth. The result is shown in Fig.2 and Tables {I,II,III}. In the tables, TC means ”True corners”, FC means ”False corners”, MC means ”Missed corners”, LE means ”Localization error” in pixels and computational time is measured in seconds. The localization error is measured in pixels and computed by Eq. 6, where (Cˆxi , Cˆyi ) is a reference corner in x, y coordinates and (Cxi , Cyi ) is the corresponding detected corner position. LE =
N 1 (Cˆxi − Cxi )2 + (Cˆyi − Cyi )2 N i=1
(6)
The synthetic artificial blocks image contains all types
TABLE I PARAMETERS OF INTEREST FOR THE EXPERIMENTS BASED ON ”A RTIFICIAL BLOCKS ” Detector Harris Noble SUSAN CSS
TC 55 59 61 35
FC 2 48 0 2
MC 6 2 0 26
LE 1.2809 1.1178 0.5251 1.0094
Time 0.2877 0.1904 0.1609 0.1546
TABLE III PARAMETERS OF INTEREST FOR THE EXPERIMENTS BASED ON ”H OUSE ” Detector Harris Noble SUSAN CSS
TC 26 39 52 62
FC 28 27 30 18
MC 55 42 29 19
LE 1.3920 1.7022 1.9570 1.0488
Time 0.5079 0.3284 0.1625 0.1885
TABLE II PARAMETERS OF INTEREST FOR THE EXPERIMENTS BASED ON ”B LOCKS ” Detector Harris Noble SUSAN CSS
TC 38 46 39 47
FC 12 21 24 9
MC 21 13 22 14
LE 1.6976 1.5813 1.4923 1.3833
Time 0.5966 0.3904 0.1460 0.1685
of corners: L-junction, Y-junction, T-junction, and Arrowjunction except X-junction such as Fig.??. Most of these corner types in the image have been detected by all the selected methods. There is no doubt that SUSAN detector has the best result. It detects all the corners without any FC, all the points are perfectly matched to the ground truth. Though the Noble detector picks up all the corners but also detects lots of unwanted ones and mixes up the oblique edge and corners. Compared to the original Harris corner detector, the T-junction corner detection has been improved by the Noble corner detector. The CSS method strongly relies on the edge detection method. Fig.3(a) shows the detected edges used in the CSS corner detector. It can be seen that the T-Junction type edge/corners have not been fully detected. Due to the complexity of real images ”Blocks” and ”House”, none of the methods can detect all the TCs. Obviously, the CSS method performs very well over both images with more than 80% TCs being detected. Also, compared to the other three methods, the FCs and MCs are much fewer. Harris corner detector has poor performance in these images, but its extension, Noble corner detector, is much better. This is because Noble corner detector avoids the difficulty in setting the parameter k in Eq. 2. The experimental results show that Eq. 3 is more sensitive and suitable for detecting different types of corners. On the whole, the CSS method produces the smallest localization errors of corners. The Noble corner detector has successfully improved the Harris corner detector. SUSAN detector has produced the smallest corner localization error in the artificial block image. All of the corner detectors were implemented in Matlab2008 on a Dual-Core(2.5GHz) PC. In all the cases, the CSS detector is the most computationally efficient, while Harris detector is the most computationally intensive. For the CSS detector, the 70 ∼ 80% computational time is spent on edge detection and the rest is to get rid of FCs. Harris and Noble detectors require the most time because they use large radius (9, 11, etc.) of regions considered in the non-maximal suppression. SUSAN corner detector produces better results
(a)
(b)
(c)
Fig. 3. [a] Canny edge detection in CSS over ”Artificial blocks”. [b,c] Canny edge detection in CSS with ”Blocks” corrupted by noise.
with less computational time. B. Comparative evaluation with noisy images Good corner detector should be robust with respect to noise(Gaussian white noise). Hence, the evaluation in this section is about the detectors’ sensitivity to different levels of noise. Obviously, the more TCs detected the better. To simulate different levels of noise, the Gaussian white noise (σ = 0.1, 0.2, 0.3, 0.4) was added to both images ”Artificial blocks” and ”Blocks” and all the TCs are defined as those with localization error less than 2.5 pixels in all the tests. The experimental results are presented in Fig.{4,5,6,7} Those figures show that as the noise level increases, the number of TCs decreases, while the numbers of useless and unwanted corners increase. This is expected, a robust detector should detect as many TCs as possible. In general, the CSS corner detector picks up the most TCs but with some strange corners. The reason for that is because the Canny edge detector in CSS mistook the shadow boundaries as edges, as illustrated in Fig.3(b) and 3(c). Since the artificial blocks’ edges and corners are more regular than those in the real blocks image, all the selected methods successfully detected the key and useful corners on the right hand side but not on the left hand side which have been blurred by the noise. The real blocks image is more complex, the noise thus heavily affects the final result. Fig.5 and Fig.7 show that except the Harris corner detector, all the others detect 80% TCs when the noise level is low (σ = 0.1). But when the noise increases, the CSS detector is more stable than the others though it still detects some strange corners. In addition, the new version of Harris corner detector – Noble – works better in this round, but both of them are sensitive to noise. As we can see, from the low to high level noise, their detection rate of TCs goes down sharply. IV. C ONCLUSION In this paper, we carried out a detailed comparative study of four corner detectors using both synthetic and real data. In summary, Harris operator suffers from poor localization
(a) Harris,σ = 0.1
(b) Noble,σ = 0.1
(c) SUSAN,σ = 0.1
(d) CSS,σ = 0.1
(e) Harris,σ = 0.2
(f) Noble,σ = 0.2
(g) SUSAN,σ = 0.2
(h) CSS,σ = 0.2
(i) Harris,σ = 0.3
(j) Noble,σ = 0.3
(k) SUSAN,σ = 0.3
(l) CSS,σ = 0.3
(m) Harris,σ = 0.4
(n) Noble,σ = 0.4
(o) SUSAN,σ = 0.4
(p) CSS,σ = 0.4
Fig. 4.
Fig. 6.
Corner detection results of the artificial blocks image.
TCs v.s. Noise in ”Artificial blocks”
and is computationally expensive but Noble corner detector has improved it and has achieved a better detection rate Cs ) in most cases. For these reasons the Noble operator ( FT Cs
Fig. 7.
TCs v.s. Noise in ”Blocks”
is widely used in practice. SUSAN detector finds many FCs along some diagonal edges. It performs well on the synthetic artificial block image and real house image, but
(a) Harris,σ = 0.1
(b) Noble,σ = 0.1
(c) SUSAN,σ = 0.1
(d) CSS,σ = 0.1
(e) Harris,σ = 0.2
(f) Noble,σ = 0.2
(g) SUSAN,σ = 0.2
(h) CSS,σ = 0.2
(i) Harris,σ = 0.3
(j) Noble,σ = 0.3
(k) SUSAN,σ = 0.3
(l) CSS,σ = 0.3
(m) Harris,σ = 0.4
(n) Noble,σ = 0.4
(o) SUSAN,σ = 0.4
(p) CSS,σ = 0.4
Fig. 5.
Corner detection results of the blocks image.
poorly on the real blocks image. For applications requiring a computationally efficient detector and working on a set of restricted images, the detection rate of the SUSAN detector is acceptable and useful. The CSS method performs best in the corner extraction. It is the fastest, the most reliable and has lowest noise sensitivity. Even though it still generates some FCs, it is acceptable - it has the highest TC detection rate. It is thus suitable to real time applications. R EFERENCES [1] Schmid, C., Mohr, R., Bauckhage, C.: Comparing and evaluating interest points. Proc. Sixth ICCV, pp. 230-235, 1998. [2] Tissainayagam, P., Suter, D.: Assessing the performance of corner detectors for point feature tracking applications. Image and Vision Computing, 22, 663-679(2004). [3] Zuliani, M., Kenney, C., Majunath, B.S.: A mathematical comparison of point detectors. Proc IEEE Computer Vision and Pattern Recognition Workshops, 2004.
[4] Kerr D., Coleman S., Scotney B.: Comparing cornerness measures for interest point detection. Proc. Int. Machine Vision and Image Processing Conf., pp. 105-110, 2008. [5] Harris, C., Stephens, M.: A combined corner and edge detector. Proc 4th Alvey Vision Conf. pp. 147-151, 1988. [6] Coleman, S.A., Kerr, D., Scotney, B.W.: Concurrent edge and corner detection. Proc. ICIP, pp. 273-276, 2007. [7] Noble, J.A.: Finding corners. Image and Vision Computing, 6(2), 1988. [8] Shi J. and Tomasi C.: Good features to track. Proc. CVPR, 1994. [9] Smith S.M. and Brady M. SUSAN - a new approach to low level image processing. International Journal of Computer Vision, 23(1), 45-78(1997). [10] Mokhtarian F. and Suomela R.: Curvature scale space for robust image corner detection. Proc. ICPR, 1998. [11] Mokhtarian F. and Mackworth A.K.: A theory of multiscale, curvaturebased shape representation for planar curves. IEEE Trans. PAMI, 14(8), 789-805(1992). [12] Xiao C.H., Yung, N.H.C.: Corner detector based on global and local curvature properties. Optical Engineering, 47(5), p. 057008(2008).
