Efficient and Robust Corner Detectors Based on Second ... - IEEE Xplore

IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 9, SEPTEMBER 2017

1393

Efficient and Robust Corner Detectors Based on Second-Order Difference of Contour Xinyu Lin, Ce Zhu, Fellow, IEEE, Qian Zhang, Xiaolin Huang, Member, IEEE, and Yipeng Liu, Senior Member, IEEE

Abstract—As one of the most significant local features of image, corner is widely used in many computer vision tasks. Corner detection aims to achieve the highest possible detection accuracy while minimizing the computational complexity. In this letter, we first introduce a new measurement termed as second-order difference of contour (SODC), and then examine its regular distribution, which is found to provide useful information to distinguish corners from noncorners. Based on the SODC distribution characteristics, we propose two novel corner detectors to measure the response of contour points using Manhattan distance and Euclidean distance, respectively. Numerical experiments demonstrate that the Manhattan detector greatly decreases the computational complexity, while the Euclidean detector outperforms the state-of-the-art corner detectors in terms of repeatability and localization error. Index Terms—Corner detection, multiscale analysis, robustness, second-order difference of contour (SODC).

I. INTRODUCTION S ONE of the most important local features of image, corner is frequently used in scene analysis, three-dimensional (3-D) reconstruction, camera calibration and robot navigation. In the field of contour-based corner detection, corner is usually defined as the point with local maximal curvature in contour or the intersection point of two line contours in 2-D images. A large amount of contour-based corner detectors were proposed [1]–[21] in the last decades. The most classical kind of contourbased corner detector is the curvature scale space (CSS) based corner detector. Mokhtarian et al. proposed CSS corner detector based on the CSS technique [2] and it became the landmark algorithm in this field. After that, many other variations [3]–[5], [7], [8] from the basic CSS corner detector have been developed. For instance, Zhong et al. proposed a direct curvature

A

Manuscript received April 12, 2017; revised June 1, 2017; accepted June 21, 2017. Date of publication July 11, 2017; date of current version August 8, 2017. This work was supported in part by the National Natural Science Foundation of China under Grant 61602091, Grant 61571102, and Grant 61603248, in part by the Fundamental Research Funds for the Central Universities under Grant ZYGX2016J199 and Grant ZYGX2014Z003, and in part by the National High Technology Research and Development Program of China 863 program under Grant 2015AA015903. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vasileios Mezaris. (Corresponding author: Ce Zhu.) X. Lin, C. Zhu, Q. Zhang, and Y. Liu are with the School of Electronic Engineering/Center for Robotics, University of Electronic Science and Technology of China, Chengdu 611731, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). X. Huang is with the Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: [email protected]). Color versions of one or more of the figures in this letter are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/LSP.2017.2724851

scale space algorithm [8] as a derivative technique of CSS to lower the computational complexity. There are also many other contour-based corner detectors which apply other mechanisms such as geometric relations between the points on the contour to extract corners. Awrangjeb et al. proposed chord to point distance accumulation (CPDA) [10] and FAST-CPDA [11] detectors in which the CPDA metric instead of the curvature is employed to develop the corner response function. Zhang et al. used difference of Gaussian (DoG) [12] operation to calculate the corner response function in multiscale space. Teng et al. proposed an effective and efficient corner detector known as chord to triangular arms ratio (CTAR) detector [16], which has been reported to have high repeatability, low localization error, and low computational complexity in robust corner detection [16]. The above-mentioned detectors have been successfully applied but they still suffer from some weaknesses mainly in two aspects. First, the CSS-based corner detectors [1]–[5], [7], [8] are inclined to produce errors and instability because of their sensitivity to the noise and local variation on the contour [10], which could be otherwise amplified leading to poor detection performance. Second, almost all of these detectors based on geometric relations involve square root operations, resulting in high complexity. In this letter, we propose two novel corner detectors based on second-order difference of contour (SODC) to enhance the performance of contour-based corner detectors. We exploit the characteristics of the high-order difference of contour and select the second-order difference to develop the corner response function. For the first detector, Manhattan distance is used as the response measure, which avoids the square root operation and streamlines the calculation considerably. Although the second detector with Euclidean distance may involve more computational complexity compared with the first one, it outperforms the other nine testing corner detectors in detection performance. The rest of this letter is organized as follows. Section II introduces the conception of SODC with the distribution examined. Two corner detectors are proposed in Section III. Numerical experiments and discussions are given in Section IV. Finally, conclusions are drawn in Section V.

II. SECOND-ORDER DIFFERENCE OF CONTOUR In DoG corner detector [11], Zhang et al. utilized the Euclidean distance between the contours at different scales to construct the corner response function, while the valuable information about the neighborhood points on the same contour

1070-9908 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

1394

IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 9, SEPTEMBER 2017

Fig. 1. (a) Contour marked in green; (b) x coordinate component of the contour; (c) y coordinate component of the contour; (d) the SODC map of the contour; (e) x component of SODC; (f) y component of SODC.

Fig. 2. (a)–(d) First-, second-, third-, and fourth- order difference map of contour.

is not fully utilized. Instead of using the difference between two evolved contours in DoG corner detector, in this letter, we introduce a novel metric using the coordinate difference between the points on the same contour. For a given contour, we denote Pi = (xi , yi ), i = 1, 2, ..., n, as the ith point on the contour, where n is the number of points on the contour. The firstand the second-order difference of contour point Pi are defined, respectively, as ΔPi = Pi+k − Pi 2

Δ Pi = (Pi+k − Pi ) − (Pi − Pi−k )

(1) (2)

where k is a predefined step size. Given a contour illustrated in Fig. 1(a), Fig. 1(d) plots the map of SODC. The five sharp corners denoted as 1, 2, 3, 4, 5 in Fig. 1(a) are easily identifiable and visually prominent on the green contour. The second-order difference results of the five corners correspond to the five arcshapes of the second-order difference map of the contour in Fig. 1(d). Besides, the sharper the contour is, the larger the distance between the farthest arc-shape point and the accumulated center becomes. For the contour points that are naturally flat and visually less significant, the corresponding points in the second-order difference map are aggregated around the central point (0, 0). Likewise, the higher order difference of the points can be obtained. From the high-order difference maps shown in the Fig. 2, we can see that with increasing order, the distribution of difference map reveals more disorderly distribution. To our observation, the SODC has exhibited regular distribution that provides useful information to distinguish corner points from others on the contour. More specifically, it is evidently seen from Fig. 1(b), (c), (e), (f) that when x or y component is considered independently, the turning points on the contour are associated with the local extreme points in the second-order difference map. We use the x coordinate component of the contour to discuss why the above-mentioned observation presents. We can categorize the contour points roughly into three classes in x component in Fig. 1(b): points at the peaks, the points far away from the peaks, and points near the peaks. We use Xi , Xp , and Xj to represent the three classes, respectively, in Fig. 3. As described in (2), the

Fig. 3. points.

Example of x coordinate component of contour with three kinds of

second-order difference of Xi can also be obtained as the subtraction of di+k ,i and di,i−k , which denote the difference between Xi+k and Xi , and the difference between Xi and Xi−k , respectively. Likewise, the second-order difference value of Xp and Xj can be obtained as shown in Fig. 3. Especially, these differences are positive or negative represented by the direction of arrows shown in Fig. 3. An upward arrow means the difference value is positive and vice versa. By comparing all the differences in Fig. 3, we can easily find that the subtraction value of the two neighboring differences for the first class of contour points (e.g., Xi ) is of the largest magnitude due to the opposite directions. However, the points in the second class (e.g., Xp ) contain two neighborhood differences with the same direction and relatively small and close difference values, so the subtraction of the two differences is biased around zero. For the contour points in between (e.g., Xj ) may have two neighborhood differences with the same or different sign (related to the step size k), and their subtraction is of the magnitude between 0 and the maximum. Since many contours are approximately piecewise linear and each corner is the junction of segments, the second-order differences tend to appear to be sparse while nonzero components of large magnitudes are associated with corners. III. SODC-BASED CORNER DETECTION A. Corner Response Function As we discussed in Section II, the distance from a point to the aggregated zero center on the second-order difference map can represent well the characteristics of the point.

LIN et al.: EFFICIENT AND ROBUST CORNER DETECTORS BASED ON SECOND-ORDER DIFFERENCE OF CONTOUR

Accordingly, we consider two types of distances in the construction of corner response function, leading to the two corner detectors, respectively. The first detector termed SODC-M utilizes the Manhattan distance. Here (0, 0), represents the coordinates of the aggregated center in the difference map, and (xi+k − 2xk + xi−k , yi+k − 2yi + yi−k ) represents the coordinates of a given point (xi , yi ). We propose the corner response function ρ according to Manhattan distance as ρ=

|xi+k − 2xi + xi−k | + |yi+k − 2yi + yi−k | . 2k

(3)

Likewise, for the second corner detector denoted as SODC-E, we employ the Euclidean distance to measure the corner response function as  (xi+k − 2xi + xi−k )2 + (yi+k − 2yi + yi−k )2 . (4) ρ= 2k B. SODC-Based Corner Detectors The outline of the SODC detectors can be summarized as follows. 1) Extract edges and contours from the original images, and label them as the line-contours or loop-contours. 2) Find the T-junctions and mark them as T-corners. 3) Utilize Gaussian kernels with different standard deviations to evolve the contours in scale space. 4) For the contours in the scale space, calculate the corner response function according to (3) and (4). 5) Multiply these corner response values at different scales as the final corner response result. 6) Perform the nonmaxima suppression and select those points with local maxima in the final corner response as the candidate corners. Those candidate corners with the final corner response value larger than a predefined threshold Q are identified to be the final corners. 7) Classify the T-corners as the final corners if they are far away from the detected corners. Like many other contour-based corner detectors, Step 1 to 3 are the preprocessing operations. More details can be found in CPDA [10], FAST-CPDA [11], and CTAR [16]. Inspired by the MSCP [7] corner detector, we take the product of the corner response values at different scales as the final corner response output to make our detectors more robust. Here, we choose scale 1, 3, and 5 to form the scale space of contours. In fact, step size k has great influence on the distribution of second-order difference map, which in turn affects the final detection performance. In our experiments, we set k as 7 for both of the two detectors. In the step of nonmaxima suppression, if the final corner response output of a point is lager than the values of the neighboring four points (two on each side), this point is identified as the candidate corner. Then, a threshold Q is utilized to refine the candidates. Here, the value of Q is empirically set as 0.025 for the first detector and 0.01 for the second detector. Finally, a T-corner is added to the final corner set if there is no detected corners in its 2 × 2 neighborhood.

1395

IV. NUMERICAL EXPERIMENTS We compare our detectors against nine state-of-the-art corner detectors in terms of two evaluation metrics. The nine popular contour-based corner detectors are CTAR[16], MSCP[7], ARCSS [5], He and Yung [4], GCM [13], DoG [12], CPDA [10], FAST-CPDA [11], and ANDD [15]. A. Evaluation Datasets and Metrics Following the standard process in literature [10], [11], [16], we use the publicly available CPDA dataset,1 including some synthetic images such as Block and real-world images such as Lena, to evaluate the performance of the contour-based corner detectors. There are totally 23 original images in CPDA dataset. Seven types of image transformations are made for the images to generate test images, including rotation, uniform scale, nonuniform scale, combined transformations, lossy JPEG compression, Gaussian noise, and shearing [10]. For any one of the original images in the dataset, there are total 378 transformed images as testing ones. Except for the CPDA dataset, we also randomly select 32 natural scene images from Mikolajczyk’s dataset2 to form a dataset. And the seven types of image transformation are also performed to generate the test dataset. Repeatability and localization error are the two widely used indicators for corner detection [5], [10], [11], [13], [16]. Repeatability R measures the proportion of repeated corners between original and test images. Localization error Le is defined as the amount of pixel deviation of a repeated corner. It is measured as the root-mean-square error (RMSE) of the repeated corner locations in the original and test images. Both of them can be computed as follows:   1 1 Nr + , (5) R= 2 No Nt   Nr  1   Le = (xoi − xti )2 + (yoi − yti )2 (6) Nr i=1 where No represents the number of reference corners in the original image, Nt represents the number of detected corners in the test image, Nr represents the number of repeated corners between the original and test image, and (xoi , yoi ) and (xti , yti ) are the positions of the ith repeated corners in the original image and test image, respectively. An RMSE value of maximum three pixels is allowed to find a repetition. B. Experimental Results and Discussions 1) Computational Efficiency: For the existing contour-based corner detectors, CTAR, CPDA, and FAST-CPDA are reported to be highly efficient [16]. In this letter, we mainly compare the computation speed of our two detectors against the three detectors. The preprocessing and postprocessing steps mentioned in Section III-B are generally similar for all the testing contourbased corner detectors. In the efficiency comparison, we mainly 1 http://users.monash.edu.au/~mawrangj/corner.html 2 http://lear.inrialpes.fr/people/mikolajczyk/

1396

IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 9, SEPTEMBER 2017

TABLE I COMPUTATIONAL EFFICIENCY COMPARISON AMONG FIVE TESTING METHODS ON THE CPDA DATASET

CPDA FAST-CPDA CTAR SODC-E SODC-M

Operations

Time (μs)

Main operations

54n

1033

n + 54n p 3n 3n

Euclidean distance involving square root operations

162 49 40

3n

32

Manhattan distance with simple arithmetic operations

(Operations: number of distance calculations for an n -point contour. n p is the number of obtained candidate points in FAST-CPDA. Time: average running time per image.)

Fig. 5. Average repeatability and localization error in two datasets with seven types of image transformations. (gn: Gaussian noise; jpg: JPEG compression; rot: rotation; us: uniform scale; nus: nonuniform scale; rotscl: combined transformations; sh: shearing).

TABLE II PERFORMANCE COMPARISON AMONG CONTOUR-BASED CORNER DETECTORS ON CPDA AND MIKOLAJCZYK’S DATASETS Algorithms

Fig. 4.

Corners of “House” detected by eight contour-based corner detectors.

focus on the running time and computational efficiency of corner response function of each detector. Table I gives the computational efficiency of the five contourbased corner detectors, including number of distance calculations for an n-point contour and averaged running time per image. The running time of the testing detectors in the MATLAB environment is obtained on a Windows 8.1 computer with 3.20 GHz Intel(R) Core(TM) i5-4570 CPU and 8.00 GB RAM on the CPDA dataset. As we know, the calculation of Euclidean distance is computationally expensive since the square root operation is involved. For a contour with n points, CPDA detector needs 54n square root operations, while FAST-CPDA requires n + 54np , where np is the number of obtained candidate points. Both CTAR and the proposed SODC-E detectors demand 3n square root operations while SODC-E achieves the best detection accuracy. In contrast, the proposed SODC-M corner detector utilizes the Manhattan distance instead of Euclidean distance to calculate the corner response function, which significantly increases the calculation speed. The running time clearly demonstrates the superiority of our proposed detectors in computational efficiency. 2) Performance Analysis: Fig. 4 shows some visualization results, with the corners of “House” image detected by the eight testing algorithms. Fig. 5 shows the average repeatability and localization error of eleven detectors with respect to the seven types of image transformations. Table II gives the overall perfor-

SODC-M SODC-E GCM DoG He and Yung CTAR MSCP FAST-CPDA CPDA ARCSS ANDD

CPDA dataset

Mikolajczyk’s dataset

R

Le

R

Le

0.7560 0.7647 0.6679 0.6781 0.6530 0.7566 0.7305 0.7250 0.7356 0.5840 0.6328

1.1036 1.0833 1.2216 1.1966 1.3167 1.1685 1.2456 1.1311 1.1096 1.3676 1.2608

0.5810 0.5790 0.5417 0.5424 0.5340 0.5668 0.5652 0.5618 0.5692 0.4526 0.4910

1.1061 1.0804 1.3803 1.3599 1.4324 1.1832 1.2561 1.1323 1.1254 1.3488 1.4259

mance of the evaluated detectors under all of the tranformations. For the CPDA dataset, SODC-E performs best in both the evaluation metrics among all the corner detectors, followed by CTAR. Although our SODC-M ranks the third, it only slightly loses to the CTAR. For Mikolajczyk’s dataset, our SODC-E and SODCM are the top two performers, followed by CPDA and CTAR. We see that the performance of CTAR detector degrades in the second dataset containing all natural images. V. CONCLUSION In this letter, we investigated the high-order difference of contour and examined the characteristics. Based on that, we developed two novel contour-based corner detectors utilizing the SODC, in which corners are identified as the points far away from the aggregated center in the second-order difference map. The Manhattan distance and Euclidean distance are used to measure the corner response, respectively. Numerical experiments showed that the two proposed detectors were superior in terms of detection results, robustness, and computational complexity.

LIN et al.: EFFICIENT AND ROBUST CORNER DETECTORS BASED ON SECOND-ORDER DIFFERENCE OF CONTOUR

REFERENCES [1] A. Rattarangsi and R. Chin, “Scale-based detection of corners of planar curves,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 4, pp. 430–449, Apr. 1992. [2] F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 12, pp. 1376–1381, Dec. 1998. [3] F. Mokhtarian and F. Mohanna, “Enhancing the curvature scale space corner detector,” in Proc. Scandinavian Conf. Image Anal., Bergen, Norway, Jun. 2001, pp. 145–152. [4] X. C. He and N. H. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng., vol. 47, no. 5, pp. 057008-1–057008-12, 2008. [5] M. Awrangjeb, G. J. Lu, and M. Murshed, “An affine resilient curvature scale-space corner detector,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Honolulu, HI, USA, Apr. 2007, pp. 1233–1236. [6] M. Awrangjeb and G. J. Lu, “An improved curvature scale-space corner detector and a robust corner matching approach for transformed image identification,” IEEE Trans. Image Process., vol. 17, no. 12, pp. 2425–2441, Dec. 2008. [7] X. H. Zhang, M. Lei, D. Yang, Y. Z. Wang, and L. T. Ma, “Multi-scale curvature product for robust image corner detection in curvature scale space,” Pattern Recognit. Lett., vol. 28, no. 5, pp. 545–554, 2007. [8] B. J. Zhong and W. H. Liao, “Direct curvature scale space: Theory and corner detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 508–512, Mar. 2007. [9] D. M. Tsai, H. T. Hou, and H. J. Su, “Boundary-based corner detection using eigenvalues of covariance matrices,” Pattern Recognit. Lett., vol. 20, no. 1, pp. 31–40, 1999. [10] M. Awrangjeb and G. J. Lu, “Robust image corner detection based on the chord-to-point distance accumulation technique,” IEEE Trans. Multimedia, vol. 10, no. 6, pp. 1059–1072, Oct. 2008. [11] M. Awrangjeb, G. J. Lu, C. S. Fraser, and M. Ravanbakhsh, “A fast corner detector based on the chord-to-point distance accumulation technique,” in Proc. Int. Conf. Digital Image Comput.: Techniques Appl., Melbourne, Australia, Dec. 2009, pp. 519–525.

1397

[12] X. H. Zhang, H. X. Wang, M. J. Hong, L. Xu, D. Yang, and B. C. Lovell, “Robust image corner detection based on scale evolution difference of planar curves,” Pattern Recognit. Lett., vol. 30, no. 4, pp. 449–455, 2009. [13] X. H. Zhang, H. X. Wang, A. W. Smith, X. Ling, B. C. Lovell, and D. Yang, “Corner detection based on gradient correlation matrices of planar curves,” Pattern Recognit., vol. 43, no. 4, pp. 1207–1223, 2010. ¨ [14] C. Topal, K. Ozkan, B. Benligiray, and C. Akinlar, “A robust CSS corner detector based on the turning angle curvature of image gradients,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Vancouver, BC, Canada, May 2013, pp. 1444–1448. [15] P. L. Shui and W. C. Zhang, “Corner detection and classification using anisotropic directional derivative representations,” IEEE Trans. Image Process., vol. 22, no. 8, pp. 3204–3218, Aug. 2013. [16] S. W. Teng, R. M. N. Sadat, and G. J. Lu, “Effective and efficient contourbased corner detectors,” Pattern Recognit., vol. 48, no. 7, pp. 2185–2197, 2015. [17] X. Zhang, Y. Qu, D. Yang, H. Wang, and J. Kymer, “Laplacian scale-space behavior of planar curve corners,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 11, pp. 2207–2217, Nov. 2015. [18] G. V. Pedrosa and C. A. Barcelos, “Anisotropic diffusion for effective shape corner point detection,” Pattern Recognit. Lett., vol. 31, no. 12, pp. 1658–1664, 2010. [19] X. Gao, F. Sattar, A. Quddus, and R. Venkateswarlu, “Multiscale contour corner detection based on local natural scale and wavelet transform,” Image Vis. Comput., vol. 25, no. 6, pp. 890–898, 2007. [20] S. Zhang, D. Yang, S. Huang, L. Tu, and X. Zhang, “Corner detection using Chebyshev fitting-based continuous curvature estimation,” Electron. Lett., vol. 51, no. 24, pp. 1988–1990, 2015. [21] I. C. Paula, F. N. S. Medeiros, F. N. Bezerra, and D. M. Ushizima, “Multiscale corner detection in planar shapes,” J. Math. Imag. Vis., vol. 45, no. 3, pp. 251–263, 2013.