A multi-level ensemble-based system for detecting microaneurysms in fundus images B´alint Antal, Istv´an L´az´ar, Andr´as Hajdu
Zsolt T¨or¨ok, Adrienne Csutak
T¨unde Pet˝o
University of Debrecen, Faculty of Informatics 4010 Debrecen, POB 12, Hungary antal.balint,lazar.istvan
[email protected]
University of Debrecen, Medical and Health Science Center 4032 Debrecen, Nagyerdei Krt. 98
[email protected]
Moorfields Eye Hospital 162 City Road, EC1V 2PD London, UK
[email protected]
Abstract— In this paper, we present a complex approach to improve microaneurysm detection in color fundus images. Microaneurysms are early signs of diabetic retinopathy, so it is essential to detect these lesions accurately in an automatic screening system. The recommended detection of microaneurysms is realized through several levels. First, a specific combination of different preprocessing methods for candidate extractors is found. Then, we select candidates voted by a certain number of the candidate extractor algorithms. At all these levels, optimal adjustments are determined by corresponding simulated annealing algorithms. Finally, we classify the candidates with a machine-learning based approach considering an optimal feature vector selection determined by a feature subset selection algorithm. Our framework improves the positive likelihood ratio for the microaneurysms and outperforms both the state-of-theart individual candidate extractors and microaneurysm detectors in these measures.
I. I NTRODUCTION Diabetic retinopathy (DR) is the most common cause of blindness in the developed countries. Microaneurysms (MA) are early signs of this disease, so the detection of such lesions is essential in the screening process. They appear as small circular dark spots on the surface of the retina (see Figure 1).
Fig. 1.
has been established, but the actual grading required for diagnostics has been performed manually. Manual grading is slow and resource demanding, so several efforts have been made to establish an automatic computer-aided screening system [1]. However, the detection of microaneurysms is still an open issue. Thus, several recent works focus on this problem, including an online challenge for MA detectors [2]. The detection of MA is usually executed through candidate extraction followed by a final classification step with applying dedicated individual algorithms. In this paper, we propose a complex approach which generalize this basic idea by splitting it to more levels. Our main motivation is that we can perform further optimization at several levels in this way. Moreover, we consider more algorithms at these different functionality levels to be able to compose an ensemble-based system to suppress the errors of individual algorithms. The proposed process with its functionality levels is shown in Figure 2.
Fundus image containing a microaneurysm.
DR can be prevented and its progression can be slowed down if diagnosed and treated early. A proper medical protocol
Fig. 2. Steps of microaneurysm detection using a multi-level ensemble-based system.
That is, first, we extract MA candidates from fundus images after preprocessing. Since different candidate extractors may react differently to preprocessing methods, we include several preprocessing algorithms and propose an optimal selection of them for each of the candidate extractors involved in the system. The optimal selection of preprocessing methods and candidate extractor algorithms is found by simulated annealing targeting maximum true positive/false positive ratio. For MA candidate extraction, we select three state-of-the-art approaches and include a novel one, as well. In the next two steps we emerge close candidates for each of the individual candidate extractors, and remove some false candidates to meet clinical criteria with following current literature recommendations, respectively. Then, we compose an ensemble of the individual candidate extractors, and apply a majority voting scheme to further improve the true positives/false positive ratio. The number of required votes and the set of the corresponding candidate extractors is determined by simulated annealing again. Since till this point MA candidates are represented at pixel level, in the next step we perform region growing to be able to classify candidate MA regions with machine learning-based algorithms. For the best classification results, we compose an ensemble from the feature vectors recommended for classification for all the individual candidate extractors and select an optimal subset from it. The rest of the paper is organized as follows. In section II we present the preprocessing methods and candidate extractors of the system, and also explain how their optimal combination is found. Section III exhibits how some certain candidates are removed based on some clinical requirements. The majority voting scheme of individual candidate extractors together with an algorithm for its optimization is described in section IV. In section V, we explain how a machine learning-based classification considering an optimal subset of feature vectors is applied to the remaining MA candidates. The performance of the proposed system is also compared with that of the individual algorithms at the corresponding sections. Finally, some conclusions are drawn in section VI. II. P REPROCESSING AND CANDIDATE EXTRACTION Preprocessing of the original fundus images is usually recommended before performing candidate extraction. Individual candidate extractors integrates specific preprocessing algorithms. In this section we explain how an optimal combination of preprocessing algorithms and candidate extractors can be found. A. Preprocessing methods The use of the selected preprocessing methods aims to enhance the accuracy of the microaneurysm detection in different ways. Namely, our experiments showed that Contrast Limited Adaptive Histogram Equalization is very effective in emphasizing locally salient values, but also produces noise. The contrast enhancement technique by Walter and Klein resulting in a grayscale image with a smooth background and emphasized salient parts, while the vessel removal and
extrapolation method aims to reduce the false positives which caused by the similar appearance of vessel parts and microaneurysms. Our results showed that, with the use of the following preprocessing methods we can increase the accuracy of the individual candidate extractors: 1) Contrast Limited Adaptive Histogram Equalization (CLAHE): Contrast Limited Adaptive Histogram Equalization [3] is a common preprocessing method for medical imaging, because it is very effective in making the interesting parts more visible. It is based on local histogram equalization of disjoint regions extracted from the image. To eliminate the boundaries between the regions, a bilinear interpolation is also applied. 2) Walter-Klein contrast enhancement (WK): This preprocessing algorithm is proposed in [4]. It aims to enhance the contrast on fundus images by applying a gray level transformation. Walter et al. defined the local contrast enhancement operator in the following way: 1 ) r 2 (umax −umin · (t − tmin ) + umin , t ≤ µf , r f −tmin ) u = − (µ 1 (u −u ) r max min 2 · (t − tmax ) + umax , t ≥ µf , r (µf −tmax )
where {tmin , . . . tmax } are the intensity values of the grayscale image, {umin , . . . umax } are the intensity values of the enhanced image, µf is the mean value of the grayscale image and r ∈ R. 3) Vessel removal and extrapolation: Most of the false positives during microaneurysm detection caused by the similar appearance of a few parts of the vessel system. Based on the idea proposed in [5], we investigate the effect of processing images with the complete vessel system removed. To fill in the holes caused by the removal, we extrapolate the missing parts. B. Candidate extraction Candidate extraction is an effort to reduce the number of objects in an image for further analysis by excluding regions which do not have similar characteristics to microaneurysms. Individual approaches define their own measurement for similarity to extract MA candidates. In this section, we provide a brief overview of currently popular candidate extractors that will be considered later on in the ensemble-based system. Namely, we cite Walter et al. [6], the Spencer-Frame method [7] [8] in its original form, and a slightly modified version of [9]. A new approach is also involved, whose development was partly motivated to improve the MA candidate extraction efficiency of the proposed ensemble-based system. As we will see later on, these methods sufficiently diverse to form a successful ensemble. 1) Walter et al.: The approach proposed in [6] is a mathematical morphology based method, which recommends contrast enhancement and shade correction as preprocessing steps. Candidate extraction is then accomplished by grayscale diameter closing. 2) Spencer-Frame: This approach is one of the most popular candidate extractors, originally proposed by Spencer [7] and Frame [8]. The algorithm uses shade correction as
preprocessing. The actual candidate extraction is accomplished by subtracting the maximum of multiple morphological tophat transformation. The resulting image is binarized after applying a Gaussian filter. Since the candidates are not a good representation for the actual lesions, a region growing step is also applied. 3) Circular Hough-transformation based: Based on the idea presented in [9], we established an approach based on the detection of small circular spots in the image. The obvious choice for this procedure is to use circular Houghtransformation, which is preceded by contrast limited adaptive histogram equalization for preprocessing. 4) Lazar et al.: The green channel of the image is inverted and smoothened with a Gaussian filter. A set of scan lines with equidistantly sampled tangents between -90 ◦ and +90 ◦ is fixed. For each direction the intensity values along the scan lines are recorded in a one dimensional array, and the scan lines are shifted vertically and horizontally to process every image pixel of the image. On each intensity profile, the heights of the peaks, and their local maximum positions are used for an adaptive thresholding. The resulting foreground indices of the thresholding process are transformed back to two dimensional coordinates, and stored in a map that records the number of foreground pixels of different directions corresponding to every position of the image. The maximal value for each position equals the number of different directions used for the scanning process. This map is smoothened with an averaging kernel and a hysteresis thresholding procedure is applied. The resulting components are filtered based on their size. For more details, see [10]. C. Optimal combination of preprocessing and candidate extraction The proposed framework aims to find an optimal combination of preprocessing methods and candidate extractors. For this task, we generate the results for each candidate extractors using the selected preprocessing method. We also consider the output generated for the original dataset. That is, we combine the results of candidate extractors applied on preprocessed images, and we also include the results on the non-preprocessed images, too. Then, we search for the optimal combination with simulated annealing. Simulated annealing [11] is a widely used global optimization method. This approach is inspired by the annealing in metallurgy. It is effective for large search space problems by using random sampling to avoid stuck in a local minimum. For the optimization, we use the following energy function to be minimized: TP , (1) E=− FP where TP stands for the number of the true, while FP stands for that of false positive candidates, respectively. To minimize the target energy E by simulated annealing, each element of the search space S relies on a combination of preprocessing methods and candidate extractors, and consists of a set of candidates which resulted by the union of the
outputs of the algorithm using the corresponding approaches. The proposed combination can be described formally by the following algorithm: Finding optimal combination of preprocessing methods and candidate extractors 1) Let T ∈ R be an initial temperature, Tmin a minimal temperature, 0 ≤ q ≤ 1, q ∈ R the temperature change, S = P {Rce,pm } the search space, where R is the result of the candidate extractor ce using the preprocessing method pm and P {X} is the power set of X. 2) Choose x ∈ S randomly and let e = E (x), S = S−{x}. 3) Choose xi ∈ S randomly and let ei = E (xi ), S = S − {xi }. 4) If T < Tmin , stop. 5) If ei < e, then x = xi , e = ei and T = T · q. Go to step 4. 6) Choose a random number r ∈ R. If accept (e, ei , T, r) = true, then x = xi , e = ei , where ( i > r, true, if exp e−e T accept (e, ei , T, r) = f alse, otherwise. 7) T = T · q. Go to step 4. Currently, we consider three preprocessing methods and four candidate extractors, but with simulated annealing the system can be easily extended to use more methods in the future. D. Results on combining preprocessing and candidate extraction We have tested our approach on 50 images selected from the Retinopathy Online Challenge (ROC) database [2]. Currently, it is the only publicly available fundus image database dedicated to measure the accuracy of microaneurysm detectors. In Table I, we give the number of true positives and the false ones found by the individual algorithms with considering only one preprocessing method. Walter
Spencer
Hough
Lazar
TP FP E TP FP E TP FP E TP FP E
Original 120 6748 0.034 45 1632 0.080 6 3090 1.041 113 771 0.017
WK 202 27811 0.024 29 1526 0.137 41 5967 0.121 116 7469 0.036
CLAHE 199 15173 0.022 60 3063 0.066 125 14467 0.038 191 20888 0.025
Vessel 154 8801 0.026 31 1342 0.122 3 1221 2.003 69 539 0.030
TABLE I P ERFORMANCE OF THE CANDIDATE EXTRACTORS USING A SINGLE PREPROCESSING METHOD .
Table II shows the optimal selections of preprocessing methods for all the individual candidate extraction algorithms. As we can see, all but one candidate extractors earned higher performance after combination. That is, the method by Lazar et al. provided better results using the non-preprocessed images. The highest TP increase was achieved by the Spencer-Frame algorithm. Walter Preprocessing: TP: FP: E: Spencer Preprocessing: TP: FP: E: Hough Preprocessing: TP: FP: E: Lazar Preprocessing: TP: FP: E:
WK, CLAHE, Vessel 254 33695 0.019 all 124 6686 0.032 WK, CLAHE, Vessel 151 18429 0.031 Original 113 771 0.017
TABLE II P ERFORMANCE OF THE CANDIDATE EXTRACTORS COMBINED WITH PREPROCESSING METHODS .
III. R EMOVING FALSE CANDIDATES Before letting the individual candidate extractor algorithms vote, we must ensure that there are no candidates too close to each other within the output of an individual algorithm. This issue is addressed by merging them. It is also important to remove any candidates falling on the vessel system. For this purpose, we have detected the vascular system with the algorithm proposed in [12] and removed the candidates falling on a vessel. Though these two steps are rather intuitive ones, they have very large importance, since their application decreases dramatically the number of false positive MA candidates. IV. VOTING ON THE CANDIDATES Each individual algorithm produces an initial set of microaneurysm candidates. Then, we establish a set of final candidates, where these candidates are voted by at least n ≥ 2 candidate extractors. The voting procedure has the following steps:
3) If sum ≥ n, then add the centroid of the candidates found by step 2 to the final set. 4) Repeat the procedure until all the initial sets become empty. The number of required votes depends on the actual algorithms. Thus, we use a dynamic optimization approach to determine this number. A. An optimal voting scheme using simulated annealing The proposed framework aims to find an optimal voting scheme for candidate extractors. This voting scheme determines a subset of the candidate extractors and the number of required votes. Since this problem induces a large search space, we choose simulated annealing again to find an optimal solution with targeting the same energy function E defined in (1). At this level, each element of the search space S consists of the results sets of a set of candidate extractors {ce1 , . . . ceL } and a required number of votes v (1 ≤ v ≤ L). Each combination occurs in S only once. The proposed combination can be described formally by the following algorithm: Finding optimal voting of candidate extractors 1) Let T ∈ R be an initial temperature, Tmin a minimal temperature, 0 ≤ q ≤ 1, q ∈ R the temperature change, S = P (({Rce }, v)) the search space, where Rce is the result of the candidate extractor ce, v is the number of required votes and P (X) is the power set of X. 2) Choose x ∈ S randomly and let e = E (x), S = S−{x}. 3) Choose xi ∈ S randomly and let ei = E (xi ), S = S − {xi }. 4) If T < Tmin or S = ∅, then stop. 5) If ei < e, then x = xi , e = ei and T = T · q, . Go to step 4. 6) Choose a random number r ∈ R. If accept (e, ei , T, r) = true, then x = xi , e = ei , where ( i true, if exp e−e > r, T accept (e, ei , T, r) = f alse, otherwise. 7) Let T = T · q. Go to step 4. Currently, we consider four candidate extractors (that is, L = 4 in the algorithm above), but with the use of simulated annealing it can be easily extended to more methods in the future.
Majority voting of candidate extractors
B. Results on voting
1) For each candidate C provided by one of the algorithms, check whether there is another candidate detected by another algorithm within a distance r ∈ R from C. 2) Let sum be the number of candidates satisfying the above proximity criterion and remove all these candidates from their respective initial sets.
We have tested our approach on the same database [2] discussed before. As a result of dynamical selections of algorithms, our ensemble system formed by the Walter, Spencer-Frame and Lazar candidate extractors requiring 3 votes to accept. Our results shows that the ensemble system provides substantially better TP / FP ratio
than the individual algorithms. The comparison of our approach with the individual extractors can be seen in Table III. Figure 3 shows a sample result, where all microaneurysm found by the ensemble with no false predictions.
Fig. 3.
Microaneurysms detected by the ensemble.
Algorithm Walter Spencer-Frame Hough Lazar Ensemble
Total 2831 5821 15664 868 165
TP 110 115 94 109 50
FP 2721 5706 15570 759 115
TP / FP 0.040 0.020 0.006 0.144 0.435
TABLE III C OMPARISON OF MICROANEURYSM CANDIDATE EXTRACTORS .
V. C ANDIDATE CLASSIFICATION To improve the TP / FP ratio we can use a classification step, which is based on certain unique features of microaneurysms. There are many literature recommendations for this task. In this section, we list these features. In addition, we select an optimal subset of them. Before the classification step, we must obtain a proper representation of the microaneurysms by applying region growing on the voted candidate centroids [13]. We have selected features based on [6], [13], [14], [15] and [10]. The complete list of the features can be seen in Table IV. To select the optimal subset of features, we used a forward feature subset selection technique [16]. The selected subset is showed in bold in Table IV. We used a KNN classifier with the selected features and successfully improved the TP / FP ratio to 0,467 with 48 TP and 103 FP, which is outperforming
the microaneurysm detector proposed in [17]. See Table V for detailed results. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51.
Feature area perimeter aspect ratio circularity total green intensity total shade corrected intensity average green intensity average shade corrected intensity normalized green intensity normalized average shade corrected intensity normalized average green intensity normalized average shade corrected intensity compactness average Gauss filter response, s = 1 average Gauss filter response, s = 2 average Gauss filter response, s = 4 average Gauss filter response, s = 8 standard deviation Gauss filter response, s = 1 standard deviation Gauss filter response, s = 2 standard deviation Gauss filter response, s = 4 standard deviation Gauss filter response, s = 8 maximum correlation coefficient minimum correlation coefficient average correlation coefficient major axis length minor axis length maximum top-hat mean top hat dynamic outer mean outer standard deviation inner mean inner standard deviation outer range inner range gray level contrast color contrast half-range area mean red intensity mean blue intensity standard deviation green intensity standard deviation blue intensity sqrt (25. * 26.) eccentricity paraboloid depth mean of squared gradient 45. / 43. 46. / sqrt(45.) ratio of full peaks to the number of scan directions on intersecting scan lines in the specific position maximal of width of full peaks on intersecting scan lines average peak height
Origin [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [13] [6] [6] [6] [6] [6] [6] [6] [6] [6] [6] [6] [6] [14] [14] [14] [14] [15] [15] [15] [15] [15] [15] [10] [10] [10]
TABLE IV S ELECTED FEATURES FOR CLASSIFICATION
VI. C ONCLUSION In this paper, we have presented a novel approach for microaneurysm detection that realizes ensemble-based solutions at several levels. In this way, we can take advantage of the strengths of several individual algorithms by organizing them into a complex system. The optimal adjustment at the different functionality levels are found by simulated annealing. As comparative results have shown, the proposed system
Algorithm Mizutani Ensemble Ensemble (classified)
Total 225 165 151
TP 20 50 48
FP 205 115 103
TP / FP 0.097 0.435 0.467
TABLE V C OMPARISON OF THE ENSEMBLE - BASED SYSTEM WITH AN INDIVIDUAL MICROANEURYSM DETECTOR .
outperforms the individual algorithms with respect to the ratio of the true and false microaneurysm candidates. Moreover, there is still room to improve the system further with e.g. including efficient future individual algorithms, extend the ensemble to other levels, or finding parameter settings for the individual algorithms that are optimal for the performance of the whole system. ACKNOWLEDGMENT This work was supported in part by the J´anos Bolyai grant of the Hungarian Academy of Sciences, and by the TECH082 project DRSCREEN - Developing a computer based image processing system for diabetic retinopathy screening of the National Office for Research and Technology of Hungary (contract no.: OM-00194/2008, OM-00195/2008, OM00196/2008). R EFERENCES [1] M. Abramoff, M. Niemeijer, M.S.A. Suttorp-Schulten, M. A. Viergever, S. R. Russel, and B. van Ginneken, “Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes,” Diabetes Care, vol. 31, pp. 193–198, February 2008. [2] M. Niemeijer, B. van Ginneken, M.J. Cree, A. Mizutani, G. Quellec, C.I. Sanchez, B. Zhang, R. Hornero, M. Lamard, C. Muramatsu, X. Wu, G. Cazuguel, J. You, A. Mayo, Q. Li, Y. Hatanaka, B. Cochener, C. Roux, F. Karray, M. Garcia, H. Fujita, and M.D. Abramoff, “Retinopathy online challenge: Automatic detection of microaneurysms in digital color fundus photographs,” IEEE Transactions on Medical Imaging, vol. 29, no. 1, pp. 185–195, 2010. [3] K. Zuiderveld, “Contrast limited adaptive histogram equalization,” Graphics gems, vol. IV, pp. 474–485, 1994.
[4] T. Walter and J. Klein, “Automatic detection of microaneyrysms in color fundus images of the human retina by means of the bounding box closing,” Lecture Notes in Computer Science, vol. 2526, pp. 210–220, 2002. [5] Saiprasad Ravishankar, Arpit Jain, and Anurag Mittal, “Automated feature extraction for early detection of diabetic retinopathy in fundus images.,” in CVPR. 2009, pp. 210–217, IEEE. [6] T. Walter, P. Massin, A. Arginay, R. Ordonez, C. Jeulin, and J. C. Klein, “Automatic detection of microaneurysms in color fundus images,” Medical Image Analysis, vol. 11, pp. 555–566, 2007. [7] T. Spencer, J. A. Olson, K. C. McHardy, P. F. Sharp, and J. V. Forrester, “An image-processing strategy for the segmentation and quantification of microaneurysms in fluorescein angiograms of the ocular fundus,” Computers and Biomedical Research, vol. 29, pp. 284302, May 1996. [8] A. J. Frame, P. E. Undrill, M. J. Cree, J. A. Olson, K. C. McHardy, P. F. Sharp, and J. Forrester, “A comparison of computer based classification methods applied to the detection of microaneurysms in ophthalmic fluorescein angiograms,” Computers in Biology and Medicine, vol. 28, pp. 225238, May 1998. [9] S. Abdelazeem, “Microaneurysm detection using vessels removal and circular hough transform,” Proceedings of the Nineteenth National Radio Science Conference, pp. 421 – 426, 2002. [10] Istvan Lazar, Andras Hajdu, and Rashid Jalal Quareshi, “Retinal microaneurysm detection based on intensity profile analysis,” 8th International Conference on Applied Informatics, 2010. [11] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, pp. 671–680, May 13, 1983. [12] J. Staal, M. D. Abramoff, M. Niemeijer, M. A. Viergever, and B. van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” IEEE Transactions on Medical Imaging, vol. 23, pp. 501 – 509, 2004. [13] B. Zhang, X. Wu, J. You, Q. Li, and F. Karray, “Detection of microaneurysms using multi-scale correlation coefficients,” Pattern Recogn., vol. 43, no. 6, pp. 2237–2248, 2010. [14] M. J. Cree, J. A. Olson, K. C. McHardy, P. F. Sharp, and J. V. Forrester, “A fully automated comparative microaneurysm digital detection system,” Eye, vol. 11, pp. 622–8, 1997. [15] A. D. Fleming, S. Philip, and K. A. Goatman, “Automated microaneurysm detection using local contrast normalization and local vessel detection,” IEEE Transactions on Medical Imaging, vol. 25(9), pp. 1223–1232, 2006. [16] N. Sondberg-madsen, C. Thomsen, and J. M. Pena, “Unsupervised feature subset selection,” in In Proceedings of the Workshop on Probabilistic Graphical Models for Classification, 2003, pp. 71–82. [17] A. Mizutani, C. Muramatsua, Y. Hatanakab, S. Suemoria, T. Haraa, and H. Fujita, “Automated microaneurysm detection method based on double-ring filter in retinal fundus images,” Medical Imaging 2009: Computer-Aided Diagnosis, Proceedings of SPIE, vol. 7260, 7260, pp. 1N1 – 1N8, 2009.
