Effective Shape-based Retrieval and Classification of Mammograms Joaquim C. Felipe1 Marcela X. Ribeiro2 Elaine P. M. Sousa2 Agma J. M. Traina2 Caetano Traina Jr2 1
2
Department of Physics and Mathematics University of São Paulo at Ribeirão Preto - Brazil
Department of Computer Science University of São Paulo at São Carlos - Brazil
[email protected]
{mxavier|parros|agma|caetano}@icmc.usp.br
ABSTRACT
1. INTRODUCTION
This paper presents a new approach to support Computer-aided Diagnosis (CAD) aiming at assisting the task of classification and similarity retrieval of mammographic mass lesions, based on shape content. We have tested classical algorithms for automatic segmentation of this kind of image, but usually they are not precise enough to generate accurate contours to allow lesion classification based on shape analyses. Thus, in this work, we have used Zernike moments for invariant pattern recognition within regions of interest (ROIs), without previous segmentation of images. A new data mining algorithm that generates statisticalbased association rules is used to identify representative features that discriminate the disease classes of images. In order to minimize the computational effort, an algorithm based on fractal theory is applied to reduce the dimension of feature vectors. Knearest neighbor retrieval was applied to a database containing images excerpted from previously classified digitalized mammograms presenting breast lesions. The results reveal that our approach allows fast and effective feature extraction and is robust and suitable for analyzing this kind of image.
The huge amount of digital images generated in hospitals and health care centers leads to the need of automatic storage and retrieval of them. Therefore, a PACS (Picture Archiving and Communication System) [12] should incorporate properties allowing to retrieve these images in a timely manner. Moreover, in order to effectively aid the physicians in their analysis and diagnosis, a retrieval task should bring images matching the criteria given by the specialists. Techniques for Content-Based Image Retrieval (CBIR) [9] deal with intrinsic visual features of images, usually color, shape and texture, to index and retrieve them. Thus, CBIR is based on processes that use similarity of features to compare images. Similarity queries performed over the pictorial content of the images deal much more with the inherent information of the data than when performed over a textual description associated to them. Textual annotations often imply two limitations: they are hard and time consuming to be produced and can present wide variations from one annotator to another. CBIR techniques allow performing automatic indexing and retrieval of images, in most cases without demanding the intervention of the user. In several applications, combining both textual and content information to perform image retrieval is an idea that deserves attention [1]. Even in this case, the development of efficient methods to deal with intrinsic features and to perform similarity comparisons is crucial to reach good results. Adding CBIR capabilities to PACS makes it more powerful to assist diagnosis, allowing easier and more efficient manipulation and organization of stored images [6] [10].
Categories and Subject Descriptors I.4.7 [Image Processing and Computer Vision]: Feature measurement – feature representation, moments. H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing – indexing methods. J.3 [Life and Medical Sciences]: Medical Information Systems.
General Terms Measurement, Documentation, Performance, Experimentation.
Keywords
Shape features have been widely employed to represent and analyze medical images [11], usually addressing specific contexts. A suitable shape algorithm must represent shapes in a low dimension, be invariant to rotation, translation and scale transformations, and retain relevant pathology information.
Content-based image retrieval, similarity measure, Zernike moments, association rules, dimensionality reduction, computeraided diagnosis.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC’06, April, 23-27, 2006, Dijon, France. Copyright 2006 ACM 1-59593-108-2/06/0004…$5.00.
Concerning the task of initial diagnostic hypothesis on breast tumor masses, a relevant feature to be analyzed in the image is the shape of the mass. Malignant tumors usually infiltrate the surrounding tissue, resulting in an irregular or hard-distinguishing contour, while benign ones present a smooth contour (Figure 1). The majority of the systems and methods dedicated to shape analysis demands previous image segmentation. Several of them use a semi-automatic process to segment the image, demanding user interaction. Even in fairly specialized domains, fully
250
automated segmentation causes several problems and is often not easy to execute [12]. In particular, segmentation of mammography images is especially difficult due to the superposing of the parenchyma over the nodules and masses, and thus the results are not trustful.
2.1 Zernike Moments Moments and functions of moments can be employed as pattern features to represent images. Such features capture global information about the image and do not require closed boundaries, as boundary-based methods (such as Fourier descriptors) do. Zernike polynomials provide a precise mathematical model that captures global shape while preserving enough information, by local harmonics. The equations related to representation of the moments can be found in [8]. The order of one moment corresponds to its capability to describe details (high spatial frequency components) of the pixel distribution of the image. Zhang and Lu [16] compared some shape descriptors that have been widely adopted for CBIR: Fourier descriptors, curvature scale space descriptors, Zernike moments and grid descriptors. The strengths and limitations of them were analyzed against properties such as affine invariance, robustness, compactness, low computation complexity and perceptual similarity measurement. Zernike moments have reached the best results in the overall evaluation.
Figure 1. Typical breast tumor masses: benign (top) and malignant (bottom) This paper presents a new approach to determine a minimal representative set of features based on shape content. Zernike moments calculated from the pixels values of images retain pattern information related to shape. We chose this technique because it can be applied to the whole region, without need of segmentation. In fact, our experimental tests have demonstrated that segmentation of breast lesion masses does not lead to precise contours. Therefore, in order to allow feature extraction without previous segmentation, the images are pre-processed (auto-leveled and filtered) and then the Zernike moments are extracted. We present a new algorithm (StARMiner – Statistical Association Rules Miner), which identifies the most relevant features to discriminate images into categories (benign and malignant features, for instance) through mining association rules. An experiment with images of tumoral masses of mammograms compares the accuracy of StARMiner with the wellknown C4.5 decision tree inducer. The most relevant features, in this case, are the moments that hold the shape details responsible for discriminating benign from malignant lesions.
2.2 Relevant Features Selection Mining association rules is a task that has been exhaustively explored in the last decades. The majority of the techniques addressing this task considers the use of categorical items. However, in CBIR context, the mining process deals with feature vectors of images, usually consisting of numerical and continuous values, and thus a suitable approach to find association rules in this context should consider quantitative data. StARMiner algorithm [14] extends the statistical rule mining techniques presented in [2] to find patterns on images. The goal of StARMiner is to implement statistical rules mining to find the features that best discriminate images into categorical classes. A rule has the format x Ai and it is identified only if the following conditions are satisfied. • The behavior of attribute Ai in images of category x must be different from its behavior in images of other categories. • The attribute Ai must present a uniform behavior in images of category x. The previous conditions are implemented in StARMiner algorithm incorporating restrictions of interest in the mining process. Let T be a database of medical images, x an image category, Tx ⊂ T the subset of images of category x and Ai an attribute. The restrictions of interest implemented in StARMiner algorithm are: 1) |AvgAi(Tx) – AvgAi(T-Tx)| ≥ mindif where: AvgAi(Z) is the arithmetic average of Ai values in the Z subset of images; mindif is the input parameter that indicates the minimum allowed difference between the average of Ai in images of category x and the average of Ai in the remaining images of the database. 2) Hypothesis test. H0 should be rejected with a confidence equal our greater than minconf. with: H0: AvgAi(Tx) = AvgAi(T-Tx) H1: AvgAi(Tx) ≠ AvgAi(T-Tx) where: minconf is the input parameter that indicates the minimum confidence to reject the H0 hypothesis.
In addition, a new technique based on the theory of fractals (FD-ASE algorithm) is used to reduce the dimensionality of feature vectors. The algorithm finds correlations between attributes and determines a set of independent ones, considering the contribution of each feature to the fractal dimension of the data set. The performance of our method is evaluated by an experiment where a 250 previously classified medical image database is used to verify the matching among the type of lesion of a query image and the retrieved ones. The remainder of this paper is structured as follows. Section 2 introduces background concepts and related techniques, while Section 3 presents the proposed method. Section 4 discusses the experiments and results reached with the developed system. Finally, Section 5 presents the conclusions.
2. TECHNIQUES This section briefly presents concepts related to the techniques supporting the method described in Section 3.
3) σAi(Tx) ≤ maxstd where: σAi(Tx) is the standard deviation of attribute Ai values in the subset of images Tx;
251
maxstd is the input parameter that indicates the maximum standard deviation of Ai values allowed in images of category x. The use of StARMiner to find rules in an image dataset allows finding rules involving attributes with high discrimination power, as the rule attributes have a particular and uniform behavior in images of a given category. The attributes that present similar behavior for all images in the database, regardless of the image category, do not contribute to the category discrimination and should be eliminated.
shape, based on the promising results reached by previous experiments using this descriptor (Section 2.1). The proposed method aims at determining a set of features (Zernike moments) which can be used as discriminators to classify images and, at the same time, as a minimal feature set to define a similarity measure to retrieve images based on shape resemblance. As illustrated in Figure 2, the method is applied in the following steps.
2.3 Attribute Correlations The “dimensionality curse” [5] degrades a variety of algorithms applied to high-dimensional complex data (images and videos, for instance), including indexing, matching, retrieval and analysis processes. Nevertheless, the existence of correlated attributes in high-dimensional datasets is very common, thus allowing the use of dimensionality reduction approaches to minimize the influence of the dimensionality curse.
Feature Extracto
Image CBIR Process
This section gives an overview of an attribute selection technique, named FD-ASE (Attribute Significance Estimator based on the Fractal Dimension), which finds a subset of relevant attributes of the dataset and also identifies groups of correlated attributes [Sousa, 2002 #130]. The technique is based on the concept of intrinsic dimension D of a dataset, that is, the dimensionality of the object represented by the data, regardless of the dimension of the space where it is embedded. For instance, the intrinsic dimension of a set of points disposed along a line is one, no matter if the set is embedded in any higher dimensional space. In essence, the FD-ASE finds groups of correlated attributes, identifying in each group a subset of base attributes, i.e., attributes to which every other attribute in the group is correlated.
FD-ASE
StARMin
Figure 2. Steps for feature characterization and selection Extraction of moments (Feature Extractor): An Image Training Set, consisting of images that represent each image class of the database, is submitted to a feature extractor, which generates a feature vector for each image, containing its Zernike moments.
The fundamental idea supporting the discovery of correlated attributes is to calculate the intrinsic dimension of incremental sequences (Si) of attributes, defined through forward attribute inclusion, and use the difference between D of consecutive sequences to identify the existence or absence of attribute correlations. To put it simply, consider a dataset A={a1, a2, … aE} composed of E attributes and a sequence of attributes Si ⊂ A, such that D(Si) denotes the intrinsic dimension of the dataset considering only the attributes in Si. An attribute ak ∈ (A - Si) is somehow correlated to at least one attribute of Si if adding ak to Si causes no meaningful change in D(Si). The particular attributes to which ak are correlated to are discovered by comparing the values D(Si ∪ ak) and D(Si ∪ ak - ai), ∀ ai ∈ Si, such that a significant difference means that attribute ai is neither correlated to ak nor to any other attribute in Si. A threshold in [0,1] is used to determine the significance of a change in the value of D, i.e., changes below the threshold are not considered meaningful. As a rule of thumb, lower thresholds are used to identify strong correlations, such as linear ones, while higher thresholds are used to identify weak correlations, such as non-polynomial.
In order to obtain invariance to translation, rotation and scale, the algorithm that calculates the moments considers the center of mass of the image and defines a radius around it that encloses the image, normalizing it to the interval [0 ,1]. Definition of relevant moments (StARMiner): For order 30, a set of 256 moments is generated for each image in Step 1. How many of them are relevant to discriminate the images into the initial classes? To answer this question, it is necessary to verify the relevance of each moment to the classification of the images. The set of feature vectors, together with the previously known class of each training image, is used by StARMiner to produce a set of statistical association rules, where the most relevant moments to discriminate the classes take part in the most confident rules in the resulting set of rules. These moments are then identified and the feature vector of each image is constructed using these values. Determination of representative moments (FD-ASE): Despite being relevant to image classification, the moments selected in Step 2 can present dependencies on each other. Thus, in this step the set of dependent moments are determined, as well as the amount of dependency that they carry.
3. METHOD FOR MINIMAL FEATURE SET DEFINITION Initially, we experimented on classical algorithms for automatic segmentation, but the results were not feasible to discriminate the images between benign and malignant, as can be seen in Section 4.1. Thus, we decided to use Zernike moments to represent image
The relevant moments of the image set (generated in Step 2) are submitted to the algorithm FD-ASE. It returns a set of representative attributes of the dataset, based on the contribution of each attribute to the dataset intrinsic dimension. Attributes that
252
do not significantly impact the intrinsic dimension are considered to be correlated to the others and thus can be discarded.
4.1 Tests with Segmentation Initially, we tried to segment the images, in order to detach the shapes of the masses. Two well-known methods were employed: the Watershed algorithm [4] and an improvement [3] of the EM/MPM algorithm [7], based on Markov Random Fields. Ten representative images of each class were segmented by each method for subsequent calculus of shape features (degree of circularity and degree of irregularity). Figure 4 shows some of these results.
Steps 2 and 3 are both necessary to reach the minimal set of representative features. If we only apply Step 2, we get the set of relevant moments for classification, but some of them may be dependent on others and thus not necessary to compare images. If we only apply Step 3, we get a set of independent moments, but not necessarily all of them are relevant to classify the images. At the end of this process, the amount of moments is reduced in one order of magnitude (from hundreds to dozens). Only these selected moments will compose the image feature vectors to be used to index and retrieve the images in a CBIR environment. This approach, besides reducing the computational cost to extract the moments, makes the searching time suitable to an operational environment, where the image database can be very large and the time spent with retrieval processes can be critical.
4. EXPERIMENTAL APPROACH A feature extraction tool – Zernike Extractor – was implemented to obtain the Zernike moments from images and to perform k-nearest neighbor queries. Figure 3 shows a screenshot of the tool. Figure 4. First row: original images (3 benign followed by 3 malignant masses); 2nd row: original images segmented by Watershed; 3rd row: original images segmented by EM/MPM From these results, we can see that these methods are not able to produce contours of the masses close enough to the original images, in order to allow to discriminate the classes. It occurs because this kind of image presents high occurrence of noise, due to the super positioning between the masses and the parenchyma.
4.2 Image Pre-processing In order to enhance image regions and execute the extraction of moments without a previous segmentation, the Zernike Extractor executes some basic pre-processing procedures: Auto-level: stretching of the maximum and minimum gray levels of the image to the maximum interval (0-255). This is important because the majority of mammography ROIs is dim. Reduction of the number of levels: the gray levels are reduced to 12. The complete interval [0, 255] is partitioned into 12 intervals and gray values that are in the same interval receive the same new value: new_gray = lold_gray * 12 / 256m
Figure 3. Zernike Extractor performing a k-nearest query A database consisting of 250 images was used to test and validate the proposed method. The images are ROIs (Regions of Interest) comprising tumoral masses, taken from mammographies. Although this amount of images is significantly smaller than the real size of a medical database, it is a common practice to use sample images during the task of evaluation and diagnosis in medical environments. Therefore, 250 images are enough to encompass a set of typical images from pathological and nonpathological groups, and to be used to choose the relevant moments and also to test the accuracy reached by the method given by a human specialist.
Median filter: this filter reduces white noise without affecting the object contours, making the image regions more homogenous. After getting the gray level interval of the image, these three processes can be applied at a unique scan on the grid of pixels of the image.
4.3 Set of Representative Moments
The images are classified either as benign or malignant, according to prior analysis done by radiologists and eventually confirmed by complementary exams. These kinds of lesions are tightly related to the image shape: benign masses have well-defined smooth contours, while the malignant ones have their contours spread over the mammary parenchyma.
The method described in Section 3 was executed over an image training set consisting of 90 images selected from the initial database. Initially, moments of order 30 were extracted, generating an amount of 256 moments for each image. The moments relevant to classification were defined by using two different algorithms, aiming at comparing their performance : the
253
well-known C4.5 decision tree inducer [13], and the StARMiner. C4.5 generated a decision tree, where 38 moments were identified as relevant. StARMiner generated a set of association rules, from which other 38 moments were identified as the most relevant.
maintains higher precision with values over than 0.7 for all recall values less than 0.5, while with C4.5 this value of recall is around 0.3. • The dimensionality reduction executed in Step 3 does not
In order to determine representative moments, the 38 relevant moments from both C4.5 and StARMiner were processed by FDASE algorithm. FD-ASE uses a threshold parameter to determine the limit of influence that makes a feature dependent or not on others. For this dataset, we set the threshold to 0.5.
affect the precision. The performance with 16 moments is the same as with 38 moments for StARMiner. The same occurs with C4.5. In order to verify that the final feature set is the minimal set that maintains the precision, one moment were removed from each feature set, resulting in 15 moments for StARMiner and 17 moments for C4.5. Figure 6 shows that precision suffers a critical decrease, proving that the feature sets that were generated in step 3 are the minimal representative ones.
The resulting analysis of influence had determined a set of 18 representative moments from the set of C4.5 38 relevant moments and another set of 16 representative moments from the set of StARMiner 38 relevant moments.
4.4 Results K-nearest neighbor queries were applied to the images from the database, taking randomly the query images and varying the values of k, for all feature sets generated in each step of the proposed method.
1.0 16 MOMENTS StARMiner + FD-ASE
0.8
precision
Figure 5 presents curves of Precision vs. Recall obtained with 256 moments (step 1), 38 moments from C4.5, 38 moments from StARMiner (step 2), 18 moments from FD-ASE after C4.5 and 16 moments from FD-ASE after StARMiner (step 3).
1.0
16 MOMENTS StARMing + FD-ASE
18 MOMENTS C4.5 + FD-ASE
15 MOMENTS StARMiner + FD-ASE
0.6 15 MOMENTS StARMiner + FD-ASE
0.4
17 MOMENTS C4.5 + FD-ASE
18 MOMENTS C.45 + FD-ASE
0.2 17 MOMENTS C4.5 + FD-ASE
38 MOMENTS StARMiner 256 MOMENTS
precision
0.8
0.0
16 MOMENTS StARMiner + FD-ASE
0.0
38 MOMENTS C4.5
38 MOMENTS C4.5
0.4
0.2
0.2
0.4
0.6
0.8
0.8
1.0
18 MOMENTS C4.5 + FD-ASE
4.5 Performance of StARMiner
16 MOMENTS StARMiner + FD-ASE
We performed a second set of experiments on another kind of images, aiming at confirming the capability of StARMiner to identify relevant attributes. The dataset consists of 704 medical images from different body regions, classified in eight categories according to the region and segmented using the improved EM/MPM algorithm [3]. For each image, 5 segments were generated and, for each segment, 6 features were extracted: the mass, the centroid x and y coordinates, the average gray level, the fractal dimension, and the linear coefficient used to estimate the fractal dimension. Thus, for each image, there was a 30-dimension feature vector.
0.0 0.0
0.6
Figure 6. Precision vs. Recall for minimal feature sets
38 MOMENTS StARMiner 18 MOMENTS C4.5 + FD-ASE
0.4
recall
0.6 256 MOMENTS
0.2
1.0
recall
Figure 5. Precision vs. Recall for different methods By analyzing the graph, we conclude that: • The results with 38 moments are better than with 256
moments for both StARMiner and C4.5. The dimensionality reduction (Step 2) has provided an important gain of precision (values around 0.8) in the region comprising recall values under 0.2. The regions of low recall are the most important into a PACS environment because k-nearest queries usually don't search for high values of k. These results testify that the dimensionality curse really damages the results: the irrelevant features disturb the clear influence of the relevant ones.
Just like in the first experiment, we executed StARMiner over the dataset using these 30 features and the algorithm identified a set of 15 relevant features, considering the task of matching the body regions represented by the images. Nearest neighbor queries were performed, concerning body regions too, using the 30 original features, the 15 relevant ones and a set of 14 features, obtained by randomly removing one from the 15 relevant set. Precision vs. Recall curves are shown in Figure 7.
• StARMiner reaches a better performance than C4.5. It
254
7. REFERENCES [1] Antani, S.K., et al., Developing a comprehensive system for content-based image retrieval of image and text from a national survey. SPIE Medical Imaging, 2005. 5748: p. 152-161. [2] Aumann, Y. and Y. Lindell. A Statistical Theory for Quantitative Association Rules. in Fifth SIGKDD Int. Conf. on Knowledge Discovery and Data Mining. 1999. San Diego, California. [3] Balan, A.G.R., et al. Fractal Analysis of Image Textures for Indexing and Retrieval by Content. in 18th IEEE Symposium on Computer-Based Medical Systems (CBMS). 2005. Dublin, Ireland. [4] Beucher, S. The watershed transformation applied to image segmentation. in 1Oth Pfefferkorn Conf. on Signal and Image Processing in Microscopy and Microanalysis. 1991. Cambridge, UK. [5] Beyer, K., et al. When is "Nearest Neighbor" Meaningful? in Int. Conf. on Database Theory (ICDT). 1999. Jerusalem, Israel. [6] Bueno, J.M., et al. How to Add Content-based Image Retrieval Capability in a PACS. in IEEE Int. Conf. on Computer Based Medical Systems - CBMS. 2002. Maribor, Slovenia. [7] Comer, M.L. and E.J. Delp, The EM/MPM Algorithm for Segmentation of Textured Images: Analysis and Further Experimental Results. IEEE Trans. Image Processing, 2000. 9(10): p. 1731-1744. [8] Khotanzad, A. and Y.H. Hong, Invariant Image Recognition by Zernike Moments. Transactions on Pattern Analysis and Machine Inteligence, 1990. 12(5): p. 489 - 497. [9] Lehmann, T.M., et al. Content-based Image Retrieval in Medical Applications for Picture Archiving and Communication Systems. in SPIE 2003. 2003. [10] Lehmann, T.M., B.B. Wein, and H. Greenspan. Integration of Content-based Image Retrieval to Picture Archiving and Communication Systems. in Medical Informatics Europe (MIE 2003). 2003. St Malo, France. [11] Mlsna, P.A. and N.M. Sirakov. Intelligent Shape Feature Extraction and Indexing for Efficient Content-Based Medical Image Retrieval. in IEEE Southwest Symposium on Image Analysis and Interpretation. 2004. Lake Tahoe, Nevada, USA. [12] Müller, H., et al., A review of content-based image retrieval systems in medical applications - clinical menefits and future directions. International Journal of Medical Informatics, 2004. 73: p. 1 - 23. [13] Quinlan, J.R., Induction of Decision Trees, in Machine Learning Magazine. 1986. p. 81-106. [14] Ribeiro, M.X., et al. Mining Statistical Association Rules to Select the Most Relevant Medical Image Features. in The First Int. Workshop on Mining Complex Data ( in conjunction with The Fifth IEEE International Conference on Data Mining). 2005. Houston, Texas, USA. [15] Sousa, E.P.M., et al. How to Use Fractal Dimension to Find Correlations between Attributes. in First Workshop on Fractals and Self-similarity in Data Mining (in conjunction with 8th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining). 2002. Alberta, Canada. [16] Zhang, D.S. and G. Lu. Content-Based Shape Retrieval Using Different Shape Descriptors: A Comparative Study. in IEEE Int. Conf. on Multimedia and Expo. 2001. Tokyo, Japan.
Figure 7: Precision vs. Recall using 30 features, 15 relevant features selected by StARMiner and 14 features obtained by randomly removing one of them These results are similar to those obtained in the first experiment. By analyzing the graphs in Figure 7 we can conclude that the results obtained with 15 features are quite similar to the results got with all 30 features. Hence, the dimensionality reduction of 50% achieved with StARMiner nearly maintains the same precision values considering the recall values under 0.7. In addition, the removal of 1 feature causes a general reduction in the precision, proving that the set of relevant features is the minimal one.
5. CONCLUSIONS Preliminary experiments show that classical methods of segmentation are not suitable to generate contours precise enough to classify breast lesions. A new method that combines classification and shape similarity retrieval of images has been presented. This method comprises techniques of Zernike moments to retrieve shape description, statistical association rules to identify relevant attributes and fractal theory to reduce dimensionality. The experimental approach applies a preprocessing procedure that discards the need of image segmentation. The results of searching a database consisting of ROIs of breast tumoral masses show that a significant reduction in the number of features – from 256 to 16 moments – can be applied, increasing the accuracy of the method. The precision values remain over 80% for recall values under 20%. The experimental results also show that the final set consisting of 16 moments is the minimal one that maintains the accuracy for classification tasks.
6. ACKNOWLEDGMENTS This research has been supported, in part, by the Brazilian National Research Council (CNPq) under grants 52.1685/98-6, 860.068/00-7 and 35.0852/94-4, and by the Sao Paulo State Research Foundation (FAPESP) under grant 04/02215-5.
255
