Comparative evaluation of support vector machine classification for

Comparative evaluation of support vector machine classification for computer aided detection of breast masses in mammography J. M. Lesniak 1 , R.Hupse2 , R. Blanc1 , N. Karssemeijer2 and G. Szekely1 1

Computer Vision Laboratory, Sternwartstrasse 7, ETH Z¨ urich, 8092 Z¨ urich, Switzerland 2 Department of Radiology, Radboud University Medical Centre, Nijmegen, 6525GA, The Netherlands E-mail: [email protected] Abstract. False positive (FP) marks represent an obstacle for effective use of computer-aided detection (CADe) of breast masses in mammography. Typically, the problem can be approached either by developing more discriminative features or by employing different classifier designs. In this paper, the usage of support vector machine (SVM) classification for FP reduction in CADe is investigated, presenting a systematic quantitative evaluation against neural networks (NNet), k-nearest neighbor classification (k-NN), linear discriminant analysis (LDA) and random forests (RF). A large database of 2516 film mammography examinations and 73 input features was used to evaluate the classifiers for their performance on correctly diagnosed exams as well as false negatives. Further, classifier robustness was investigated using varying training data and feature sets as input. The evaluation was based on the mean exam sensitivity in 0.05 to 1 false positives on the free-response receiver operating characteristic curve (FROC), incorporated into a 10-fold cross validation framework. It was found that SVM classification using a Gaussian kernel offered significantly increased detection performance (P = 0.0002) compared to the reference methods. Varying training data and input features, SVMs showed improved exploitation of large feature sets. It is concluded that with SVM-based CADe a significant reduction of false positives is possible outperforming other state-of-the-art approaches for breast mass CADe.

Keywords: Computer aided detection, mammography, breast imaging, classification, support vector machine, neural network, linear discriminant analysis, random forests, nearest neighbors

Comparative evaluation of SVM for CADe of breast masses in mammography

2

1. Introduction In breast cancer screening, the primary goal is early identification of the disease in order to allow for effective treatment. Radiologists aim at detection of mammographic signs such as calcifications, asymmetries, architectural distortions or masses, whereas the vast majority of screening examinations are normal. Computer aided detection (CADe) systems are designed to provide supplemental decision support to the radiologist for more effective cancer detection, aiming at high sensitivity of the CADe systems. Particularly the computerized detection of masses is challenging, as there is a great variation of appearances and superimposed tissue may mimic malignancies, leading to false positive (FP) detections which are among the obstacles for effective clinical use of CADe. Consequently the reduction of FPs together with the improvement of detection sensitivity is an active area of research (Nishikawa et al. 2006) (Tang et al. 2009). A CADe system usually consists of an initial detection phase where regions of interest are segmented, a feature extraction stage where a variety of region descriptors are extracted and a final interpretation stage, in which a classifier assigns a malignancy score to each of the previously identified regions. Finally, the classifier output scores are thresholded to determine which regions shall be flagged for inspection. Typically a low threshold is applied, admitting a large number of false positives to avoid that the system misses abnormalities (Hupse & Karssemeijer 2010). One approach to false positive reduction is the construction and selection of good features (Cheng et al. 2006). However, the choice and configuration of the classification method may have a considerable impact on detection performance, as it serves as the final decision maker in a CADe system. Consequently, stable generalization on unseen data is determined by the interplay of input features and classification model. To this end, the practical performance of the classification stage is affected by the learning principle, robustness against insignificant features or outliers, adaptation to data characteristics such as class imbalances and the effective strategy for model selection and complexity control. Among the most popular classification methods for mass CADe are neural networks, k-nearest neighbors, Fisher’s linear discriminant analysis, Bayesian networks or decision trees (Cheng et al. 2006) (Elter & Horsch 2009). Further, support vector machines (SVM) have proven to perform favorably on different classification problems (Burges 1998). In CADe, SVMs have been investigated on a number of practical problems including microcalcification detection and discrimination, featureless mass detection, identification of architectural distortions, significance analysis of breast mass features or multiple view fusion for mass detection (El-Naqa et al. 2002) (Campanini et al. 2004) (Guo et al. 2005) (Wei et al. 2005) (Mavroforakis et al. 2006) (Wei et al. 2011). Their attractive properties for CADe include a well founded framework in statistical learning theory, explicit control of misclassification cost to account for class imbalances and learning of flexible nonlinear models from data. In this work, an SVM-based system was investigated for its FP reduction

Comparative evaluation of SVM for CADe of breast masses in mammography

3

capabilities in breast mass CADe. The purpose of this study was to present a systematic quantitative evaluation of SVM-CADe against several other popular classification methods, extending our previous investigations of classifiers for mass detection (Lesniak et al. 2011). For comparison, a large database of 2516 image examinations was available, including 363 correctly diagnosed as well as 204 false negative exams. The resulting training data comprised 7544 film mammograms with 921 cancer annotations and 73 region descriptors. The reference methods were Fisher’s linear discriminant analysis (LDA), a three layer backpropagation neural network (NNet), k-nearest neighbors (kNN) and random forests (RF). Performances were assessed by calculating the mean true positive fraction on the free response operating characteristic (FROC) curve, focusing on low-FP rates in a 10-fold crossvalidation framework. The evaluation of SVM CADe and its reference methods was twofold. First, CADe systems based on each classifier were trained and their FROC was analyzed on the complete database as well as subsets with correctly diagnosed cancers and false negative image examinations. Second, the robustness of the methods was compared by varying the input features based on RF variable importance ranking and subsampled training datasets. 2. Materials and Methods 2.1. Database The database consisted of scanned film mammography images of breast masses taken from 1539 patients participating in breast cancer screening. These comprised 485 cases with biopsy proven cancer masses. Among the cases were also patients which exhibited calcifications and benign masses, which were labeled as negative instances. A majority of patients had temporal imaging with up to two exams included, comprising 268 exams with initially missed cancers. One exam consisted of either two or four images, summing up to 10064 images in total. Preceding mass detection, the raw scanned images were preprocessed. This included downsampling the images to a resolution of 200 µm, segmentation of the skin outline, fading of the pectoral muscle and enhancement of the peripheral area of the breast. Next, mass candidate regions were segmented by using a set of five pixel-based features measuring stellate structures and gradients in the surrounding areas of each pixel. These pixel-based features served as input to a set of neural network classifiers, which assigned a “mass likelihood“ to each pixel in the images. The resulting likelihood map yielded seed points for dynamic programming based segmentation of mass candidate regions (Timp & Karssemeijer 2004). For each segmented mass candidate region, in total 73 region-based features were derived. The features comprised mass-likelihoods resulting from the initial detection stage, normal tissue context features, descriptors for spiculation, gradient measures, linear texture, contrast, morphology and border as well as density estimations, grey level statistics and location measures (Hupse & Karssemeijer 2009) (te Brake et al. 2000).

Comparative evaluation of SVM for CADe of breast masses in mammography

4

Table 1. Overview of region-based descriptors obtained from feature extraction. In total, 10 categories with 73 features in total were available. Feature Set

Description

N

Likelihood Context Spiculation Gradient Linear Texture Contrast Morphology, Border Density Grey Level Location

Mass likelihood derived from pixel-based features Normal tissue reference regions Orientations towards center of region Grey level gradient around region center Linear texture inside and outside of segmentation Contrast between inside and outside regions Border continuity, acutance, compactness, size Fatty vs. glandular tissue Variance, intensity and isodensity of grey levels Location, dist. to nipple, skin and pectoral muscle

9 12 7 4 5 6 9 6 5 10

Σ

73

Each group of features also comprised correlated descriptors, i.e. variants of features which were transformed (e.g. normalized) or related to special areas of the segmentation such as its outer band. Correlated features were included as these bear the potential to augment classification (Guyon & Elisseeff 2003). In Table 1, a detailed overview of the feature sets is presented. Further, an independent dataset of 1546 normal images for the normalization of data was available. 2.2. Support Vector Machine Classification Support Vector Machines (SVM) are a kernel-based learning method rooted in structural risk minimization (Cortes & Vapnik 1995). For labeled training data of the form (xi , yi ), i ∈ {1, . . . , l}, where xi is an n-dimensional feature vector and y ∈ {−1, 1} the labels, a decision function

f (x) = (hw, φ(xi )i + b)

(1)

is found with weight vector w and bias value b representing a separating hyperplane. By projecting the data using a mapping φ(x), nonlinear decision boundaries in the input data space can be obtained. The separating hyperplane is found by maximizing distances to its closest data points, embedding it in a large margin which is defined by support vectors. Finding the hyperplane while maximizing the margin is formulated as the following optimization problem:

min w,b,ξ

X X 1 hw, wi + C + ξi + C − ξi 2 i∈+ i∈−

subject to yi (hw, φ(xi )i + b) ≥ 1 − ξi , ξi ≥ 0.

(2)

Comparative evaluation of SVM for CADe of breast masses in mammography

5

For inseparable data, the soft margin SVM formulation allows support vectors inside the margin, which are penalized by slack variables ξi (see Equation 2). The amount of margin and misclassification errors during training is bound and the tradeoff between the amount of slack and margin maximization is controlled by a penalty term C. Unequal misclassification costs can be implemented by different penalties C + and C − for the negative and positive class (Ben-Hur & Weston 2010). In this way class imbalances, which are often encountered in CADe, can be reflected during training. The optimization problem can be efficiently solved using an equivalent formulation to Equation 2 using Lagrangian multipliers. In this formulation, data is represented exclusively by dot products, which can be replaced with a kernel function K(x, x0 ) = φ(x)T φ(x0 ), allowing large margin separation in the kernel space. The choice of the kernel function determines the mapping φ(x) of the input data into a higher dimensional feature space, in which the linear separating hyperplane is found (see also Equation 1). In this work, nonlinear classification using the Gaussian kernel

K(x, xi ) = exp(−

k x − xi k2 ) σ

(3)

was investigated. It comprises one free parameter, the kernel width σ, which controls the amount of local influence of support vectors on the decision boundary. Practically, large margin classification is known to be influenced by different scales of features thus standardization is recommended (Ben-Hur & Weston 2010). SVMs have shown to be capable of handling high dimensional data well, as its performance bounds on classification error do not explicitly depend on the dimensionality of the input data (Cristianini & Shawe-Taylor 2000). 2.3. Reference Classification Methods 2.3.1. Linear Discriminant Analysis. Fisher’s linear discriminant analysis (LDA) is a discriminative method which is commonly used in CAD for the purpose of feature selection and classification (Cheng et al. 2006). The method finds a linear discriminant function wT x in a supervised fashion by projecting the features such that kµ ˆ1 − µ ˆ2 k2 R(w) = σ ˆ12 − σ ˆ22

(4)

is maximized, i.e. the distance between class means shall be maximal and the within class scatter shall be minimal (with µ ˆi and σ ˆi representing empirical class means and scatter). The only assumption about data is that the within-class scatter matrix is non-singular, and for Gaussian distributions it can be shown that the linear decision boundary obtained is the same as for the optimal Bayes classifier. Nevertheless, if posterior distributions are highly nonlinear in x, suboptimal results might be obtained even on linearly separable data (Cherkassky & Mulier 2007).

Comparative evaluation of SVM for CADe of breast masses in mammography

6

2.3.2. K-Nearest Neighbors. In k-nearest neighbors (k-NN) classification, the decision boundary is constructed locally in an area around the query sample, minimizing the local misclassification risk 1X R(w) = yi − wKk (xi , x0 ), k i=1 n

(5)

with Kk = 1 if the estimation point xi is among the k nearest neighbors of x0 . The risk is minimized by assigning w the label of the majority class in the neighborhood and the Euclidean distance was used as a proximity measure in this work. K-NN are sensitive to scaling of the input variables and thus may suffer from the curse of dimensionality, nevertheless perform well on a range of practical problems including breast CAD (Elter & Horsch 2009). 2.3.3. Neural Network. As a widespread approach in mammography mass CAD Neural networks (NNet) exist in a variety of configurations (Tang et al. 2009). Multilayer neural networks comprise a set of interconnected nodes organized into several layers, i.e. an input, one or more hidden, and an output layer, whereas the nodes represent an activation function which receives weighted inputs to generate an output score. Three layer neural networks are used in a large variety of applications as these are able to model arbitrary functions, leaving the control of complexity as an important aspect in model selection to obtain optimal performance (Duda et al. 2001). In this work, a reference setup using a three layer neural network was used which was reported before for successful application in CADe (Hupse & Karssemeijer 2009). For training the backpropagation algorithm was used with random initializations of the weights. The optimal number of training cycles was determined by a validation learning curve, whereas training was stopped when the validation performance reached a maximum in order to avoid overfitting. Imbalances in the input dataset were accounted for by presenting negative and positive instances at a fixed ratio during backpropagation learning. As random initializations may lead to differing results in optimization due to local minima, an ensemble of five networks was used and their averaged output served as effective malignancy score (Jiang 2003). 2.3.4. Random Forests. The principle of random forests (RF) is the aggregation of a large ensemble of decision trees (Breiman 2001). During training, each individual tree in the ensemble is fitted by sampling the training data with replacement (bootstrap) and growing the tree to full depth on the training sample. The optimal data split at each tree node is determined by randomly choosing m of the available P input variables and selecting the one which splits the node best. In this work, node splitting was guided by the Gini cost function

Comparative evaluation of SVM for CADe of breast masses in mammography

G(N ) = 1 −

2 X

p2 (ωi ),

7

(6)

k=1

which measures node impurity using p(ωi ) as the fraction of features in class i at node N . The best split was the one which decreases node impurity the most. Further, by calculation of the mean decrease in Gini (MDG) for each variable over all trees, RF allow to obtain a variable importance ranking. The final RF classification score is determined by collecting the votes of each of the n trees in the forest for either class and outputting a vote ratio. As the method is based on decision trees, the splits in the nodes are always parallel to the coordinate axes of the features. 2.4. Classifier Comparison Method Each classifier was used to generate scores for the entire dataset using a 10-fold cross validation (CV) scheme, whereas imaging sets from individual patients were not distributed among the CV splits but kept together. The ratio of positive to negative cases was the same in each of the 10 subsets, yielding a comparable ratio of positive and negative regions in the individual crossvalidation sets. Subsequently, a free-response receiver operating characteristic (FROC) curve was computed for each classifier on the full dataset. In order to allow for one single summary performance metric, the mean true positive fraction (MTPF) of the FROC curve in the interval of 0.05 to 1 average false positives per normal image was computed on a logarithmic scale, which allowed to evaluate over a range of possible operating points matching settings of current CADe systems in practice (Hupse & Karssemeijer 2009): 1 M T P F (0.05, 1) = ln(0.05) − ln(1)

Z

ln(1)

ln(0.05)

s(f ) , f

with f being the average number of false positives (FP) per image and s(f ) the exam sensitivity. An abnormal exam was regarded as correctly detected, if in any of the two or four images an annotated region was hit by a segmented region, whereas a hit was obtained if the center of mass of the segmented area was inside the ground truth annotation outline. In case multiple hits were present, the one with the highest malignancy score was chosen (Kallenberg & Karssemeijer 2008). The statistical significance of MPTF differences was calculated for each pair of classifiers showing the smallest performance discrepancy using the bootstrapping method. For this purpose, exams were resampled with replacement 5000 times from the entire dataset. At each step of resampling, FROC curves were calculated based on the malignancy scores of the resampled exams and the difference between MTPF of the classifiers was recorded. Based on the MTPF samples 95% confidence intervals were obtained. Differences in MTPF were considered statistically significant at a level of α = 0.05, whereas p-values were calculated as the fraction of MTPF differences which were zero or negative (Samuelson & Petrick 2006) (Hupse & Karssemeijer 2009).

Comparative evaluation of SVM for CADe of breast masses in mammography

8

3. Experiments and Results The database was split into a separate set for feature ranking and a test set, whereas the ratio of positive to negative cases was chosen equal in both sets. The feature ranking set comprised data from 829 image examinations with 186 containing malignant masses (383 patients, 2520 images), resulting in 12539 candidate mass regions with 290 positives. Analogously, the test set comprised data from 2516 image examinations (1156 patients, 7544 images) with 567 showing a malignant mass of which 204 were initially diagnosed as false negatives, yielding 37631 candidate regions with 921 positives. For both sets the full amount of 73 features was available. 3.1. Classifier Parameterization Prior to classification of the test sets, the individual classifier parameterizations were determined. LDA could be applied directly, since all necessary parameters were inherently estimated from the training data. The NNet was configured with a learning rate ν = 0.005, 12 hidden nodes, random initialization of the internal weights and a sampling ratio of 9:1 for the presentation of negative and positive instances during learning. The validation learning curve was generated using an independent data set of 224 images. Finally, the output of an ensemble of five NNets was averaged to generate the classification output. For SVM, k-NN and RF the free model parameters were estimated by grid search in the training data. For each method, a parameter configuration from a search grid was selected, five-fold crossvalidation was performed inside the training data and the area under the curve (AUC) of the pooled receiver operating characteristic (ROC) was computed. The parameter configuration yielding largest AUC was selected for classification of the test set. In√RF, the √ number √ of splits 1 was chosen based on the number of features P as n ∈ {b 2 P c, b P c, b2 P c} with 1000 trees. For k-NN, the search range was set to k ∈ {40 + i ∗ 50|i = 0, 1, . . . , 10}. For the SVM, the Gaussian kernel width σ was estimated by sampling 1000 training data items, computation of the pairwise distances and deriving the 30-, 50-, 70- and 90-percentile of the sampled distances. The whole procedure was repeated 10 times and the averaged values were used as search grid. The penalty parameters were chosen as C + ∈ {1, 10, 50, 100} and C − := 1. 3.2. FROC Evaluation on Complete Database The five CADe systems with different classifiers were trained on the entire database using all available features and compared with FROC analysis. Table 2 outlines the obtained mean sensitivities using the entire database. The corresponding summary FROC curves are depicted in Figure 1(a), whereas in Figure 1(b) separate FROC curves for diagnostic and false negative exams are presented, outlining the generally higher sensitivity obtained on diagnostic exams as false negative exams typically contain very subtle lesions.

Comparative evaluation of SVM for CADe of breast masses in mammography

9

Figure 1. FROC curves showing the average number of false positives on normal images against the exam sensitivity for the individual classifiers using all available features. On the left, the FROC curves on the entire database with 567 positive image examinations are depicted, while the right plot presents the FROC for 363 diagnostic exams and 204 false negative exams with the upper resp. lower set of curves. The grey vertical bars indicate the range used to compute MTPF differences.

(a) All exams

(b) Diagnostic and false negative exams

The parameterization found for SVM was obtained with average C¯ + = 10 (for all folds) and σ ¯ = 204.13 with standard deviation sd = 24.11 during CV, corresponding to stable model selection. For k-NN, k¯ = 330 with sd = 80 was chosen, indicating a mild amount of variance together with rather large neighborhood choice, which could be caused by the standardization of input data, but optimized AUC in the given search range. For RF, consistently n = 4 splits were chosen. Direct comparison of performance showed that LDA offered the lowest MTPF, while RF and k-NN showed better results. NNet and SVM offered the best performances, whereas SVM showed a significantly larger MTPF than the NNet (P = 0.0002), providing an improvement of 3.39%, 5.84%, 9.08% and 20.33% with respect to NNet, k-NN, RF and LDA. As indicated in Figure 1(b), the performance increase mainly resulted from improved sensitivity in diagnostic image exams, while on false negative exams no clear difference between SVM and NNet could be identified. 3.3. Variation of Training Data and Input Features 3.3.1. Feature Ranking and Training Data Subsampling. In the second part of the analysis, the classifiers were evaluated using different combinations of feature sets and training data. For this purpose, an importance ranking of all 73 input features was obtained by fitting an RF classifier to the feature ranking dataset with 10000 trees and √ N = b 73c = 8 splits. Based on this ranking, 15 feature sets were constructed by

Comparative evaluation of SVM for CADe of breast masses in mammography Classifier

MTPF(0.05, 1)

Compared to

95% CI

P-value

SVM NNet K-NN RF LDA

0.6860 0.6635 0.6481 0.6289 0.5700

NNet k-NN RF LDA -

[0.0114,0.0496] [0.0022,0.0473] [0.0064,0.0440] [0.0455,0.0876] -

0.0002 0.0278 0.0072 < 0.0001 -

10

Table 2. Comparison of the classification algorithms according to the mean true positive fraction of exams (MTPF) in the interval 0.05 to 1 FP per normal image on logarithmic scale. Included are the 95% confidence intervals (CI) of the bootstrapped differences and corresponding p-values.

iteratively adding the next 5 best features to the current set, starting from five features and including the full set as well. The proportional composition of the sets is shown in Figure 2. The first set comprised exclusively likelihood features. In the subsequent steps, context, grey level and spiculation features were entered, followed by contrast, location, gradient and linear texture features. By a feature set size of 40 all feature categories were covered, so that subsequent feature sets mainly differed by inclusion of correlated features, i.e. additional features taken from the already included feature groups. In addition to the definition of different feature sets, the training data for each CV fold were subsampled so that 10%, 25%, 50% and 100% of the cases could be used for classifier training while a balanced ratio of positive and negative cases was kept. Figure 2. Composition of the feature sets used for classifier comparison based on stepwise filtering of a random forest variable importance ranking. In total, 15 feature sets with minimal five and maximal 73 features were obtained comprising features from 10 groups.

3.3.2. Comparative Analysis of Classifier Stability. SVM and the reference classifiers were trained using each combination of training data set size and feature sets and evaluated using MTPF as performance summary metric on the test data. The results are depicted in Figures 3 and 4, allowing an overview of the individual classifier robustness

Comparative evaluation of SVM for CADe of breast masses in mammography

11

against data and feature variation resp. a direct comparison of absolute performances of the classifiers. Figure 3. Mean exam sensitivity (MTPF) for each classifier trained on different feature sets and subsampled training data. The best detection performance for a given combination of features and training data is indicated with markers on the surface.

(a) LDA

(b) RF

(d) NNet

(c) K-NN

(e) SVM

As presented in Figure 3, the gradual increase of training data size caused a steady increase in performance for most classifiers, with LDA being an exception as no particular differences in performance could be noted. SVM reached the best absolute performances when using 70 features, however, the difference to using all features was not strongly pronounced and decreases with growing data set size, confirming the findings from Section 3.2. Analogously, all other classifiers provide best performance when using the complete training data and feature set, whereas e.g. NNet shows a less pronounced performance increase as the SVM when adding another 50% of training data in the last step. With smaller training sets, NNet, k-NN and RF exhibit peaking phenomena, i.e. the addition of features resulted in reduced detection performance. This effect was most pronounced for RF but vanished when using all available training data. Adding features resulted in a performance increase in two phases: first, until approx. one third of all features were added, a rapid performance increase was observed which was equivalent to adding features ranked higher by RF. Subsequently, a second phase was observed, showing less rapid performance increase when correlated and lower ranked features were included. In Figure 4, a direct comparison of the detection performances shows that the SVM offered only low detection performances in the first phase of feature addition, while RF, k-NN and NNet performed similar to each other with larger MTPF

Comparative evaluation of SVM for CADe of breast masses in mammography

12

and only minor instabilities. Further analysis indicated that the varying performances of SVM were connected to the choice of the penalty parameter C + during model selection. Using only 10% of training data, varying selections of C + were used for classification during CV. When more training data were used, the stepwise addition of features caused the penalty parameter to converge to a stable choice of C + = 10, offering superior performance compared to the other classifiers. Figure 4(d) illustrates this effect, as here the performance increase becomes visible when all different descriptor groups were covered by the feature set, which was equivalent to a constant penalty parameter choice of C + = 10. Once a stable configuration with larger feature sets was reached, SVM showed similar (10% training data) or better MTPF (25%-100% training data) compared to NNet and the other reference classifiers. Figure 4. Comparative evaluation of the mean exam sensitivity for classifiers trained on 10%, 25%, 50% and 100% training data and varying feature sets obtained from RF feature ranking on validation data.

(a) 10% training data

(b) 25% training data

(c) 50% training data

(d) 100% training data

Comparative evaluation of SVM for CADe of breast masses in mammography

13

4. Discussion and Conclusion In this work, an SVM-based approach for detection of breast masses in mammography was presented and evaluated against four reference systems based on NNet, k-NN, RF and LDA. The evaluation scheme aimed at analyzing the CADe capabilities of the different classifiers for malignant breast mass detection, using a large database comprising correctly diagnosed exams from screening as well as false negatives. Model selection for the individual classifiers was guided by practical considerations for the application in CADe, which does not exclude the option that other configuration variants could perform differently. With respect to model selection complexity, LDA was the most favorable approach as all parameters were inherently estimated from data. The SVM was configured with two free parameters which had to be found via a time consuming grid search scheme. For the k-NN and RF the parameters were estimated analogously. The NNet reflected a state-of-the-art CADe setup requiring rather complex model adaptation, as a validation set together with an oversampling scheme for the positive class were required to avoid overtraining and learn differing misclassification costs in imbalanced data. Further, an ensemble of five NNets was needed to ensure classification stability due to random initialization of weights. Using the complete data and feature set, FROC analysis of the different CADe systems indicated that SVM classification was able to provide a statistically significant increase in exam sensitivity, particularly when an operating point with low number of false positives was desired. More detailed analysis suggested that on false negative image exams, SVM and NNet were both among the best performing methods, while the superior SVM detection on the complete database mainly stemmed from higher mean sensitivity on diagnostic exams, outperforming the reference classification methods. The consistency of these findings was further examined by varying training data and feature input. A series of inclusive feature sets of increasing dimensionality was constructed based on iteration of an importance ranking, which helped to ensure that feature sets always contained discriminative features, offering a realistic scenario for CADe training on each set. For variable importance assessment, RF offered a robust multivariate feature ranking mechanism which provided useful results, as all classifiers showed a rapid performance increase when descriptors were added which were judged highly discriminative by RF. Among the highest ranked features were likelihood and context descriptors, as these represent summary classifications of pixel-wise spiculation and gradient, making them good individual predictors. Similarly, context features analyze likelihoods of different corresponding regions in an image examination. Among the highest ranked genuine region-based features were spiculation and density measures, which are well known radiological mass lesion descriptors. The lower ranked variables mostly comprised features from previously covered categories, which was reflected in a less pronounced mean sensitivity increase upon their addition. In smaller databases, often the inclusion of correlated features is avoided in order

Comparative evaluation of SVM for CADe of breast masses in mammography

14

to achieve better generalization capabilities. However, on larger datasets correlated features bear the potential to improve the signal to noise ratio, which could be observed in this study as these contributed to better detection for all classifiers. Further, often dedicated feature subsets are constructed using feature selection with the focus to improve performance, as discriminative subsets are identified and the curse of dimensionality can be alleviated thanks to increased stability. To this end, different feature selection schemes have been proposed in the literature, which have the potential to provide improved performances by selecting dedicated feature subsets for specific classifiers. However, the evaluation did not suggest an immediate need for feature selection, as a substantial performance degradation was not observed for large feature sets and stepwise feature inclusion showed little variance as a large database was used. Using subsampled data, RF were an exception, as the increase of dimensionality caused performance deterioration, indicating that feature selection or different parametrization could provide a benefit. Direct comparison of classifiers indicated that LDA was robust against feature and data variation, but offered least overall performance, particularly when more training data was provided. These observations suggest that nonlinear models were more suitable for detection as these showed a general trend for performance increase using more data and features. SVMs are commonly regarded as well suited for high-dimensional data, as their generalization error bounds do not explicitly depend on input dimensionality, which is supported by the results as large feature sets provided excellent SVM detection performance. Conversely, the choice of the correct parametrization appeared more important, as instabilities were observed on small input feature sets, leading to reduced detection performance compared to the nonlinear reference methods such as NNet. Nevertheless, once parameter estimation stabilized, a consistently superior detection performance was found for a large variety of feature sets and training data set sizes, suggesting that the provided texture, grey level as well as correlated features were more efficiently exploited by the SVM. In summary, this work provides insight into the effect of classifier selection on CADe of breast masses, emphasizing that careful choice and parameterization of the classification method can significantly impact CADe performance. To this end, an SVM-based CADe scheme was presented and evaluated against several other popular classifiers, showing that the SVM was able to significantly improve detection performance at operating points with low FP-rates. These observations were confirmed on varying training and feature sets, stressing the importance of stable parameterization for SVM which can eventually lead to superior exploitation of rich feature sets. It is concluded that Gaussian kernel SVM classification is an effective tool for improving MG CADe, outperforming several state-of-the-art approaches while offering a low complexity framework.

Comparative evaluation of SVM for CADe of breast masses in mammography

15

Acknowledgments This work was funded by the European 7th Framework Program, HAMAM, ICT2007.5.3. References Ben-Hur A & Weston J 2010 Methods in Molecular Biology 609, 223–239. Breiman L 2001 Machine learning 45(1), 5–32. Burges C 1998 Data Mining and Knowledge Discovery 2(2), 121–167. Campanini R, Dongiovanni D, Iampieri E, Lanconelli N, Masotti M, Palermo G, Riccardi A & Roffilli M 2004 Physics in Medicine and Biology 49, 961. Cheng H, Shi X, Min R, Hu L, Cai X & Du H 2006 Pattern Recognition 39(4), 646–668. Cherkassky V & Mulier F 2007 Learning from data: Concepts, Theory and Methods 2nd edn WileyIEEE Press. Cortes C & Vapnik V 1995 Machine learning 20(3), 273–297. Cristianini N & Shawe-Taylor J 2000 An introduction to support Vector Machines and other kernel-based learning methods Cambridge Univ Pr. Duda R, Hart P & Stork D 2001 Pattern classification 2nd edn Wiley. El-Naqa I, Yang Y, Wernick M, Galatsanos N & Nishikawa R 2002 IEEE Transactions on Medical Imaging 21, 12. Elter M & Horsch A 2009 Medical Physics 36, 2052–2068. Guo Q, Shao J & Ruiz V 2005 Journal of Physics: Conference Series 15, 88–94. Guyon I & Elisseeff A 2003 Journal of Machine Learning Research 3, 1157–1182. Hupse R & Karssemeijer N 2009 IEEE Transactions on Medical Imaging 28(12), 2033–2041. Hupse R & Karssemeijer N 2010 Physics in Medicine and Biology 55, 2893–2904. Jiang Y 2003 IEEE Transactions on Medical Imaging 22(7), 913–921. Kallenberg M & Karssemeijer N 2008 Proceedings of the 9th international workshop on Digital Mammography pp. 343–349. Lesniak J, Hupse R, Kallenberg M, Samulski M, Blanc R, Karssemeijer N & Sz´ekely G 2011 Proceedings of the SPIE Medical Imaging 7963, 52–58. Mavroforakis M, Georgiou H, Dimitropoulos N, Cavouras D & Theodoridis S 2006 Artificial Intelligence in Medicine 37(2), 145–162. Nishikawa R, Kallergi M, Orton C et al. 2006 Medical Physics 33, 811–814. Samuelson F & Petrick N 2006 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro pp. 1312–1315. Tang J, Rangayyan R, Xu J, El Naqa I & Yang Y 2009 IEEE Transactions on Information Technology in Biomedicine 13(2), 236 –251. te Brake G, Karssemeijer N & Hendriks J 2000 Phyics in Medicine and Biology 45(10), 2843–2858. Timp S & Karssemeijer N 2004 Medical Physics 31, 958. Wei J, Chan H, Zhou C, Wu Y, Sahiner B, Hadjiiski L, Roubidoux M & Helvie M 2011 Medical Physics 38, 1867. Wei L, Yang Y, Nishikawa R & Jiang Y 2005 IEEE Transactions on Medical Imaging 24(3), 371–380.