Initial Results of Automated Melanoma Recognition H. Ganster, M. Gelautz, A. Pinz
Institute for Computer Graphics, Technical University Graz Muenzgrabenstrasse 11, A-8010 Graz, Austria E-mail:
[email protected]
M. Binder, H. Pehamberger
Department of Dermatology, University of Vienna Medical School Waehringer Guertel 18-20, A-1090 Vienna, Austria
M. Bammer, J. Krocza
Austrian Research Center Seibersdorf, A-2444 Seibersdorf, Austria
Abstract
In order to test the ability to automate early skin cancer recognition, a system for the computerized analysis of images obtained from epiluminescence microscopy (ELM) has been developed. As an initial step, the boundaries of the skin lesion are determined by global thresholding and morphological operations. Then, a set of features containing information about the malignity of the lesion is extracted. This set includes both shape and radiometric features. Finally, a KarhunenLoeve-transform and a minimum distance classi cation are applied in order to classify the lesion as either benign or malignant. It was found that 86% of the test images were classi ed correctly.
1. Introduction During the last fty years the occurrence of melanoma has dramatically increased. While in the thirties of this century one out of 100 000 people living in the United States or Europe su ered from melanoma, this number has risen to 15 out of 100 000 nowadays, with the tendency still increasing. This trend is also true for Austria, where currently between 1500 and 2000 cases are reported every year. Because of the curability of melanoma by surgical excision in early stages of tumor development, early tumor recognition is of utmost importance. During the last years a signi cant improvement in early tumor recognition has been achieved
by using the so-called epiluminescence microscopy (ELM) [7, 8]. This technique uses oil immersion to render the epidermis translucent, thus giving insight into subsurface structures of the skin which are not visible otherwise. This delivers a set of new features which have turned out to improve the reliability of early diagnosis considerably. In the following, some ELM-criteria used by dermatologists for the classi cation of pigmented skin lesions are brie y outlined [7]: Pigmentation: Asymmetry of pigmentation is indicative of a malignant lesion. Depigmentation: Depigmentation represents absence or diminuition of pigment within a pigmented lesion. In benign lesions, depigmentation is regular and usually found at the center, whereas in malignant lesions it is irregular, located anywhere in the lesion, and often found at the periphery. Color: Irregular bluish or gray-blue areas can almost exclusively be found in malignant lesions. Brown globules: In benign lesions they are uniform in size and shape and regularly distributed, whereas variations in size and space and irregular distribution indicate malignity. Black dots: When present in benign lesions, they only occur at the center and are regular in size, shape, and distribution. In malignant lesions they also occur at the periphery, vary in size and shape, and are irregularly distributed. This paper deals with the computerized analysis of digitized ELM-images of pigmented skin lesions, the nal goal being the automatic classi cation of the lesions as benign or malignant (i.e. melanoma). The analysis was carried out in several steps: The rst step was the segmentation process, in which the object of interest (i.e. the skin lesion) was extracted from the background in the scanned (analogdigital converted) ELM-image. In the next step, a set of parameters which contain information about the malignity of the skin lesion was determined, and algorithms for the extraction of these parameters were developed and tested. The choice of the appropriate parameters was based on information found in medical literature (for example, [7] and [8]), as well as on personal discussions with experienced dermatologists. Finally, a minimum distance classi cation based on the previously extracted parameters was carried out, and the results were evaluated. In recent literature on computerized medical imaging several publications dealing with the automated analysis of skin lesions can be found. Some articles focus on the automatic detection of tumor boundaries ([3, 10, 2]), others put an emphasis on the extraction of characteristic features ([9, 1]). However, none of the algorithms presented was suited to the peculiarities of images obtained from epiluminescence microscopy. Therefore, new algorithms had to be designed in order to exploit the advantages of ELM over conventional clinical diagnosis. Our data set consisted of 80 color slides, which had been provided by the Department of Dermatology at the University of Vienna. The slides were digitized at a resolution of 1000 dpi in 'true color' (i.e. 8 bit/pixel in red, green, and blue). However, all color images are reproduced in black and white in this paper (Figs. 1, 5, 10, 11). A classi cation of the test images into three groups, namely (a) benign, (b) malignant, and (c) dysplastic, was performed by dermatologists of the above mentioned department. First, the diagnosis of the pigmented skin lesions was made clinically. After clinical diagnosis and pathology all lesions were surgically excised and processed for routine histopathological examination. The histopathological diagnosis was used as the gold-standard of truth. Of the total number of 80 images
21 were classi ed as malignant and 29 were recognized as benign. The remaining 30 images were classi ed as so-called dysplastic nevi, which means that lesions may become malignant with a reasonably high probability. Figure 1 shows examples of benign (a), malignant (b), and dysplastic (c) pigmented skin lesions.
(a) benign
(b) malignant
(c) dysplastic
Figure 1: Examples of pigmented skin lesions. We will rst discuss the segmentation process. This is followed by the feature extraction and classi cation process. Finally, the results will show that the initial e orts already provide the correct result in 86% of all cases examined thus far.
2. Segmentation In order to be able to analyze a pigmented skin lesion, the rst step is to extract the object of interest (i.e. the skin lesion) from the background. This was done by global thresholding in a bimodal histogram. The threshold value was determined in HSV (hue, saturation, value) space, by means of histogram smoothing Number 1500 1000 500
0
100 200
Sat
Figure 2: Smoothed his- Figure 3: Binary image aftogram. The threshold ter thresholding the origican be found at 184 and nal image (Fig. 1 c). is marked with an arrow.
Figure 4: Binary mask for the feature extraction step. All disturbing objects are removed.
and searching for local minima. The histogram (S-value) corresponding to Fig. 1 c can be seen from Fig. 2: The threshold value which is marked there was used to create the binary image shown in Fig. 3. Normally, these images show not only the object of interest, but also several smaller objects, which have to be removed. The object of interest might also contain holes which have to be lled by morphological operations (Fig. 4). In the last step of the segmentation the binary mask is used to nd the boundary of the skin lesion. Examples of images with the detected boundaries superimposed are given in Fig. 5.
(a) benign
(b) malignant
(c) dysplastic
Figure 5: Examples of boundaries in pigmented skin lesions.
3. Feature Extraction Since it was not possible to compute the ELM-features discussed in the introduction directly, we had to nd a set of features which approximate the ELM-features, and, therefore, allow to distinguish between the three groups of skin lesions. These features can be divided into two main categories, namely shape features and radiometric features. Each of the two categories comprises several well known and other more speci c features, which result from visual classi cation by dermatologists.
3.1. Shape Features
The most basic features are area, perimeter, and compactness. Area is just the sum of pixels inside the object. The perimeter can be calculated as the sum of pixels at the object's boundary. Compactness of an object is de ned as the ratio of area to perimeter. Further features can be calculated with the help of the center of gravity (Fig. 6). Polar distances are the distances from the center of gravity to the boundary of the object. Derived features are minimum, maximum, mean value, and variance of the polar distances. For polar minimum and maximum both absolute and relative values (in relation to the polar mean value) were determined. We also computed the ratio of minimum to maximum polar distance, called eccentricity.
We determined a surrounding rectangle with minimal area (minimum bounding rectangle, Fig. 7), using the orientation of the object to nd this rectangle. The distance from the center of the rectangle to the center of gravity serves as a feature. A very characteristic feature is the structure of the lesion boundary. From the chain code [6] (Fig. 8) of the object a feature called errors in chain code was derived, which is a measure of boundary irregularity.
Figure 6: The center of Figure 7: Plot of the min- Figure 8: Chain code plotgravity is marked by a imum bounding rectangle ted into the object. cross in the binary mask. of the binary mask.
3.2. Radiometric Features
Some very important features can be extracted from the color components, because the homogeneity or irregularity of the color inside the lesion is indicative of the malignity of that lesion. We converted the RGB (red, green, blue) image to the HSV and the HLS (hue, lightness, saturation) system ([4], pp. 584-599) and took minimal, maximal, mean value and the variance of each color component from HSV and the lightness component from HLS. Further features of interest were the sharpness of the color transition at the border of the lesion, and the smoothness of the color in the interior of the lesion. To extract these two features, rays from the center of gravity were cast, and the color distribution along these rays was examined. Fig. 9 shows the plot of lightness (vertical axis) versus location on the ray (horizontal axis). The average grade of the rays on the lesion border and the number of local minima on the rays serve as parameters for the sharpness of the color transition and the smoothness, respectively. A lesion often contains di erent parts whose colors can be clearly distinguished (Fig. 10). The ratio of dark to light regions turned out to be a useful feature. Sometimes, tiny dark spots can be found inside the lesion (Fig. 11). The related parameters are the number of di erent spots and the area covered by the spots compared to the full area of the lesion.
Lightness[%] 100
50
0
Skin Border Lesion
20 40 60
Dist
Figure 9: Color ray from the center of gravity. The borders for the calculation of the grade are marked.
3.3. Feature Examples
Figure 10: The dark lesion Figure 11: Example for area is marked with the in- the determined spots. ner contour. For illustration the original boundary of the lesion is also plotted.
A total number of 33 features was extracted. Tables 1 and 2 show the mean values of these features for the three classes.
Diagnosis
Area Perimeter Pol. min. abs. Pol. min. rel. Pol. max. abs. Pol. max. rel. Pol. mean Pol. variance Eccentricy Compactness Distance Rect. Errors in chaincode
malignant dysplastic benign 16041 19050 13492 639 754 521 49.00 56.35 48.63 0.3789 0.3268 0.3443 103.27 108.25 83.44 9.54 10.80 12.38 72.13 78.46 64.42 217.57 186.98 89.43 2.14 1.93 1.75 2.13 2.70 1.65 7.65 8.19 4.12 15.82 27.50 10.69
Table 1: The mean values of the extracted shape features, computed from 48 sample ELM-images.
4. Classi cation 4.1. Method
Because of the high dimension of the feature space, a Karhunen-Loeve-transform ([5], pp. 148-156) was applied. This delivered a set of feature combinations which
Diagnosis
malignant dysplastic benign Hue min. 7.06 6.35 6.79 Hue max. 54.80 44.12 43.41 Hue mean 20.78 17.72 19.79 Hue variance 81.34 75.84 103.52 Sat min. 55.04 62.76 63.48 Sat max. 92.44 93.24 93.53 Sat mean 81.29 85.09 84.93 Sat variance 40.04 43.46 64.26 Val min. 8.91 16.55 22.65 Val max. 97.74 98.92 94.82 Val mean 53.87 62.33 73.11 Val variance 774.16 534.87 302.04 Lightness min. 5.60 9.78 12.84 Lightness max. 65.85 64.20 64.51 Lightness mean 31.94 36.11 42.41 Lightness variance 280.20 213.92 144.56 Grade of color 1.33 1.28 1.25 Loc. min. on ray 3.18 3.43 3.00 Dark area : Full area 15.82 27.50 10.69 Number of spots 9.55 10.80 12.38 Area (spots) : Full area 0.3789 0.3268 0.3443 Table 2: The mean values of the extracted radiometric features, computed from 48 sample ELM-images. were no longer correlated and sorted according to the variance of their data. It turned out that only the rst six components of the transformed feature vectors contained useful information. The higher order components contained just data with very small variance, thus not a ecting the classi cation. The separation of the skin lesions into di erent classes was carried out by using a supervised minimum distance classi cation ([5], pp. 580-599). Out of the 80 images available, we used 48 images as a training set for the minimum distance classi cation. These 48 images contained 11 malignant, 17 benign and 20 dysplastic lesions. The remaining 32 images served as a test set for the previously trained minimum distance classi cation. During the training phase the mean vectors of the classes were calculated. Afterwards, in the test phase, these mean vectors were used as representatives of the di erent classes.
4.2. Results
In a rst attempt, we used all three groups of images (benign, malignant, and dysplastic). When the training phase was nished, we rst applied the classi cation to the training set itself. The results can be found in Table 3. Note that none of the malignant lesions was misclassi ed as benign, and vice versa. Then, the classi cation was carried out on the actual test data. As can be seen from Table 4,
Diagnosis
malignant dysplastic benign Overall Percentage
Category
malignant dysplastic benign 9 2 0 5 11 4 0 1 16
Correctness 82% 55% 94% 75%
Table 3: Classi cation results for the training set. The data set consisted of three groups of skin lesions.
Diagnosis
malignant dysplastic benign Overall Percentage
Category
malignant dysplastic benign 7 3 0 3 3 4 1 3 8
Correctness 70% 30% 67% 56%
Table 4: Classi cation results for the test set. The data set consisted of three groups of skin lesions. 67% of the benign and 70% of the malignant lesions were classi ed correctly. Again, none of the malignant lesions was misclassi ed as benign. However, the results concerning the group of dysplastic lesions were not satisfying: Only 30% of them were recognized correctly. The lower success rate for the dysplastic class is not unexpected: It re ects the fact that even for experienced dermatologists this class is more dicult to handle, since it represents the transition between benign and malignant. Although the recognition of the dysplastic lesions was not satisfactory, the results concerning the benign and malignant lesions were quite encouraging. Therefore, further tests were carried out using only these two groups as test data. The results of these tests can be seen from Tables 5 and 6: The diagnostic score for the test set was 90% for malignant and 83% for benign lesions, resulting in an overall success rate of 86%.
Diagnosis
malignant benign Overall Percentage
Category
malignant benign 11 0 0 17
Correctness 100% 100% 100%
Table 5: Classi cation results for the training set. The data set consisted of two groups of skin lesions.
Diagnosis
malignant benign Overall Percentage
Category
malignant benign 19 1 2 10
Correctness 90% 83% 86%
Table 6: Classi cation results for the test set. The data set consisted of two groups of skin lesions.
5. Conclusion and Outlook Tests on a data set consisting of skin lesions which had previously been classi ed as either benign or malignant delivered good results. The classi cation of those skin lesions which are just at the transition between benign and malignant still poses problems. In order to tackle this problem, the immediate next step will be to nd out whether another classi cation method is better suited to the problem than the minimum distance classi cation which is used at the moment. At the next stage of the project, more test data will be delivered by a new digital ELM-camera which has been developed only recently at the Department of Dermatology at the University of Vienna, in cooperation with the Austrian Research Center Seibersdorf. Together with this new image acquisition system, the algorithms developed so far and still to be re ned are expected to contribute signi cantly to an understanding of automated early melanoma recognition.
6. Acknowledgements The authors would like to thank the Austrian Research Center Seibersdorf for providing a grant under which part of this work was performed.
References [1] A.P. Dhawan and P. Kini. Pigment Pattern and Depth of Invasion Analysis of Skin Lesions for Malignant Melanoma Detection. In Proceedings of SPIE, volume 1231, pages 75{83. The International Society for Optical Engineering, 1990. [2] A.P. Dhawan and A. Sicsu. Segmentation of images of skin lesions using color and texture information of surface pigmentation. Computerized Medical Imaging and Graphics, 16(3):163{177, May 1992. [3] F. Ercal, M. Moganti, W. Stoecker, and R. Moss. Detection of Skin Tumor Boundaries in Color Images. IEEE Transactions on Medical Imaging, 12(3):624{627, September 1993. [4] J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes. Computer Graphics: Principles and Practice, pages 584{599. Addison-Wesley Publishing Company, second edition, 1990.
[5] R.C. Gonzales and R.E. Woods. Digital Image Processing. Addison-Wesley Publishing Company, 1993. [6] S.D. Kim, J.H. Lee, and J.K. Kim. A New Chain-Coding Algorithm for Binary Images Using Run-Length Codes. Computer Vision Graphics Image Processing, 41:114{128, 1988. [7] H. Pehamberger, M. Binder, A. Steiner, and K. Wol . In Vivo Epiluminescence Microscopy: Improvement of Early Diagnosis of Melanoma. The Journal of Investigative Dermatology, 100(3):356{362, March 1993. Supplement. [8] A. Steiner, M. Binder, M. Schemper, K. Wol , and H. Pehamberger. Statistical Evaluation of Epiluminescence Microscopy Criteria for Melanocytic Pigmented Skin Lesions. Journal of the American Academy of Dermatology, 29(4):581{ 588, October 1993. [9] W.V. Stoecker, W.W. Li, and R.H. Moss. Automatic Detection of Asymmetry in Skin Tumors. Computerized Medical Imaging and Graphics, 16(3):191{197, May 1992. [10] S.E. Umbaugh, R.H. Moss, and W.V. Stoecker. An Automatic Color Segmentation Algorithm with Application to Identi cation of Skin Tumor Borders. Computerized Medical Imaging and Graphics, 16(3):227{235, May 1992.
