Image Retrieval-Based Decision Support System for ... - IEEE Xplore

Image Retrieval-Based Decision Support System for Dermatoscopic Images Md. Mahmudur Rahman, Bipin C. Desai Dept. of Computer Science Concordia University Montreal, QC, H3G 1M8, CANADA mah [email protected]

Abstract This paper presents a content-based image retrieval system for dermatoscopic images as a diagnostic aid to the dermatologists for skin cancer recognition. In this context, the ultimate aim is to support decision making by locating, retrieving and displaying relevant past cases along with diagnostic reports. However, most challenging aspect in this domain is to extract local lesion specific image features and define the relevance between query and database images for retrieval. A fast and automatic segmentation method to detect the lesion from background healthy skin is proposed. This method first transforms a color image into an intensity image by utilizing domain specific image properties and NBS color distance in HVC color space. Lesion mask is detected by fusing individually segmented images based on iterative thresholding. Lesion specific local color and texture features are extracted and represented in the form of mean and variance-covariance of color channels and in a reduced PCA sub-space. Finally, for effective image retrieval, a similarity matching function is defined based on the fusion of a Bhattacharyya and Euclidean distance metric. The performance of the retrieval system is evaluated using average precision on a collection of 358 images, which demonstrates effectiveness of the proposed approach.

1. Introduction Skin is the largest and one of the most vulnerable organs, covering the entire outside of the human body and it is continually renewed. Though seldom life threatening, skin disorders can be uncomfortable and may cause chronic disabilities. In addition, skin cancer in the form of malignant melanoma is one of the most common cancers in human being in the world [1, 2]. Detection of malignant melanoma in its early stages considerably reduces mortality, hence a crucial issue for dermatologists. Dermatoscopy is the technical name for skin surface

Prabir Bhattacharya Institute for Information Systems Engineering Concordia University Montreal,Quebec, CANADA [email protected]

microscopy or epiluminescence microscopy (ELM) [3], a technique that allows in vivo microscopic examination of skin lesions, and it has already proven to be an effective tool for early detection of skin cancers [1, 2, 3]. The technique involves application of immersion oil to skin, which eliminates light reflection from the skin surface and renders the stratum corneum transparent. Using a magnifying lens, the structures of the epidermis and epidermal-dermal junction can then be visualized. Digitization of the images after initial visual assessment permits storage and often used for comparison purposes if a lesion is being followed over time [3]. Hence, large collections of dermatoscopic images are generated in clinical practices for image guided diagnosis and follow-up studies. There is an increase interest in the use of digital images in locating, retrieving and displaying relevant past cases. It also creates an opportunity to develop automated decision support and training or teaching based systems. The nature of dermatoscopic images, specially color variation and differential structures specific to skin lesions, makes this an ideal application area for content-based image retrieval (CBIR). In CBIR, access to visual information is performed at a perceptual level, using objective measurements of the visual content (e.g., color, texture, shape etc.) and with appropriate similarity model [4]. In medical domain, there are already some successful implementation of CBIR systems [5]. Although, most of the systems currently available are based on radiological images of a specific body part and orientation . Until now, most of the work in dermatology area have focused on the problem of skin cancer detection, here the likelihood of malignancy is computed based on some feature extraction and classification schemes [1, 2, 6]. In clinical practice, the most popular diagnostic criteria for malignant melanoma is ABCD rule [3], where the different letters stand for Asymmetry, Border irregularity, Color variation and Differential structures. While the ABCD rule of dermatoscopy is extremely useful in the clinical environment, each criteria are evaluated subjectively and depends solely on the physician’s level of

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

expertise and experience [1]. However, complete integrated diagnostic systems are hardly found in clinical use and there is yet a CBIR system to be developed (as far our knowledge) in this domain. Motivated by this, we propose a CBIR approach of digital dermatoscopic images. In this context, the ultimate aim of the CBIR system would be to provide dermatologists with a decision support tool in the form of a display of relevant past cases, along with proven pathology and other suitable information. CBIR could be used to presents cases that are not only similar in diagnosis, but also similar in appearance and in cases with visual similarity but different diagnoses [5]. Hence, it would be useful as a training tool for medical students, residents, and researches to browse and search large collection of disease related illustrations using their visual attributes. Based on the observation of specific characteristics of dermatoscopic images, we propose a fast and automatic segmentation method for lesion detection, lesion specific local color and texture feature extraction and a similarity matching function for image retrieval in effective manner. The proposed approach is presented in the next sections, where Section 2 and 3 present the initial image preprocessing and segmentation technique for lesion detection. Section 4 presents lesion specific feature extraction and Section 5 presents the proposed similarity matching function. Experimental results are provided in Section 6 and finally, we conclude in Section 7.

2. Image pre-processing The main processing step towards a complete analysis of pigmented skin lesion is the detection of the lesion from the healthy skin. Detection of the lesion is a difficult problem in dermatoscopic images as the transition between the lesion and the surrounding skin is smooth and even for trained dermatologist, it is a challenge to distinguish accurately [1]. However, it requires some pre-processing steps to facilitate the segmentation process by removal of unwanted objects or artifacts and color space transformation. We perform a preprocessing step before segmentation that transforms an image in RGB to HVC color space and then to an intensity image based on the specific observation of the dermatoscopic images. Most dermatoscopic images are captured in a certain way that lesion is generally situated close to the center of the image and occupy a reasonable central portion of the image. Conversely, the background healthy skin surrounds the lesion and thus, are more likely to be visible along the periphery or border of the images. Color changes from the background to a lesion or from a lesion to the background is more important then the color variations within a lesion or in the background [7, 8]. Many color spaces have been designed to facilitate the

color specification and the key issue always being considered in selection of a perceptually uniform color space. HVC color space comes from Munsell color coordinate system, which is considered for its successful imitation of human color perception [9]. It represents a color in terms of hue (H), which indicates the types of the color; value (V), which tells the total amount of light; and chroma (C) that describes how much white light is mixed with the color (purity). There are several ways to mathematically transform the RGB to the HVC color space. Because CIEL∗ a∗ b∗ color space is known for its good perceptual correspondence and simple computation, RGB values are first transformed into CIEXY Z, and then changed to CIEL∗ a∗ b∗ , and then altered to HV C values using the following formulas [9]: X = 0.607R + 0.174G + 0.201B Y = 0.299R + 0.587G + 0.114B Z = 0.000R + 0.066G + 1.117B L∗ = 116(Y /Y0 )1/3 − 16, (Y /Y0 ) > 0.008856 L∗ = 903.3(Y /Y0 ), (Y /Y0 ) ≤ 0.008856 a∗ = 500((X/X0)1/3 − (Y /Y0 )1/3 ) b∗ = 200((Y /Y0 )1/3 − (Z/Z0 )1/3 ) H = arctan(b∗ /a∗ ) V = L∗ C = (a2 + b2 )1/2 X, Y, Z are the primaries in XYZ color system, while X0 , Y0 , Z0 are values of a nominally white object-color stimulus, which are usually chosen to be 0.9642, 1.0 and 0.8249 respectively. If ΔH, ΔV and ΔC be the differences of H, V, C color components of an image pixel A = (H1 , V1 , C1 ) and its background B = (H2 , V2 , C2 ), then NBS color distance between A and B is defined as follows [10]: ENBS (A, B) = 1.2∗  2π ΔH)} + (ΔC)2 + (4ΔV )2 (1) 2C1 C2 {1 − cos( 100 There is a close relation between the human color perception and the N BS color distance, which is shown in Table 1 [10]. Taking advantage of the above properties of HV C space and N BS distance, we transform the image from original RGB space to HV C space and determine the mean HV C values of the pixels from the border (2 pixels wide from each side) and central portion (a small central window of 15 × 15 pixels in size) of an image. Next, a color image in HV C space is transformed into an intensity image in such a way that the intensity at a pixel shows the N BS color distance of that pixel with the color of the background (mean of border) or the color of the foreground

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

Table 1. Correspondence between human color perception and NBS distance NBS Value

Human Perception

0 ∼ 1.5

Almost the same

1.5 ∼ 3.0

Slightly different

3.0 ∼ 6.0

Remarkably different

6.0 ∼ 12.0

Very different

12.0 ∼

Different color

(a)

Figure 3. Thresholed images obtained from Fig. 2 (a) and 2 (b).

(a) (a)

(b)

(b)

(b)

Figure 1. (a) dermatoscopic image (original) (b) dermatoscopic image (grey level).

Figure 4. Lesion masks obtained from Fig. 3 (a) and 3 (b).

the pre-processed intensity images. lesion (mean of central image). Using this approach, we obtain images that has high values in lesions and low values in the background or vice versa. By this transformation, lesion can be distinguished better from the background skin, and would be suitable for later processing; as an example is shown in Fig. 2. Whereas, Fig. 1 shows the original image and its simple gray level conversion. The immersion oil used in the dermatoscopic imaging technique may produce small air bubbles manifesting themselves as small white ellipses, lines and dots. This artifacts can be considered as impulsive noise and is reduced by using a median filter. The next step is the segmentation operation to be performed on

(a)

(b)

Figure 2. (a) Intensity image (NBS distance of the pixels to the background) (b) Intensity image (NBS distance of the pixels to the foreground).

3. Segmentation & lesion detection Various image segmentation methods have been proposed in the literature to delineate lesion boundaries from skin cancer images [1, 2, 7, 8]. Hance et al. [8] compared six different color image segmentation techniques for skin cancer images and found that lowest average error could be achieved by adaptive thresholding and when two or more techniques are combined, the accuracy can be improved further. L. Xu at al. [7] proposed an elaborate method by reducing the color image to an intensity image utilizing distance measure in CIELAB color space and used double thresholding and elastic curve fitting technique to finally detect the lesion boundary. Our approach is somewhat similar to the pre-processing steps in [7]. However, their proposed method has three parameters, which may require user interaction to tune and also larger processing time. Thresholding is a computationally inexpensive and fast technique for image segmentation, which is suitable for real time CBIR application. However, thresholding simply based on gray or luminance values of a dermatoscopic image is not enough as the transition between lesion and background skin is often very smooth and not clear. Hence, we perform the above pre-processing operation to increase the distinguishing ability between the lesion and the background skin. Correct threshold detection is crucial for successful segmentation.

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

We utilize the following iterative algorithm for automatic threshold selection:

1: 2: 3: 4: 5: 6: 7: 8: 9: 10:

Compute μB , the mean intensity level of the border pixels as background Compute μF , the mean intensity level of the center pixels as foreground lesion Told = 0 Tnew = (μB + μF )/2 while Tnew = Told do μB = fN(x,y) for f (x, y) < Tnew and NB = B # background-pixels μF = fN(x,y) for f (x, y) ≥ Tnew and NF = F # foreground-pixels Told = Tnew Tnew = (μB + μF )/2 end while

After finding the threshold values for different intensity images generated in the pre-processing stage, we obtain the binary images as shown in Fig. 3. The main purpose of seg-

4. Local Feature extraction Feature extraction from skin lesion is another very important step for an effective retrieval system of dermatoscopic images. Many feature selection and extraction strategies have been proposed [2, 3, 6] from the perspective of classification of images as malignant or benign. Different features are calculated, which attempt to reflect the parameters used in medical diagnosis, such as ABCD rule or more advanced features. These features are certainly effective for the classification purposes, as seen from the performances of some classification-based systems in this domain [1, 2, 6]. However, features good for classification or distinguishing one disease from another, may not be suitable for the presentation. In a retrieval system, we are looking for similar images in terms of color, texture, shape etc. By selecting and extracting good representative features, we may be able to identify images similar to an unknown query image, whether it belongs to the same disease group or not. In this direction, suitable local color feature in the form of a feature vector is extracted by considering the mean or average color of the lesion in HV C color space and variance-covariances of the color channels by estimating the covariance matrix. If average color feature favg of a lesion is represented as favg = (μH , μV , μC ), where μH , μV and μC is the average H, V and C values in HV C space, then the cross-correlation among color channels due to the off diagonal of the 3 × 3 covariance matrix ΣI of the lesion of I is estimated as I  1 (fx − favg )(fxj − favg )T NI − 1 j=1 j

N

ΣI = (a)

(b)

Figure 5. (a) Original Image (b) Final segmented image with lesion mask.

mentation in this application is to find a lesion shape mask. Thus we are only interested in the threshold separating the light (dark) skin background and the darker(lighter) skin lesion as in Fig. 3. In some cases, the segmentation produces several skin lesion candidates due to other small non-lesion objects. So, a post-processing operation is applied on the binary segmented images based on morphological operation of opening and closing. This processing step removes tiny artifacts or holes from the background or lesion images. The mask of the lesion is derived as the largest region within the image as shown in Fig. 4. After detecting the lesion masks from the segmented images, a simple union (OR) operation is applied to obtain the final lesion mask as shown in Fig. 5. Usually the largest object is the skin lesion and is thus selected for further processing for feature extraction.

(2)

where fxj is the color vector of pixel xj and NI is the number of pixels of the lesion of I. Since the covariance matrix is symmetric, only 6 values of it need to be stored in the feature vector for later similarity matching based on Bhattacharyya distance metric. For Euclidean distance measure, local texture features are extracted from the grey level co-occurrence matrix (GLCM) [13]. GLCM is defined as the sample of the joint probability density of the gray levels of two pixels separated by a given displacement d and angle θ. We obtained four co-occurrence matrices for four different orientations (horizontal 0◦ , vertical 90 ◦ , and two diagonals 45 ◦ and 135 ◦ ) and normalize the entries [0,1] by dividing each entry by total number of pixels. Higher order features, such as energy, maximum probability, entropy, contrast and inverse difference moment are measured based on each GLCM to form a five dimensional feature vector and finally obtained I , by concatenata twenty dimensional feature vector fglcm ing the feature vector of each GLCM. In addition, for color histogram-based feature, a 64-dimensional histogram is cre-

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

I ated in vector form as fhisto on RGB color space. We uniformly quantized RGB space into 4 bins for each channel, which results in 64 bins for color histogram. I I I = (fhisto + fglcm ) dimenSince, the combined fcombined sion of the color histogram and texture feature vector (64 for color and 20 for texture for a total of 84) is large enough, we applied principal component analysis (PCA) based dimension reduction technique to reduce the feature dimension I for image (d → m |d = 84, m = 10) and called it as fpca I.

Figure 6. Block diagram of the CBIR system.

5. Similarity matching on local features

single similarity matching function as follows:

Similarity measure is a quantity that reflects the strength of relationship between two features and is usually normalized into a range of [0, 1] [4]. Generally, a similarity measure function is the converse of a distance function. So, when similarity is one (i.e. exactly similar), the distance is zero and when the similarity is zero (i.e. very different), the distance is one. For similarity matching based on color covariance-based feature, the distance between the regions or segmented lesions of query image Q and database image I is computed by way of the Bhattacharyya distance metric as follows [12]:

S(Q, I) = wBhatt SBhatt (Q, I) + wEuclidean SEuclidean (Q, I) (5) Here, wBhatt and wEuclidean are non-negative weighting factors with normalization (wBhatt + wEuclidean = 1), which needs to be selected experimentally. Fig. 6 shows the block diagram of the proposed CBIR approach. When a query image is submitted, pre-processing, segmentation and feature extraction are performed the same way as database images as shown in the bottom level of Fig. 6. The features of query and database images are then matched in a similarity retrieval subsystem. The match score is compared and sorted, where the best k matches are shown to the query interface according to their ranks.

 −1 1 Q I T (ΣQ + ΣI ) DBhatt (Q, I) = (favg − favg ) 8 2

Q (favg

   (ΣQ +ΣI )    2 1 I − favg ) + ln  2 |ΣQ ||ΣI |

6. Experiments & results (3)

Q I and favg are the average color feature vectors, where favg and ΣQ and ΣI are the covariance matrices of the lesions of Q and I respectively. Equation (3) is composed of two terms, the first one being the distance between feature vectors of image regions, while the second term gives the class separability due to the difference between covariance matrices. Euclidean distance measure is used for comparing feature vectors of Q and I in PCA sub-space as

 m   Q Q I I fpcai − fpca DEuclidean (Q, I) = ||fpca − fpca || =

i i=1

(4) For the above distances, the following function is used to transform the distance measures into a similarity measures 2 is the disas S(Q, I) = exp−D(Q,I)/σD(Q,I) , where σD(Q,I) tance variance computed for each distance measure separately over a sample image set. After the similarity measures of color and PCA-based features are determined as SBhatt (Q, I) and SEuclidean (Q, I), we aggregate them into a

To evaluate the effectiveness of the proposed retrieval system, experiments were performed on a image database with 358 dermatoscopic images collected from dermatology image atlases [3, 11]. This collection contains pigmented skin lesions of three categories as benign or common nevi, dysplastic nevi and melanoma. We have selected all the database images as query images and used query-byexample method, where the query is specified by providing an example image to the system. For a quantitative evaluation, the performances of each category are compared based on average precision curves by evaluating top 200 returned results. Precision is the ratio of the number of relevant images returned to the total number of images returned. A retrieved image is considered a match if it belongs to the same category as the query image. We have assigned wBhatt = 0.7 and wEuclidean = 0.3 as weight to the color and texture based similarity measure since Bhattacharyya distance measure performs better due to the consideration of cross-correlation among color channels. The performances of the three image categories are compared based on average precision curves by evaluating top N = {10, 20, 50, 70, 100, 150, 200} returned results. Fig. 7 presents the average precision curves for three different image categories (melanoma, benign and dysplastic nevi). From Fig. 7, it is clear that best accu-

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE

tion” IEEE Trans. on Medical Imaging, vol. 20, No. 3, pp. 233-239, 2001. [3] S. Menizies, W. McCarthy, K. Crotty, and C. Ingvar, An Atlas of Surface Micorscopy of Pigmented Skin Lesions: dermoscopy, Second Edition, McGrawHill, 2003. [4] A. Smeulder, M. Worring, S. Santini, A. Gupta, R. Jain, “Content-Based Image Retrieval at the End of the Early Years.”, IEEE Trans. on Pattern Anal. and Machine Intell., vol.22, pp. 1349–1380, 2000.

Figure 7. Average precision curves of three categories.

racy is achieved by melanoma category, which is more important from diagnostic point of view as it can distinguish images better from other categories. The performance of benign and dysplastic image categories are not as good as melanoma, since low-level color and texture properties may overlap between these two categories and requires more advanced features for comparison.

7. Conclusions In this paper, we have presented a CBIR system as a diagnostic aid for dermatoscopic images. The results of this paper indicate that the proposed approach (image processing, segmentation, similarity matching techniques) is effective to retrieve visually similar lesions from a database. We conjecture that by presenting images with known pathology that are visually similar to the image being evaluated, the use of such a retrieval system may provide a more intuitive aid to dermatologist, potentially leading to improvement in their diagnostic accuracy. Since our main goal is to build an integrated system for dermatoscopic images, we wish to incorporate information from other sources in the form of a case or lab reports of histopathological tests and also investigate the clinical benefit for computer-aided diagnosis.

References [1] P. S. Saugeon, J. Guillod, J. P. Thiran, “Towards a computer-aided diagnosis system for pigmented skin lesions” Computerized Medical Imaging and Graphics, vol. 27, pp. 65-78, 2003. [2] H. Ganster, A. Pinz, R. Rhrer, E. Wildling, M. Binder,H. Kittler, “Automated Melanoma Recogni-

[5] H. Mller N. Michoux, D. Bandon, A. Geissbuhler, “A Review of Content-based Image Retrieval Applications Clinical Benefits And Future Directions”, International Journal of Medical Informatics vol. 73, pp. 1–23, 2004 [6] M. Binder, H. Kittler, A. Seeber, A. Steiner, “Epiluminescence microscopy-based classification of pigmented skin lesions using computerized image analysis and an artificial neural network”, Melanoma Research , vol. 8, pp. 261-266, 1998 [7] L. Xu, M. Jackowski, A. Goshtasby, D. Roseman, S. Bines, C. Yu, A. Dhawan, and A. Huntley, “Segmentation of skin cancer images,” Image Vis. Computing, vol. 17, pp. 65-74, 1999 [8] S. E. Umbaugh, R. H. Moss, W. V. Stoecker, and G. A. Hance, “Automatic color segmentation algorithms: With application to skin tumor feature identification,” IEEE Eng. Med. Biol. Mag., vol. 12, no. 3, pp. 75-82, 1993. [9] M. Miyahara, Y. Yoshida, “Mathematical transform of (r,g,b) color data to Munsell(h,v,c) color data”, SPIE Proceedings in Visual Communication and Image Processing, vol.1001, pp.650–657, 1988. [10] Y. Gong, G. Proietti, C. Faloutsos, “Image Indexing and Retrieval Based on Human Perceptual Color Clustering”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p. 578, 1998 [11] “Dermnet: the dermtologist’s image resource”, Dermatology Image Atlas, available at http://www.dermnet.com/ [12] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed., Academic Press, 1990. [13] R. M. Haralick, Shanmugam, I. Dinstein, “Textural features for image classification.”, IEEETrans Syst,Man, Cybernetics vol. SMC-3, pp. 610-21, 1973.

Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems (CBMS'06) 0-7695-2517-1/06 $20.00 © 2006 IEEE