BAYESIAN BASED CLASSIFIER FOR MINING IMAGE. CLASSES. Zaher Al Aghbari. Department of Computer Science. University of Sharjah. P.O.Box 27272 ...
IADIS International Conference on Applied Computing 2005
BAYESIAN BASED CLASSIFIER FOR MINING IMAGE CLASSES Zaher Al Aghbari Department of Computer Science University of Sharjah P.O.Box 27272, Sharjah, UAE.
Rachid Sammouda Department of Computer Science University of Sharjah P.O.Box 27272, Sharjah, UAE.
Jamal Abu Hassan Department of Computer Science University of Sharjah P.O.Box 27272, Sharjah, UAE.
ABSTRACT In this paper, we demonstrate how semantic categories of images can be learnt from their color distributions using an effective probabilistic approach. Many previous probabilistic approaches are based on the Naïve Bayes that assume independence among attributes, which are represented by a single Gaussian distribution. We use a derivative of the Naïve Bayesian classifier, called Flexible Bayesian classifier, which abandon the normality assumption to better represent the image data. This approach is shown to yield high accuracy results on classifying image databases as compared to it counterpart the “Naïve Bayesian classifier” and the widely used K-Nearest Neighbor classifier.
1. INTRODUCTION Content-based image classification and retrieval systems have been the subject of many multimedia data mining research works in recent years. Image mining deals with the extraction of implicit knowledge, image data relationship or other patterns not explicitly stored in image databases. Early systems, such as QBIC [5], VisualSEEK [11], Netra [9] and MARS [10], facilitate classification, indexing, and retrieval of images, mainly, based on low-level features of images, such as color, texture and shape. Although such systems enjoy the advantage of being fully automatic, they are hard to use by non-expert users due to the semantic gap that exist between the semantic user needs and low-level system requirements. To bridge this semantic gap, semantic classification of images has attracted much interest from the data mining community. The work in [16] is one of the earliest multimedia data mining systems that include a component for mining classes of images. However, the automatic extraction of semantic elements of images is not reliable as apposed to the low-level features. For example, the color distribution in terms of color histogram can be simply extracted for any image, but the automatic identification of, say, trees or sky in the image may be uncertain. For reliable identification of image elements, the work in [7] manually identifies (labels) the image element (called visual keyword) from samples of visual content domain. Then, an image instance to be classified is compared against those visual keywords to detect the specific visual keywords of the image instance. A different approach to organize the images into semantic categories is shown in [12], in which training sample regions of images are classified by a neural network into 11 clusters (e.g. Brick, Cloud, Grass, etc.).
329
ISBN: 972-99353-6-X © 2005 IADIS
A different classifier was created for every cluster. Similarly, the approach in [15] classifies the content of an image into one or more of 10 regions (e.g. sky, water, night, etc.). However, the approach in [15] uses a probabilistic approach to define a semantic codebook of every class. The back-propagation neural network classifier was used in [1] to classify breast cancer regions in the mammographic images based on extracted statistical parameters into three categories: normal, benign and malign. A different work in [3] utilized the self-organizing maps (SOM) to mine out clusters of flag images then label those clusters with relevant concepts. Moreover, based on the k-nearest neighbor classifier, the work in [6] classifies internet images into photographical versus synthetic images. Then, the photographical images are classified into portrait versus non-portrait. Unlike the probabilistic approaches that construct a general explicit description of the target function from training samples, the k-nearest neighbor classifier approach construct a local approximation to the target function that applies in the neighborhood of a new image query. Generally, automatic labeling of images, or image regions, may result in some uncertainty, leading to multiple potential labels for image regions. The approach in [2] proposes a probabilistic approach to label small areas of one set of images as either man-made images or natural, and the local areas of another set were labeled as either inside or outside. Such semantic labeling is based on estimating the class likelihoods from color-texture features of local areas of an image. Another approach [13] proposes a Naïve Bayesian classifier to classify vacation images. Then, in [14], the authors developed an incremental learning paradigm for the probabilistic approach of [13]. Unlike the approach of [2], which labels local areas of an image, in [13], the labeling is based on the global feature vector of the image. It classifies images into city versus landscape and then further classifies the landscape images into sunset, forest and mountain classes. In this paper, we demonstrate a probabilistic approach that produced high accuracy results on semantic image classification. This probabilistic approach is an extension of the Naïve Bayes called the Flexible Bayes which is developed in [8]. For brevity, we call our classifier a Flexible Bayesian classifier (FBclassifer) and its rival a Naïve Bayesian classifier (NB-classifier). The FB-classifier improves on its rival NB-classifier by eschewing the single Gaussian assumption in favor of kernel density estimation; however, it retains the independence assumption. Although a single Gaussian distribution may provide a reasonable approximation to many real-world distributions, it is not always the case especially for image databases as we have concluded from our experiments. We will demonstrate that our classifier, FB-classifier, yields higher accuracy results of semantic image classification as compared to its rival, NB-classifier, and another well known non-probabilistic classifiers, K-nearest neighbor classifier (KNN-classifier). The rest of this paper is organized as follows: in Section 2, we discuss the properties of the NB-classifier and FB-classifier. In Section 3, we introduce the system layout of the image classifier. Then, in Section 4, we discuss and evaluate the result of experiments and compare the results of our classifier to other well known classifiers. Finally, we conclude the paper in Section 5.
2. BASEYIAN CLASSIFIER Bayesian classifier offers a simple yet powerful supervised probabilistic framework for image classification. Bayesian classifiers are used to represent and learn probabilistic knowledge from an image database. The performance goal of a Bayesian classifier is to learn the class knowledge from a training set, where each instance in this set is associated with a class, and then to accurately predict the classes of test instances. Although the Bayesian model assumed independence (attributes are independent) and normality (attributes are represented by a single Gaussian distribution), the model has been successfully adopted to classify image databases [2][8][13][14][15]. In this paper, we will use the FB-classifier that results in higher classification accuracy of image data. Before introducing the FB-classifier, we will first briefly discuss the NB-classifier, which is the model from which the FB-classifier was extended.
2.1 Naïve Bayesian Classifier The NB-classifier can be viewed as a specialized form of a Bayesian network, termed naïve because it relies on two important simplifying assumptions: independence and normality. That is it assumes that the predictive attributes xik of an observed image xi are conditionally independent given the class cj. These
330
IADIS International Conference on Applied Computing 2005
assumptions support very efficient algorithms for both learning and classification [8]. An NB-classifier is often represented graphically as shown in Figure 1, where the direction of the arrows state that the predictive attributes xi1, xi2, …, xin are conditionally independent given the class cj. Let a set of classes C = c1, c2, …, cm denote the classes of the observed images (training set) X = x1, x2, …, xs. Also, consider each observed image xi as a vector of random variables denoting the predictive attribute values xi1, xi2, …, xin. Thus, given a test instance x to be classified, first, using Bayes rule (Eq. 1) we compute the posterior probabilities of each class and then predict the class with the highest probability as the class of x.
P (c j | x i ) =
P ( x i | c j ) P (c j ) P ( xi )
(1)
From the training set, P(cj) is computed by counting the number of occurrences of cj. For each attribute xik, the number of occurrences is counted to determine P(xi). Similarly, assuming categorical attributes, the probability P(xi|cj) can be estimated by counting how often each value xik occurs in the class in the training set. For numeric attributes, which is the case in image data, it is often assumed that the values of attributes of each class are normally distributed. Hence, such a distribution is represented by its mean µ and standard deviation σ. As a result, a probability of an observed value can be efficiently estimated by Eq. 2.
1
P ( xik | c j ) =
2Πσ j
−
e
( x − µ j )2 2σ 2j
(2)
cj
P(x1|cj)
P(x2|cj)
…
P(xn|cj)
Figure 1. A Bayesian network that represent the NB-classifier.
Since an image has n independent attributes, we compute P(xik|cj) for every attribute and then estimate P(xi|cj) by the conjunction of all conditional probabilities of the attributes as shown in Eq. 3. n
P ( xi | c j ) = ∏ P( xik | c j )
(3)
k =1
The posterior probability, Eq. 1, is estimated for every class and then predict the class with the highest probability as the class of the test instance x. The NB-classifier is a simple and efficient approach to classify new image instances; however, it relies on the normality assumption, which does not hold for image data as we concluded from our experiments. Thus, we utilize an extension of the NB-classifier, called FB-classifier that abandons the normality assumption.
2.2 Flexible Bayesian Classifier Instead of estimating the density function of each continuous attribute by a single Gaussian, the FB-classifier uses kernel density estimation [8]. Kernel estimation of a continuous attribute is the average of kernel densities of a continuous attribute xik in the training points associated with class cj, as depicted by Eq. 4. Let Eq. 3 be P(xi|cj)=g(xik, µj, σj), which is the density estimation on a continuous attribute by using the NBclassifier. Then, the kernel density estimation of a continuous attribute is evaluated as follows:
331
ISBN: 972-99353-6-X © 2005 IADIS
P ( xi | c j ) =
1 ∑ g ( xik , µ k , σ j ) n k
(4)
Single Gaussian estimation
Kernel estimation
xik Figure 2. illustrates the Single Gaussian estimation versus the kernel estimation of the Flexible Bayesian method over a histogram of a continuous attribute.
Where k ranges over the training points of the continuous attribute xik in class cj. Also, µk=xik. And σj=1/(Nj)0.5, where Nj is the number of training instances observed with class cj. A graphical comparison between the single Gaussian estimation and kernel estimation of the density of a continuous attribute is shown in Fig. 2. Note that the kernel density estimation is more local than that of the single Gaussian estimation.
3. IMAGE CLASSIFIER Our image classifier can learn high-level semantics of classes using low-level image features, specifically color distributions. Using the FB-classifier, the image classifier learns the information of classes from the training set and then it estimates the class membership of a new image instance. As shown in Fig. 3, first, the system extracts the color histograms of images from the training set. Each feature vector (color histogram) contains the RGB values that represent the color distribution of the corresponding image. The RGB color space is quantized into 64 colors, which is enough to represent the different colors in an image for the purpose of classification as we have concluded from our experiments. Note that each feature vector in the training set includes the class information to which its corresponding image is associated. Next, the training module utilizes these feature vectors to train the system and thus generate a model for each of the n classes that exist in the training set based on Eq. 4. When a new image instance is to be classified, first, its color histogram is extracted in the same way as that of the training set. Then, the classification module computes the conditional probability of each of the n classes using the FB-classifier given the new image instance. As a result, the new image instance is assigned to the class with the highest probability. Training Training set
class 1 Feature Extraction
Classification class 2
New Image Instance class n Figure 3. The image FB-classifier system layout.
332
IADIS International Conference on Applied Computing 2005
Beach
Garden
Tomato
Banana
Desert
Snow
Sunset
Copper
Tiger
Wood
Rose
Gorilla
Figure 4. 12 representative images of the 12 classes in the training set
4. EXPERIMENTS
100 90 80 70 60 50 40 30 20 10 0
Naïve
Gorilla
Tiger
Wood
Copper
Tomato
Rose
Banana
Snow
Sunset
Desert
Beach
Flexible
Garden
Accuracy
We implemented the system using C++ on a PC running Windows XP with Pentium 1.3GHz CPU and 256MB RAM. We have used a database of 420 images collected from various sources on the Internet. The images are of size 256 × 256 and stored as bmp images. The collection contains 12 different classes (Beach, Garden, Desert, Snow, Sunset, Rose, Banana, Tomato, Copper, Tiger, Wood, Gorilla) as shown in Fig. 4. For each of the classification experiments, we randomly selected 10 images from each class as a test set. To have more accurate results, we considered 3 different test sets that are randomly selected. Note that each test set contains different randomly selected images. The accuracies shown below are the average of the 3 test sets. To show the superiority of the FB-classifier over the NB-classifier, we conducted the classification experiment using both classifiers. Fig. 5 compares the classification accuracy of the 12 classes. Notice that the FB-classifier is significantly more accurate than the NB-classifier for all classes except for Snow and Banana classes. As we mentioned in Section 2, NB-classifier is expected to perform better for classes whose training datasets exhibit normal distribution. On the other hand, the FB-classifier is expected to perform better for classes that violate the normality assumption. By closely examining the attributes of the color histograms of training datasets of both classes (Snow and Banana), in which the FB-classifier was less accurate than the NB-classifier, we noticed that these attributes exhibit somewhat normal distributions
Class
Figure 5. Comparison of classification accuracy between FB-classifier and NB-classifier.The high accuracy in
performance of the FB-classifier over the NB-classifier comes at the expense of an increase in the computational cost during the classification phase [8]. Let the dimensionality of the feature vector be d, the number of the training images be t, and the number of test instances to be classified be h, then the computational cost for the classifying the h test instances of the NB-classifier is O(hd) and that of the FBclassifier is O(hdt).
333
ISBN: 972-99353-6-X © 2005 IADIS
We have compared the FB-classifier to another well known classifier that is the KNN classifier to show the superiority of our classifier. Before presenting the results of the classification accuracy comparison on our image database, we introduce the KNN-classifier and the parameters used in the experiment. K-Nearest Neighbors classifier (KNN-classifier) assumes that each instance in the training set include the class information. When classification is to be conducted for a new image instance, the distance between the new instance and each item in the training set is computed. Only the K closest items are considered, as shown in Figure 6. Then, the new instance is classified in the majority class among the K closest items. For more details on the KNN classifier, refer to [4]. In our experiment, we set K = 10. 10 9 8 7
i
6 5 4 3 2 1 1
2
3
4
5
6
7
8
9 10
Figure 6. Illustrates classification of a new instance i using KNN-classifier. Here, K=3 and i will be classified in the class to which most of the closest items in the K set are members.
Figure 7 compares the classification accuracy of FB-classifier and KNN-classifier. Clearly, the FBclassifier is more accurate than the KNN-classifier for most of the classes. The FB-classifier is more accurate than the KNN-classifier in 11 classes and less accurate in only one class (Snow). Note that the Snow class is one of the two classes in which the FB-classifier was less accurate than the NB-classifier due to the tendency of their attributes to have normal density distributions. Flexible
KNN
100 Accuracy
80 60 40 20 Gorilla
Wood
Tiger
Copper
Tomato
Banana
Rose
Sunset
Snow
Desert
Garden
Beach
0
Class
Figure 7. Comparison of classification accuracy of FB-classifier and KNN classifier.
5. CONCLUSION In this paper, we presented a new classifier, called FB-classifier, for image databases. Our classifer is more accurate than its counterpart the NB-classifier in 10 out of 12 classes. The main reason for the high classification accuracy of the FB-classifier as compared to its rivals is that the FB-classifier abandons the normality assumption since the densities of attributes of image data do not distribute normally as we have found from our experiments. Also, our classifier is more accurate than the widely used classifiers KNNclassifier in 11 out of 12 classes.
334
IADIS International Conference on Applied Computing 2005
REFERENCES [1] M-L Antonie, O.R.Zaiane, A.Coman. "Application of Data Mining Techniques for Medical Image Classification". 2nd Int'l Workshop on Multimedia Data Mining, USA, 2001. [2] B.Bradshaw. “Semantic Based Image Retrieval: A Probabilistic Approach”. ACM Multimedia, 2000. [3] D.Deng. “Mining Visual Concepts for Image Retrieval: A Case Study”. 5th Biannual Conference on Aritificial Neural Networks and Expert Systems, Dunedin, Nov. 2001, pp.46-51. [4] M.H.Dunham. “Data Mining: Introductory and Advanced Topics”. Prentice Hall, Pearson Education, Inc. 2003. [5] M.Flickner, H.Sawhney, J.Ashley, Q.Huang, B.Dom, M.Gorkani, J.Hafner, D.Lee, D.Petkovic, D.Steele, P.Yanker “Query By Image and Video Content: The QBIC System”. IEEE Computer Magazine, Sep. 1995. [6] T.Gevers, F.Aldershoff, A.W.M.Smeulders. “Classification of Images on Internet by Visual and Textual Information”. Proc. of SPIE Internet Imaging, vol.3964, 1999. [7] J-H.Lim “Explicit Query Formulation with Visual Keywords”. ACM Multimedia, Oct. 2000. [8] G.H.John, P.Langley. "Estimating Continuous Distributions in Bayesian Classifiers", 11th Conf. on Uncertainity in Artificial Intelligence, 1995. [9] W.Y.Ma, B.S.Manjunath. “Netra: A Toolbox for Navigating Large Image Databases”. IEEE Int’l Conf. On Image Processing (ICIP), Oct. 1997. [10] S.Mehrotra, Y.Rui, M.Ortega, T.S.Huang. “Supporting Content-Based Queries over Images in MARS”. IEEE Int’l Conf. On Multimedia Computing and Systems, 1997. [11] J.R.Smith, S-F.Chang. “VisualSEEK: A fully automated content-based image query system”. ACM Multimedia., Nov. 1996. [12] C.P.Town, D.Sinclair. “Content Based Image Retrieval Using Semantic Visual Categories”. Technical Report TR2000-14, AT&T Laboratories Cambridge, 2000. [13] A.Vailaya, A.Jain, M.Figueiredo, H.J.Zhang. “Content-Based Hierarchial Classification of Vacation Images”. IEEE Int’l Conf. On Multimedia Computing and Systems, Jun. 1999. [14] A.Vailaya, A.Jain. "Incremental Learning for Bayesian Classification of Images", IEEE Int’l Conf. on Image Processing. October 1999 . [15] W.Wang, Y.Song, A.Zhang. “Semantic-Based Image Retrieval By Region Saliency”. Int’l Conf. On Image and Video Retrieval, July, 2002. [16] O.Zaiane, J.Han, Z-N.Li, J.Hou. “Mining Multimedia Data”. CASCON’98: Meeting of Minds, pp 83-96, Toronto, Canada , Nov.1998.
335
