an efficient statistical content-based color image

An Efficient Statistical Content-Based Color Image Indexing and Retrieval Mohammad A. Al-Jarrah, and Faruq A. Al-Omari, Member, IEEE

Abstract-- The world has recently witnessed a tremendous development in the various fields of information technology. Traditional databases do not provide a satisfactory environment for variant media types such as audio video and still images. An efficient statistical image indexing technique has been developed. The mean, variance, and skewness of the image color histogram distributions were used as a feature vector to tag images in the database system. The RGB color space was used. Therefore, the feature vector formed from the mean variance, and skewness of the three basic components of the color space. It has been shown using statistical analysis that the statistical feature vector entries give sufficient knowledge about the histogram distribution for image indexing and retrieval systems. Hence, The proposed system has shown its superiority over traditional histogram based systems in term of feature index properties and length as well as performance. Index Terms-- Content-based image indexing, image retrieval, query by statistical features, image database systems.

I. INTRODUCTION

I

n the new era of information technology, traditional databases do not provide a satisfactory environment for variant media types such as audio video and still images [10,11,12]. Instead of small text tags, search engines of This work was supported in part by Yarmouk University, Irbid, Jordan and by the Hijjawi Scientific Foundation, Amman, Jordan. M. A. Al-Jarrah is with the Computer Engineering Department, Yarmouk University, Irbid 21163, Jordan (telephone: +962 2 7276277 ext. 4408, e-mail: [email protected]). F. A. Al-Omari is with the Computer Engineering Department, Yarmouk University, Irbid 21163, Jordan (telephone: +962 2 7276277 ext. 4405, e-mail: [email protected]).

ICITA2002 ISBN: 1-86467-114-9

databases should emphasize on the content of video streams and still images. For efficient image retrieval, the image database should be processed to extract a representing feature space vector for each member image in the database. The feature space vector resembles a tag for the corresponding image in the database based on which image retrieval is performed. The crucial element in the performance of the retrieval process is the selection of the entries of the feature space vector. Several key image features have been utilized by many researchers in this field [1,2,4,5,6,7,10,16,17]. Chief among these features are spatial-based features, colorbased features, texture-based features, and motion-based features. The complexity of algorithms used to extract these features varies according the nature of the processed images and the type of features. Spatial- and texture-based features require some gained knowledge about the objects contained in the image. Therefore, edge and boundary detection operators, and advanced image segmentation techniques are required to extract the key features. For video clips, motion-based features require more advanced approaches like optical flow techniques. On the other hand, colored-based techniques generally require more simplified approaches. No particular prior knowledge about the image content is required. However, these techniques do not incorporate spatial distribution of colors in the image. In fact, the nature of problem in hand in which all “close” hits are retrieved with different matching scores, softened the burden on using more tightened pattern recognition techniques. Therefore, colored-based approaches have been widely utilized [1,2,3,4,16,17]. For color histogram extraction, different color spaces [1] have been utilized such as RGB, HSV, and LUV representation. Other researches used multi-scale transforms to extract a discriminating vector for each image including wavelet transform, [2,4,9,17]. Although histogram-based techniques have been widely considered to be efficient color indexing

techniques, for remote search engines and distributed database systems, it involves a relatively large amount of information that should be exchanged among system components via its messaging system. Moreover, to enhance the performance of the retrieval system, researchers tend to add more features to the histogram vector to include local properties [14]. Therefore, larger amount of information, intuitively the index vector data, should be exchanged among the system components. In this paper we proposed content-based image indexing technique using statistical properties of the color distribution in the image, including first, second, and third order statistical moments. In this technique the index vector is very small compared to the histogram vector. The results show that our proposed algorithm performance is similar to color histogram techniques. The remaining of this paper addressed the fundamentals of image indexing in section (2), the feature vector definition in section (3), the proposed statistical technique in section (4). Discussions and experimental results are presented in section (5), and finally conclusions in section (6).

II. IMAGE INDEXING For efficient image retrieval, the image database should be processed to extract a representing feature space vector for each member image. The feature space vector resembles an index for the corresponding image in the database based on which image retrieval is performed. Therefore, for a 2-D image I(r,c), a mapping function f is defined that maps the image data into an n-dimensional feature space vector Vn such that,

f : I (r ,c ) → V = {v1 , v 2 ,L , v n } .

(1)

A similarity measure between a query image, Q(r,c), and the database image, I(r,c), is evaluated to rank the closeness of the query image to I(r,c). This measure (R) maps the index vectors into a Boolean domain, such that for given a query image Q,

R (VQ , VI ) → {SIMILAR , DISSIMILAR} L∀ I ∈ D

(2)

where VQ and VI are the index vectors for images Q and I, respectively, and D is the image database. Image histogram is one of the very basic representing features of colored images in which the color intensity distribution in an image is portrayed. Many researchers [1,5,7,8,13,14] have used this approach in many different ways to index images. However, an RGB true-colored image results in three 1-D histograms consisting of 256 entries each. This makes the index vector relatively long.

III. FEATURE VECTOR DEFINITION Normalized histograms can be viewed as probability density functions (pX(x)) that can be described using central moments. Where the ith central moment, Mi, of a normalized histogram, HN, is defined as

Mi

=

∞

∫x

i

⋅ H N ( x ) dx .

(3)

−∞

Statistical analysis has shown that the first few moments give enough knowledge about the distribution. Gaussian distribution for example is fully described in terms of its first and second moments namely mean and variance. Therefore, we have used the first-, second-, and third-order statistics of each normalized color histogram as key features representing that histogram. More precisely, the mean (µ), variance (σ2), and skewness (s), for each color in the RGB image were used. Hence, the feature index vector, Vn, is 9-dimensional rather than 768dimensional when the entire histogram is taken as a feature vector.

IV. STATISTICAL INDEXING AND RETRIEVAL TECHNIQUE The proposed indexing and retrieval system is divided into two main stages as shown in Figure (1). In the first stage, the proposed statistical feature vector defined in section (3) is extracted once for all images in the database, D, and saved as a tag for all corresponding images. In the second stage, a query image, Q, is processed in the same manner and a query statistical feature vector, V nQ, is extracted. The query statistical feature vector is compared against all entries in the feature database to map all images in the database into similarity Boolean domain. The comparison is carried out using a similarity measure that ranks the closeness of the query image to each database image. The following subsection introduces a similarity measure that is used in the proposed system.

A . Similarity Measure The matching module in the developed image indexing and retrieval system illustrated in Figure (1) transforms the index vectors of the query image and the database image into a rank ranging from –1 to 1. This rank measures the closeness of the corresponding two images. For this purpose, we defined this measure (S) as

SI

=

VQT V I + V IT VQ T Q

T I

V VQ + V V I

,

∀ I ∈D .

(4)

where VQ and VI are the query and database image feature vectors, respectively. According to this definition, when the two feature vectors are the same SI comes to unity. Hence the two images are completely similar. On the other hand, when the two vectors are completely opposite, SI becomes –1, which means that the two images are completely dissimilar. The next step in the matching process involves mapping the obtained ranks (SI)’s into the Boolean domain defined in equation (2) according to a user defined threshold rank (ST). Where equation (2) could be rewritten as

R (S I (V Q ,V I ), S T ) → {SIM , DISSIM L Image Database

I(r,c)

RGB Histograms

Q(r,c)

RGB Histograms

(5)

∀I∈D

HIR(P) HIG (P) HIB (P)

Key Feature Extraction

VI9

Feature Database

Database Image Indexing

Query Image

}

HQR(P) HQG (P) HQB (P)

Key Feature Extraction

VQ9 Matching

Query Image Indexing Top K Retrievals

Figure 1: A block diagram of the statistical image indexing and retrieval system.

The search and retrieving engine portrays all images in the database categorized as SIMILAR.

B . Properties of the Extracted Statistical Feature Vector The entries of the feature vector posses several properties that make these features superior to using the histogram as a feature vector. Some of these properties are retained in the histogram data, but some others are unique to the entries of the feature vector described above. Chief among these properties are translation, rotation and reflection. The histogram is invariant to these operations [15]. Hence, the mean, variance, and skewness remain unchanged. Therefore, the statistical feature vector remains unchanged as well. However, when point operations are performed on an image its histogram is modified in a predictable way. As a matter of fact, if f is a mapping function that resembles a linear point operation (PB=a.PA+b) performed on an image. Then the modified image histogram is related to the original histogram such that [15]

H B (PB ) =

1  P −b  H A B . a  a 

(6)

where PB and PA are the resulting and original image pixels, respectively. Consider two special cases of point operations, namely, adding a constant, and taking the negative of the image. We’ll discuss the direct influence of each of these operations on the traditional image retrieval schemes based on histograms versus the influence on the proposed scheme. If a constant, b, is added to the color or contrast of the original image data, then the resulting image histogram would be shifted left or right by the amount of the constant, b. which means that a direct comparison of point by point between the two histograms of the original and modified images would result in a relatively high difference depending on the constant, b. However, the shifted histogram will have the same variance and skewness and only a shift on the mean value will occur. A minor change could occur on the variance and skewness in the case of saturation around the upper or lower pixel levels. Therefore, retrieval based on direct histogram comparison will lead to some misleading results, whereas retrieval based on the extracted features of the histogram will be more accurate. On the other hand, when the negative of the image is taken, then the modified histogram will be a mirror reflection of the original histogram around the mid range of the pixel value space. This keeps the variance and skewness values the same. However, the point by point comparison of two histograms will definitely lead to big difference. Therefore, more reasonable results will be obtained using the proposed scheme as compared to direct histograms comparison.

V. EXPERIMENTAL RESULTS AND DISCUSSION To validate the developed indexing and retrieval technique, a collection of 2500 digital images in different file formats (JPEG, GIT, TIF, BMP, and MPEG video frame) and different image sizes were assembled in a database. Several types of images were included like natural scenes, animal pictures, birds, humans, cartoon characters, city scenes, electronic devices, and others. The true-RGB color space was used, i.e. 256 quantization levels for each color. The database was then processed through the image database indexing stage of the developed system and a feature database was created. At the same time, a second feature database was created with tags corresponding to the three normalized color histograms depicted as feature vectors. The purpose of the second feature database is to perform a comparison between the results obtained using the developed technique with the traditional histogram-based techniques. The similarity measure used to retrieve images based on the developed scheme was as defined in the aforementioned discussion

whereas the similarity measure used to retrieve images based on the traditional histogram-based technique was the Euclidean distance [5,8,14] between the two feature vectors in hand. That means a similarity index of zero corresponds to full match where as a similarity index of unity corresponds to full mismatch. The threshold was depicted so to that enough similar images were retrieved to demonstrate the goal behind the experiment. In both systems, when an existing image in the database is selected as a query image, the same image was retrieved as a top match. However, when the image is preprocessed the two systems behave differently. The image processing toolbox of MATLAB® was used to pre-process images. Two main experiments were conducted to validate the developed technique. In the first experiment, an image from the database was selected. The contrast of the image was altered by adding 15% and 50% brightness to the original image. Figures (2) and (3) portrays the results for two sample query images obtained from both systems when 15% brightness is added. For the first query image, the top is the query image obtained by adding 15% brightness to the original database image. Second row corresponds to the first five hits obtained from the developed system. The original image appears in the first place with a similarity rank of 0.999764. The rank for the other four hits was 0.997191, 0.996538, 0.995271, and 0.993525, respectively. The third row corresponds to the first five hits obtained from the traditional histogram-based system. The ranks are ٠٫٢٢٨٣١٨, ٠٫٣١٠٧٨٩, ٠٫٣٢١٨١١٥, ٠٫٣٢١٩٠٢, and ٠٫٣٢٣٢١٤٥. The original image came in the fifth place. For the second query image, the top is the query image obtained by adding 15% brightness to the original database image. Second row corresponds to the first five hits obtained from the developed system. The original image appears in the first place with a similarity rank of 0.999326. The rank for the other four hits was 0.987871, 0.969763, 0.965729, and 0.964964, respectively. The third row corresponds to the first five hits obtained from the traditional histogram-based system. The ranks are ٠٫٢٨٩٦٨٦, ٠٫٣١٥٢٦٦, ٠٫٣٢٠٥٠٤, ٠٫٣٣١٦٦٣, and ٠٫٣٣٥٣٣٧ . Whereas the original image came in the ٥٠th place. As can be seen in the figures, the developed technique performs much better than the traditional histogram-based technique. Hence, this developed technique is more robust against contrast changes in the original image. However, when 50% brightness is added to the original image of Figure (2), the developed system retrieved the original image in the 16th place with a rank of 0.971444, whereas the traditional histogram-based system retrieved the original image in the 102nd place with a rank of 0.791669. When the original image of Figure (3) was processed by adding 50% brightness, the developed system retrieved it as 103rd hit as compared to 136th hit in the histogram-based system.

In the second experiment, however, the negative of the images was taken. Then the negative images were passed through both systems. Table (1) demonstrates the behavior of both systems in response to all query images in all conducted experiments. The table below emphasizes the robustness and reliability of the proposed technique as compared to traditional histogram-based techniques when images are altered through linear point operations. The tabulated results in Table (1) agree to a great extent with the analysis and discussion aforementioned in section (4.2). Furthermore, these results emphasize the efficiency of the developed system by providing an overall performance better than what could be achieved using traditional histogram-based techniques. On the other hand, the amount of information that should be exchanged among system components via its messaging system has been reduced dramatically. Table 1: Experimental Retrieval Results Obtained on PreProcessed Images Using Both the Developed System and the Traditional Histogram-Based System CONDUCTED EXPERIMENT

DEVELOPED SYSTEM ( Hit# , Rank)

TRADITIONAL SYSTEM ( Hit# , Rank)

1st Hit, S=0.999764

5th Hit, D= 0.323215

16th Hit, S=0.971444

102nd Hit, D=0.791669

1st Hit, S=0.999326

50th Hit, D=0.455543

103rd Hit, S=0.753409

136th Hit, D=0.786767

1st Hit, S=0.999646

62nd Hit, D=0.881424

15% Brightness Increase on the Tiger Image 50% Brightness Increase on the Tiger Image 15% Brightness Increase on the River Image 50% Brightness Increase on the River Image Negative(Tiger)

VI.

CONCLUSIONS

In this paper, we developed a statistical content-based image indexing and retrieval system. In this system a new statistical index feature vector has been introduced based on the first-, second-, and third-order statistical moments. The introduced feature vector is sufficient and compact, which is suitable for distributed database systems. Several experiments have been conducted. The results obtained from these experiments show the reliability and robustness of the developed system. Moreover, better results were obtained using the developed system compared to traditional histogram-based systems.

ACKNOWLEDGMENT Authors would like to extend their sincere gratitude to Eng. M. Al-Shar’e for his great effort in collecting and preprocessing images and for his participation in implementing the code.

REFERENCES [1] D. Park, J. Park, T. Kim, and J. Han, “Image indexing using weighted color histogram,” Proc. 10th Int. Conference on Image Analysis and Processing, September, 1999 [2] E. Albuz, E. Kocalar, ad A. Khokhar, “ Scalable color image indexing and retrieval using vector wavelets,” IEEE Trans. on Knowledge and Data Eng., vol. 13, no. 5, pp. 851-861, Sep. 2001. [3] D. Slater and G. Healy, “The illumination-invariant matching of deterministic local structure in color images,” IEEE Trans. on PAM, vo1. 9, no. 10, pp. 1146-1151, October 1997. [4] N. Sebe et al, “Color indexing using wavelet-based salient points,” Proc. of the IEEE Workshop on Content-based Access of Image and Video Libraries, June 2000. [5] J. Huang et al, “Image indexing using color correlograms,” Proc. of the 1997 IEEE conf. on Computer Vision and Pattern Recognition, pp. 762-768, 1997. [6] T. Shih et al, “An efficient color-spatial indexing and retrieval scheme for image database,” 7th IEEE Int. Conf. on Parallel and Distributed Systems: Workshops, 2000. [7] P. Mlsna and J. Rodriguez, “Efficient indexing of multicolor sets for content-based image retrieval,” Proc. of the 4th IEEE Southwest Symposium on Image Analysis and Interpretation, April, 2000 [8] A. K. Jain, A. Vailaya and W. Xiong, “Query By Video Clip,” Multimedia Systems: Special Issue on Video

Libraries, vol. 7, no. 5, pp. 369-384, 1999. [9] Y. Chang, W. Zeng, I. Kamel, and R. Alonso, “Integrated image and speech analysis for content-based video indexing,” Proc. of the 1996 IEEE Int. Conf. on Multimedia Computing and Systems, June 1996. [10] A. Smeulders et al, “Content-based image retrieval at the end of the early years,” IEEE Trans. on PAMI, vol. 22, no. 12, Dec. 2000. [11] R. Brunelli, O. Mich, and M. Modena, “A survey on the automatic indexing of video data,” J. of Visual Communication and Image Representation, vol. 10, pp. 78112 1999, [12] R. Steinmetz and K. Nahrstedt Multimedia: Computing, Communications and Applications, Prentice Hall, New Jersey, 1995. [13] A. K. Jain and A. Vailaya, “Image retrieval using color and shape,” Pattern Recognition Journal, vol. 29, pp 1233-1244, Aug. 1996. [14] Z. Zheng and C. Leung, “Automatic image indexing for rapid content-based retrieval,” Proc. of the IEEE int. Workshop on Multi-Media Database Management Systems, pp. 38-45, 1996. [15] K. Castleman, Digital Image Processing, Prentice Hall, New Jersey, 1996. [16] Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, November 2000. [17] Yossi Rubner, Leonidas J. Guibas, and Carlo Tomasi. The Earth Mover's Distance, Multi-Dimensional Scaling, and Color-Based Image Retrieval. Proceedings of the ARPA Image Understanding Workshop, New Orleans, LA, May 1997, pages 661-668.

Figure 3: Retrieval results obtained form the developed and the traditional system, respectively. Figure 2: Retrieval results obtained form the developed and the traditional system, respectively.