Content-Based Remote Sensing Image Retrieval Using Image Multi ...

Content-Based Remote Sensing Image Retrieval Using Image Multi-feature Combination and SVM-Based Relevance Feedback Lijun Zhao1,2, Jiakui Tang1, Xinju Yu1,2, Yongzhi Li1,2, Sujuan Mi1,2, and Chengwen Zhang1,2 1

Yantai Institute of Coastal Zone Research, Chinese Academy of Sciences, Yantai, 264003, P.R. China 2 Graduate University of Chinese Academy of Sciences, Beijing, 100049, P.R. China [email protected]

Abstract. In order to narrow the gap between user query concept and low-level features in content-based image retrieval, the support vector machine (SVM) based relevance feedback technique is introduced. However, remote sensing images are one kind of images with special spectral features. Relevance feedback mechanism hasn’t been widely used in content-based remote sensing image retrieval (CBRSIR). Therefore, to test the effectiveness in CBRSIR, a SVM based relevance feedback algorithm based on SVM classification theory is adopted in CBRSIR to boost remote sensing image retrieval accuracy. The experimental results show that the SVM-based relevance feedback algorithm performs well in remote sensing image retrieval and has good potential in practical applications.

1

Introduction

Facing huge image resources, how to retrieve accurately and efficiently is an urgent problem to solve. From text-based image retrieval in late 1970s to content-based image retrieval (CBIR) in early 1990s, qualitative changes have taken place in image retrieval. CBIR extracts information clues from the media content, using similarity matching technology and relevance feedback, so as to reduce search time for retrieval in large database. As one kind of images, the remote sensing (RS) image has its own characteristics. Besides color, shape, texture features in common images, it also has spectral feature, spatial feature and metadata. The remote sensing image is also multi-dimensional, large-scale and information-abundant. Therefore, study on content-based remote sensing image retrieval (CBRSIR) becomes particularly important. However, most of the studies at present mainly focus on the methods of different visual information extraction and their effects on CBRSIR [11, 2]. Few researches were done on retrieval using combination features. Moreover, to narrow the gap between low-level features representation of images and user’s high-level semantic concepts, relevance feedback is employed to learn user’s query concepts [6], with support vector machine (SVM) Z. Qian et al. (Eds.): Recent Advances in CSIE 2011, LNEE 124, pp. 761–767. © Springer-Verlag Berlin Heidelberg 2012 springerlink.com

762

L. Zhao et al.

based relevance feedback being a very popular one [10]. However, this relevance feedback algorithm hasn’t been widely applied in CBRSIR. Therefore, to test the effectiveness of SVM-based relevance feedback method in CBRSIR, combination remote sensing image features are used and SVM-based relevance feedback is adopted in the paper.

2

Features Extraction in Remote Sensing Images

Color and texture are mainly used in common image retrieval, and they are still useful in remote sensing image retrieval. Besides, some contents in remote sensing image like spectral feature should also be emphasized and used. 2.1

Color and Texture Features

The color feature is one of the most commonly used features in image retrieval because of its simplicity and rotational invariance. Color images are usually represented by RGB color space. And the most commonly used measurement is to form feature vector by mean and variance in R, G, B color space respectively. Many natural images include textures which have become another type of important and useful visual cues for image retrieval. Compared with common images, texture of remote sensing image is more complex and abundant, and is more sensitive to different entities. In recent years, a variety of texture analysis methods have been studied [7]. At present, there are three widely used texture feature extraction methods, including Gabor filters [8], wavelet transform [1], and co-occurrence matrix [4]. 2.2

Spectral Feature

Spectral feature is the physical basis for remote sensing images, and it is also one of the most important characteristics different from common images. Spectral feature mainly embodies in multi-spectral and hyper-spectral remote sensing images, and through spectral feature extraction, it can be effectively used in CBRSIR. Take Landsat TM image as an example, each of its seven bands has corresponding application field and different band combinations can form some new useful features [3]. There are many useful multi-spectral features, such as normalized difference of vegetation index (NDVI), normalized difference of building index (NDBI) [3], normalized difference of water index (NDWI) [3].

3 3.1

Support Vector Machine Based Relevance Feedback Support Vector Machine

SVM is a state-of-the-art machine learning technology proposed by Vladimir N. Vapnik, and it is based on statistical learning theory. Since it has strong theoretical foundations based on Structural Risk Minimization instead of Empirical Risk Minimization [9], SVM has superior generalization performance compared with some traditional learning methods.

Content-Based Remote Sensing Image Retrieval

763

Given the training samples T={(x1, y1),…, (xl, yl)} ∈ (Rn×Y)l, training data xi ∈ Rn, their class labels yi ∈ Υ = {1, −1}, i = 1,…,l, where l is the number of training samples, the separating hyperplane is constructed by solving the following optimization problem: l 1 2 w + C ∑ ξi w ,b ,ξ 2 i =1 subject to yi ((w ⋅ Φ ( xi )) + b ) ≥ 1 − ξ i , i = 1," ,l ,

minimize

(1)

ξ i ≥ 0 , i = 1," ,l .

In (1), Φ (·) is a mapping function to map training data x into a high-dimensional space, where “·” is an inner product. The parameter C is the penalty parameter which is to balance the model complexity and training error, and ξi is the slack parameter which allows for some training samples to be within the margin. The optimization of SVM is usually solved in a dual form as follows:

(

)

l 1 l l ∑ ∑ yi y j K xi , x j α i α j − ∑ α j 2 i =1 j =1 j =1

minimize α

l

subject to ∑ yi α i = 0 ,

(2)

i =1

0 ≤ α i ≤ C , i = 1," ,l . In (2), K(xi, xj) is a kernel representing for (Φ (xi)·Φ (xj)). The kernel function K has various types, such as linear, polynomial, radial basis function (RBF), and sigmoid. By solving (2), α* = (α1*,…, αl*)T can be obtained. Choose a component αj* that is within the range (0, C), and, thus, the decision hyperplane can be represented as: f (x ) = ∑ yi α*i K (xi , x ) + b* , l

i =1

l

(

b = y j − ∑ yi α*i K xi , x j *

i =1

3.2

)

(3) .

SVM-Based Relevance Feedback

SVM-based relevance feedback mechanism is the use of SVM theory in relevance feedback. SVM assumes the retrieval results as training samples, and based on the retrieval results, user selects the relevant images and the irrelevant ones. The relevant ones form positive sample set, and the irrelevant ones form negative sample set. After studying the training sample set by using SVM, a SVM classifier f(x) will be obtained to express user’s retrieval goal. To the image Ii in the database, a score can be calculated as score(Ii) = f(xi). The bigger the score(Ii) is, the closer Ii will approach to the query image. Sort the scores of all the images in descending order. And then, top k images are returned and step into next feedback round. Repeat the process until the user satisfies [10, 12].

764

L. Zhao et al.

As a special kind of images, the relevance feedback process of remote sensing images is similar to that of common images. Therefore, SVM can also be used in the relevance feedback process of CBRSIR.

4 4.1

Experiments Experimental Environment

To evaluate the effectiveness of SVM in CBRSIR, scenes of different times and places of remote sensing images are selected. All the scenes of remote sensing images are cropped with non-overlapping regions to the size 512×512. After filtration of useless information, 419 sub-images are generated to construct a remote sensing image database, including three categories, that is, vegetation, waters and rivers. The features used are described in Sect. 2. In order to evaluate the performance of the retrieval results, two performance measures average precision and Avg-p [12] are used. The value of Avg-p usually indicates the tightness of the relevant images in the retrieval sorting results of front positions. The larger the value is, the better the performance will be. 4.2

Parameter Selection

In the experiment, to reduce user’s fatigue, the number of returned images is set k=20, and the time of relevance feedback is set four times to test the improvement of retrieval performance. The kernel function in SVM is RBF kernel shown as follows:

(

)

(

)

K xi , x j = exp - λ || xi - x j ||2 .

(4)

In (4), λ is a kernel parameter. The SVM classification algorithm is performed by LibSVM toolbox (http://www.csie.ntu.edu.tw/~cjlin/libsvm). Grid search is adopted to find the optimal parameters of C in SVM and λ in RBF kernel [5]. 4.3

Experimental Results

For an objective performance evaluation of SVM-based relevance feedback in CBRSIR, 100 remote sensing images are randomly selected as query images. For each query, four rounds of relevance feedback are performed using SVM with color texture and spectral features. Average precision and Avg-p for the retrieval of the 100 query images are calculated as the overall precision and Avg-p. Table 1 is comparison of retrieval precision using only common image features and spectral feature added image features. Among all the features, color and texture are common image features, while spectrum is remote sensing image feature. Under the circumstances of only using these two kinds of features separately, the precision of color and texture features is a little higher than that of spectral feature by 7.1%, but both of them are less than 0.5. However, the combination of color texture and


765

spectrum further improves the precision, which indicates that spectral feature has positive effect on retrieval result. To further boost the retrieval performance using combination features, SVM-based relevance feedback is introduced. Fig.1 gives the average precision change of CBRSIR using SVM. And Fig.2 shows the change of Avg-p with SVM-based relevance feedback. All the figures indicate that average precision boosts with relevance feedback going on, especially in the first two rounds of relevance feedback. Meanwhile, Avg-p goes up correspondingly, and its value gets closer to 1, which means that SVM-based relevance feedback can return relevant remote sensing images at front positions compactly. Thus, SVM-based relevance feedback mechanism can do well in CBRSIR and achieve good performance. Fig. 3 visually shows the results after initial retrieval and first feedback round. Table 1. Comparison of Average Precision Using Different Kinds of Features Average Precision of Initial Retrieval

Spectrum

42.05%

Color+Texture

49.15%

Color+Texture+Specturm

50.30%

Precision

Features

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

Average Precision

0

1 2 3 Number of Relevance Feedback

4

Fig. 1. Average precision change using SVM-based relevance feedback 1 0.95 0.9

Avg-p

0.85

Avg-p

0.8 0.75 0.7 0.65 0.6 0.55 0.5 0

1 2 3 Number of Relevance Feedback

4

Fig. 2. Avg-p change using SVM-based relevance feedback

766

L. Zhao et al.

(5-a) initial retrieval result

(5-b) retrieval result after first feedback round

Fig. 3. Results after initial retrieval and first feedback round. The upper left image in (a) is the query image and the similarity descends from left to right, top to bottom.

5

Conclusions and Future Work

In the paper, the combination of multi-feature including color, texture and spectrum feature is used to perform remote sensing image retrieval. It is proved that spectral feature plays an important role in CBRSIR. And then, SVM-based relevance feedback is applied, using SVM decision hyperplane function to mine user’s semantic information. The experimental results show that based on spectrum added image features, SVM-based relevance feedback can greatly improve the retrieval performance in CBRSIR, and it is proved to be effective in remote sensing image retrieval. Although the experimental results showed that SVM-based relevance feedback using spectral feather is of high performance in TM multi-spectral images retrieval, more studies and retrieval experiments on other multi-spectral and hyper-spectral remote sensing images should be carried out in the future work.

References 1. Balamurugan, V., AnandhaKumar, P.: An integrated color and texture feature based framework for content based image retrieval using 2D Wavelet Transform. In: International Conference on Computing, Communication and Networking, pp. 1–16 (2008) 2. Cheng, Q.M., Yang, C.J., Chen, F.X., et al.: Application of M-band wavelet theory to texture analysis in content-based aerial image retrieval. In: 2004 IEEE International Geoscience and Remote Sensing Symposium, vol. 3, pp. 2163–2165 (2004) 3. Du, P.J., Chen, Y.H., Tang, H., et al.: Study on content-based remote sensing image retrieval. In: 2005 IEEE International Geoscience and Remote Sensing Symposium, vol. 2, pp. 707–710 (2005) 4. Haralick, R.M., Shanmugam, K., Dinstein, I.: Texture features for image classification. IEEE Transactions on Systems, Man, and Cybernetics 3(6), 610–621 (1973) 5. Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification. Department of Computer Science and Information Engineering, National Taiwan University (2004)


767

6. Karthik, P.S., Jawahar, C.V.: Analysis of relevance feedback in content based image retrieval. In: 9th International Conference on Control, Automation, Robotics and Vision, pp. 1–6 (2006) 7. Liu, L., Kuang, G.Y.: Overview of image textural feature extraction methods. Journal of Image and Graphics 14(4), 622–635 (2009) 8. Lu, L.Z., Liu, R.Y., Liu, N.: Remote sensing image retrieval using color and texture fused features. Journal of Image and Graphics 9(3), 328–332 (2004) 9. Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998) 10. Wang, X.J., Yang, L.L.: Application of SVM relevance feedback algorithms in image retrieval. In: International Symposium on Information Science and Engineering, vol. 1, pp. 210–213 (2008) 11. Xie, H.S., Wang, L.G.: Comparison and analysis of color histogram in content-based remote sensing image retrieval. Computer Systems & Applications 18(7), 71–75 (2009) 12. Zhang, L., Lin, F., Zhang, B.: Support vector machine based relevance feedback algorithm in image retrieval. J. Tsinghua Univ (Sci. & Tech.) 42(1), 80–83 (2002)