Google image search engine to collect domain specific images. We also keep these images in ranking order according to the Google image result. This set of ...
Does Ontology Help in Image Retrieval? — A Comparison between Keyword, Text Ontology and Multi-Modality Ontology Approaches Huan Wang, Song Liu, and Liang-Tien Chia Centre for Multimedia and Network Technology, School of Computer Engineering Nanyang Technological University, Singapore 639798 {wa0004an,
pg03988006, asltchia}@ntu.edu.sg
ABSTRACT Ontologies are effective for representing domain concepts and relations in a form of semantic network. Many efforts have been made to import ontology into information matchmaking and retrieval. This trend is further accelerated by the convergence of various high-level concepts and low-level features supported by ontologies. In this paper we propose a comparison between traditional keyword based image retrieval and the promising ontology based image retrieval. To be complete, we construct the ontologies not only on text annotation, but also on a combination of text annotation and image feature. The experiments are conducted on a medium-sized data set including about 4000 images. The result proved the efficacy of utilizing both text and image features in a multi-modality ontology to improve the image retrieval.
Categories and Subject Descriptors H.5.1 [INFORMATION INTERFACES AND PRESENTATION]: Multimedia Information Systems
General Terms Algorithms, Performance, Experimentation
1.
INTRODUCTION
Image retrieval has always been one of the most active research fields. Nowadays most popular web image retrieval systems are based on keyword searching in the surrounding text of images. However, this kind of retrieval requires adequate text information with correct keyword description about the corresponding images, which is not always true in real situations. Moreover, most pure text-based retrievals simply ignore the helpful image features which can be extracted through multimedia analysis. Some irrelevant images are retrieved for this reason. Content-base image retrieval(CBIR) has been studied for many years. It is a technique to extract image features like dominant color, color histogram, texture, object shape, and so on. The main problem is the semantic gap between the low-level image features and high-level human-understandable concepts. Though efforts like relevance feedback(RF) has been made to bridge this gap, lots of user interactions are involved. Ontologies are designed to capture shared knowledge and overcome the semantic heterogeneity among domains. Knowledge is
collected by expertise of individuals and represented in description logics. Machine can understand these uniformed representations and the gap between human-readable knowledge and machine-understandable logics is naturally bridged. A few applications[5] have used ontologies purely from MPEG-7 feature descriptors. For example, [6] designed and proved ontologies as middle-level structures to bridge the semantic gap between low-level feature and high-level concept. The main contribution of this paper is a comparison between the aforementioned traditional approaches and the ontology approaches. Though many people have tried to use vector or treestructure approaches, no actual application or explicit result are shown. In order to accomplish this experiment, we designed different ontologies in a selected domain and constructed corresponding domain knowledge. We compare the retrieval performance of different approaches and discuss their pros and cons. To the best of our knowledge, no existing works have compared keyword based retrieval with ontology based retrieval, or pure text ontology based retrieval with multi-modality ontology based retrieval in the domain of image retrieval. The rest of this paper is organized as follows: Section 2 introduces some related work. Section 3 focuses on the discussion of the designed ontology model, which is later used in the comparison. The experimental results and conclusions are given in Sections 4 and 5 respectively.
2. RELATED WORK In this section, we have a brief retrospect to previous works on different approaches. These approaches include content-based image retrievals(CBIR), text-based image search engines, and some ontology-based retrieval systems. Some of the techniques are applied in our comparison. The most intuitive way of web image retrieval is to use the surrounding textual information. Many textbased image search engines have been designed and made available on WWW, such as Y ahoo(T M ) and Google(T M ) . These textbased search engines search for images by text features, such as file name, annotation and so on. In our experiment, we use the Google image search engine to collect domain specific images. We also keep these images in ranking order according to the Google image result. This set of images and its ranking order is used as a retrieval result of the representative keyword search engines. Other than text information, image content is also a useful resource for retrieval. CBIR captures image features and organizes them into a meaningful retrieval architecture [7]. However, the semantic gap between the image feature and high-level concept is still an open issued. Ontology has been validated in practice[5][6][4] to bridge
Copyright is held by the author/owner(s). MM’06, October 23–27, 2006, Santa Barbara, California, USA. ACM 1-59593-447-2/06/0010.
109
Animal Description Ontology Section 3.1 Animal Domain Ontology
Section 3.2 Textual Description Ontology
Section 3.3 Visual Description Ontology
AnimalConcept Textual Concept
Jackal
Domesticdog Fox
color
Distribution
Wilddog
Wolf RedWolf
CapeFox
Pixcolor
ImageProperty
Outdoor P 1
Graph
Colorful
Europe
Wildlife
WhiteFur RedFur
Image
Building
hasDistribution
NoneFur
ColorFur Indoor
Red Asia
GreyWolf
Fur
Content
Greysacle
Gray America
RedFox
Visual Concept
Habitat
Canine
BrownFur
Asia
RedWolf
hasFur hasPixColor
hasColor
P4 BrownFur
Colorful
Red hasContent
Wild
RedFox hasFur hasDistribution
RedFur
USA Section 3.4 Generated Image Concept
rdfs:subClassOf containConcept
Property and its
Class
value
Class-Generated Class relationships
Generated Class
Figure 1: Layer structure of Ontologies this semantic gap. Furthermore, attempts[1][3] have been made to combine text information with image features for better retrieval performance. However, these works either depend on domains with uniform or simple image contents, or fail to give explicit experiment result.
3.
Table 1: Performance of image classification Classification Colorful/Graylike Photograph/Drawing Outdoor/Indoor HumanRelevantScene/Buildings/Wildlife Greenery/Sand/Stone/Snow/Others WhiteFur/RedFur/GrayFur/BrownFur/NonAnmial
ONTOLOGY STRUCTURE
In this section we briefly discuss the structures of the text and multi-modality ontologies, based on which we do image retrieval and get comparable results with text-based image retrieval. As our main focus is the comparison result, we do not include details of ontology construction, which is available at [8]. The experimental domain is canine, which is a sub-domain of animal. It is a challenging domain due to animals’ varied shapes and complex living environments. Therefore ontology is effective on capturing and integrating various aspects of information. This ontology is well-defined for semantic research scenarios and open to further extension in our future work.
0.921 0.842 0.806 0.794 0.814 0.634
3.2 Textual Description Ontology Textual description ontology is purely based on text and it is used to encapsulate high-level narrative animal description. By this ontology, certain animal is associated with its domain knowledge. That is why this ontology works better than single keyword on capturing semantic interpretations from different context. Several classes have been defined like “ScientificName”, “Diet”, “Habitat”, “Distribution” and “ColorDescription”. And semantic relationships have been generated to connect different concepts including “hasName”, “hasDiet”, “hasHabitat”, “hasDistribution”, “hasColorDescription”. The class and relationship definitions of the ontology are extracted from the BBC Science & Nature Animal category as well. We also generate general knowledge ontologies like geographical ontology, color ontology which are associated with the text ontologies by the relationships defined above.
3.1 Animal Domain Ontology The animal domain ontology is the basis of the following text ontology and multi-modality ontology. It provides semantic information of taxonomy definition for the target domain and handles the classification of animal species. This work is usually done by domain expert, while in our case, we derive the formal definition and domain knowledge from the BBC Science & Nature Animal category1 . It provides standard and unified descriptions in various aspects for around 620 animals. 20 subspecies under the domain of canine are collected as our experiment subjects. We re-define the hyponyms relationship between two concepts as subclass property in this ontology. For example, fox is a kind of canine (hyponyms), and therefore fox is defined as a subclass of canine in animal domain ontology. A motivating example is that without this domain ontology, a dhole image will not be returned for a search for wild dog while it actually should be. 1
ACCR
3.3 Visual Description Ontology Besides the single-modality textual description ontology, we define a multi-modality ontology which combines textual description and image features. First we build specific knowledge base in which the classes and relationships are extracted from low-level features. Then we incorporate this visual description ontology with the aforementioned text ontology to the multi-modality ontology. This combined ontology works better on images with loosely-coupled text annotation. We formulate each image classification scheme as a class in the
http://www.bbc.co.uk/nature/wildfacts/animals a z.shtml
110
Arctic Fox Retrieval Result
ontology and define the image categories under this classification scheme as its subclasses. These classes include GreyLikeImage, ColorImage, ContentType, OutdoorScene, IndoorScene, BuildingRelevant, HumanRelevant, WildlifeScene, FurColor. A complete list of relationships are extracted from low-level features as follows: “hasPixColor”, “hasPixProp”, “hasEnvironment”, “hasContent” and “hasFur”. So far, we have high-level descriptions generated not only from textual information, but also from low-level image attributes.
35
100
80
60
40 Arctoc Fox(Google) Arctic Fox(Text Ontology) Arctic Fox(Multi−Modality Ontology) Arctic Fox(Optimal)
20
0
Figure 1 provides a structure of the canine ontology in our system. We use ellipse and rectangle to represent pre-defined class and generated class respectively. The horizontal line in the middle is to separate the two different kinds of classes. Part of the ontology is omitted due to the limited space. Two examples red fox and red wolf are given to show how we define concrete animal concept using ontology. We can see red fox and red wolf are two generated classes of superclass canine. From the textual description ontology, we know red fox has distribution in USA and red wolf has distribution in Asia. The visual description ontology indicates the fur color of red wolf is brown and the fur color of red fox is red, even though in the query they share the same keyword red. The visual description information is helpful to filter a majority of inaccurate results. For instance, we can reasonably infer from an indoor background that a wild cape fox is not likely to exist in the image.
0
50 100 150 Number of Images Retrieved(in ranking order)
Number of Correct Images Retrieved
Number of Correct Images Retrieved
120
3.4 Examples of the Generated Classes
4.
Bush Dog Retrieval Result
140
30
25
20
15
10 Bush Dog(Google) Bush Dog(Text Ontology) Bush Dog(Multi−Modality Ontology) Bush Dog(Optimal)
5
0
200
0
140
60
120
50
40
30
20 Coyote(Google) Coyote(Text Ontology) Coyote(Multi−Modality Ontology) Coyote(Optimal)
10
0
0
50 100 150 Number of Images Retrieved(in ranking order)
200
Ethiopian Wolf Retrieval Result
70
200
Number of Correct Images Retrieved
Number of Correct Images Retrieved
Coyote Retrieval Result
50 100 150 Number of Images Retrieved(in ranking order)
100
80
60
40 Ethiopian Wolf(Google) Ethiopian Wolf(Text Ontology) Ethiopian Wolf(Multi−Modality Ontology) Ethiopian Wolf(Optimal)
20
0
0
50 100 150 Number of Images (in ranking order)
200
Figure 2: A comparison of image retrieval results between different approaches(1)
EXPERIMENTAL RESULTS
In the experiment we compare the ontology-based image retrieval systems with the Google image Search, which is among the best keyword based search engines and handles over 2 billion images. The experiment data set is set up by a total of 4000 images using the top 200 Google images of each of the 20 canine subspecies. Google image is used in our experiment as it is accessible and other researchers can easily compare our performance with their experimental results. The reason for using only the top 200 ranking results of Google image is that this set of images are statistically and visually higher in significance and ranking. The data set of the web images and their web pages is downloaded by our image crawler. The experimental results for low-level feature extraction using SVM is shown in Table1, which is used to build the multi-modality ontology. For the comparison we use the Google image Search with text ontology-based retrieval and multi-modality ontology-based retrieval. For semantic matchmaking of ontologies, we choose RACER version 1.9[2] as our reasoner since it is able to provide consistency checking of the knowledge base, computing entailed knowledge via resolution and processing queries through complex reasoning. To evaluate the performance of high-level information extraction based on low-level features, we list the average correct classification rates (ACCR) in Table 1. In the classification, one third randomly selected data are used as training samples and rest of the data are used as test samples. We repeat each classification 10 times and calculate the ACCR. The last set of classification does not achieve very good classification performance because the fur color of fox is affected by the change of illumination and viewing angle.
formance of textual description ontology retrieval is slightly better than keyword based search. However, we find that text ontology retrieval is still hampered by the lack of text information within the web page. For example, if no related concept and relationship is extracted from the surrounding text, the generated class of this image is void. The result ranking is based on the degree of match: exact match, subsume match and disjoint. As a void class is disjoint with any pre-defined canine class, the image is ranked low in the final result even if it is correct.
4.2 Keyword, Text Ontology versus MultiModality Ontology From the figure, we can see the multi-modality ontology-based retrieval outperforms others by returning more relevant images with higher ranking. In the best case arctic fox, the multi-modality ontology-based retrieval almost overlaps the optimum blue line, which returns the N correct images in the first N ranking positions. The result benefits much from high accuracy of image feature classification in WhiteFur, whose ACCR is 0.826(this ACCR value is different from the one shown in Table 1 as that value is the Average ACCR of all fur types). However, there are gaps between the multi-modality ontology results and the optimal results in most cases. We presume it could be due to one or more of these reasons: First the performance could be affected by the accuracy of image feature classification; Second, the lack of text information in the web pages will result in less correspondence in text ontology and multi-modality ontology; Third, concurrently we have not completed study on the accuracy of rule-based engines and reasoners, we are not sure if the reasoner we use provides the best matchmaking result.
4.1 Keyword versus Text Ontology We apply semantic matchmaking on the 200 top-ranking Google images with web pages and present some results of the overall performance of different approaches in Figure 2. In this test, the 200 images in each category include the training and test data used in previous image classification test. We can see that the overall per-
4.3 Comparison of Precision Results In Web image search and retrieval systems, precision is a very important guideline to measure the performance of a retrieval sys-
111
Top 20 Image Retrieval Result
Google result Text ontology result Multi−Modality ontology result
45
Number of Conrrect Image Retrieved
Number of Correct Image Retrieved
Top 40 Image Retrieval Result 50
Google result Text ontology result Multi−Modality ontology result
25
20
15
10
5
40 35 30 25 20 15 10 5
0
0
5
10 Animal Subspecies
15
0
20
0
5
Top 60 Image Retrieval Result
Number of Correct Images Retrieved
Number of Correct Images Retrieved
20
Google result Text ontology result Multi−Modality ontology result
90
60 50 40 30 20 10 0
15
Top 80 Images Retrieval Result 100
Google result Text ontology result Multi−Modality ontology result
70
10 Animal Subspecies
80 70 60 50 40 30 20 10
0
5
10 Animal Subspecies
15
20
0
0
5
10 Animal Subspecies
15
20
Figure 3: A comparison of image retrieval results between different approaches (2)
6. REFERENCES
tem because most users will browse only limited numbers of results. Therefore recall rate for a Web image retrieval is not that crucial. In Figure 3, we show the correct number of image retrieved in the top 20, 40, 60 and 80 for all 20 canine subspecies. The 20 subspecies are: 1. aardwolf, 2. African wild dog, 3. bateared fox, 4. black jackal, 5. cape fox, 6. arctic fox, 7. gray fox, 8. red fox, 9. kit fox, 10. bush dog, 11. coyote, 12. dhole, 13. dingo, 14. Ethiopian wolf, 15. fennec fox, 16. golden jackal, 17. gray wolf, 18. maned wolf, 19. red wolf and 20. spotted hyena. From the figure, we can see for top 20 image retrieval result, nearly all images retrieved by the multi-modality ontology are correct. In most cases, ontology-based image retrieval can achieve better retrieval precision than keyword-based image search. As we only implement generic image classification mechanism which is not particularly designed for the target domain, we can see from Table 1 that if we only apply the image retrieval based on image classification results, we may fail to outperform the normal textual based image retrieval mechanism. However, by combining the high-level textual information with low-level image features, we are able to improve the retrieval precision by about 5 to 30 percent.
5.
[1] Y. A. Aslandogan, C. Thier, C. T. Yu, J. Zou, and N. Rishe. Using semantic contents and wordnet in image retrieval. In Proceedings of 20th annual international ACM SIGIR conference on Research and development in information, pages 286–295, 1997. [2] V. Haarslev and R. Moller. Racer system description. In Proceeding of International Joint Conference on Automated Reasoning, pages 701–705, 2001. [3] S. Hammiche, S. Benbernou, M.-S. Hacid, and A. Vakali. Semantic retrieval of multimedia data. In MMDB, pages 36–44, 2004. [4] B. Hu, S. Dasmahapatra, P. Lewis, and N. Shadbolt. Ontology-based medical image annotation with description logics. In Proceedings of The 15th IEEE International Conference on Tools with Artificial Intelligence, pages 77–82, 2002. [5] J. Hunter. Adding multimedia to the semantic web - building an mpeg-7 ontology. International Semantic Web Working Symposium, August 2001. [6] S. Liu, L.-T. Chia, and S. Chan. Ontology for nature-scene image retrieval. In On the Move to Meaningful Internet Systems 2004: CoopIS, DOA, and ODBASE, pages 1050–1061, June 2004. [7] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Trans. PAMI, 22(12):1349–1379, DECEMBER 2000. [8] H. Wang, S. Liu, C. Zhou, and L.-T. Chia. Ontology Construction. available at: http://cemnet.ntu.edu.sg/pet device/wanghuan/Ontolo gy%20Construction.pdf, April 2006.
CONCLUSION
This paper has presented a comparison between keyword-based image retrieval and the ontology-based image retrieval. Different approaches have their own pros and cons. Keyword-based approach is user friendly and easy to apply with an acceptable retrieval precision, while semantically rich ontology addresses the need for complete descriptions of image retrieval and improves the precision of retrieval. However, the lack of text information which affects the performance of keyword approach is still a problem in text ontology approach. Ontology works better with the combination of image features. Though there is a trade-off between the complexity and performance, ontology is still a viable choice when better performance is expected with a smaller result set.
112
