An Ontology Based Framework for Retrieval of ...

Available online at www.sciencedirect.com

ScienceDirect Procedia Computer Science 84 (2016) 169 – 176

7th International conference on Intelligent Human Computer Interaction, IHCI 2015

An Ontology Based Framework for Retrieval of Museum Artifacts Manoj Kumar Sharmaa*, Tanveer J Siddiquia a

Department of Electronics and Communication, University of Allahabad Allahabad, India 211002

Abstract This paper proposes ontology based conceptual framework for storage and retrieval of Digitized Museum Artifacts. The proposed framework uses ontology structure for automatic image annotation. It supports semantic retrieval by combining ontological concepts, visual and textual features automatically extracted from images and their textual descriptions. The Ontology-driven analysis module automatically generates annotation for domain objects. This paper also reports a new dataset designed for its evaluation. The dataset consists of images displayed in various galleries of Allahabad museum along with their textual description. We have collected 1200 images and extracted their visual and textual features for the purpose of retrieval. © 2016 byby Elsevier B.V.B.V. This is an open access article under the CC BY-NC-ND license 2015The TheAuthors. Authors.Published Published Elsevier (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Scientific Committee of IHCI 2015. Peer-review under responsibility of the Organizing Committee of IHCI 2015

Keywords: Image retrieval; Image analysis; Image annotation; Ontology; Information retrieval; Knowledge based system; Knowledge sharing; knowledge discovery.

1. Introduction The recent growth in numerous key technologies has greatly simplified creation, processing and on line delivery of visual and textual content. This results in enormous growth in the amount of digital content available in unstructured and non-indexed forms on the web in personal as well as commercial collections. In order to provide access to this data to users we need tools and techniques to automatically analyze, index and manage the visual content. A desirable key functionality is to make the content access in terms of semantics it represents. However, there exists a significant gap between the desired semantic level of access and the existing image retrieval system. The existing

*

Corresponding author. Tel.: +919838097584. E-mail address: [email protected]

1877-0509 © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of IHCI 2015 doi:10.1016/j.procs.2016.04.083

170

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176

image retrieval systems either use manually annotated keywords (keyword-based approach) or visual features for retrieval (Content-Based Image Retrieval). The keyword-based approach supports semantic retrieval but has several limitations. First, it is not scalable. Second, due to the subjectivity of the human annotator, the annotations may not be consistent or complete. Third, it may be infeasible to describe visual content simply using words. Content-Based Image Retrieval (CBIR) systems use visual features such as color, texture and shape for retrieval. A user formulates a query by providing examples of images similar to the desired ones. A retrieval model computes similarity between the query image and images in the database. The results are ranked based on the computed similarity values to perform retrieval. Although this approach is less time consuming and more user friendly1, the representation of image using visual features only involves a loss of information which is referred to as semantic gap. One way to overcome this limitation is to integrate visual descriptors along with the available textual and ontological descriptions to support semantic retrieval3. The use of domain knowledge appears to be a promising way by which higher-level semantics can be incorporated into techniques that capture the semantics through automatic analysis. In this paper, we propose a retrieval framework for museum artefacts. A museum preserves artefacts of scientific, artistic, cultural, or historical importance which attracts general public as well researcher and specialists. More and more museums are implementing digitization project to extend their reach beyond the wall. An online museum is not just a collection of artifacts but augments the presentation with useful textual description. Further, the artefacts are usually organized in different categories. Accordingly the digital counterpart of it will consist of images of artefacts and their textual and ontological descriptions. This underlines the need of a new form of retrieval and presentation method that can utilize the rich content of museum database to provide efficient access to cultural heritage content. In order to organize museum data in a meaningful manner so that the relevant and useful information for a user’s query can be searched and retrieved, a domain otology is defined. The proposed framework combines ontological, textual and visual descriptors to support semantic retrieval. The rest of the paper is organized as follows: section 2 briefly reviews existing work. Section 3 describes the proposed framework. In section 4, we discuss the dataset designed for the evaluation of the proposed framework. Finally, conclusions are made in section 5. 2. Related works Efficient image searching, browsing and retrieval tools are required by user from various domain including, art, fashion, crime prevention, medicine, remote sensing etc.For this purpose many retrieval system have been developed. Which can be broadly categorized into text based and content based. Keyword based system use keyword to annotate each image in the database using keywords that are used in the retrieval process. This approach supports semantic but is criticized due to its subjective nature, being time consuming and expensive, further difficulty in describing visual feature using text appropriately7. To overcome the above disadvantage in text based retrieval system, content based image retrieval (CBIR) was introduced. In CBIR images are indexed by their visual content, such as color, texture, and shape for retrieval. These features can be extracted automatically. Images in the database are represented as vector of extracted visual features instead of textual annotation. User formulates a query by providing examples of images similar to the desired ones. The retrieval model compute similarities between images in database and the query representation, and rank results are based on the computed similarity values. The query, and retrieval models may a clustering module, which expedites searching in large image database. A detail survey of CBIR system can be found in 2. In the past decade, a few commercial prototype systems have been developed based on CBIR paradigm, such as QBIC 4, Photobook5, VisualSEEK 6.Although this approach is less time consuming and more user friendly, the representation of image using visual feature alone involves a loss of information which is referred to as semantic gap.

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176

Efforts have been made to overcome the semantic gap problem through the use of relevance feedback, automatic annotation, semantic templates, ontological description, etc. Automatic image annotation technique attempt to associate keywords for an image automatically. Supervised and unsupervised learning has been widely used for automatic tagging. Among the most commonly applied machine learning techniques are Hidden Markov Models, Support vector Machines and Neural Networks. Text and image fusion is another way to handle semantic gap problem8.Westerveld et al. 9 combined image features and words from collateral text into one semantic space by using latent semantic Indexing for representing the image/text content. Berg et al. 10 process the nearly parallel image text pairs found in the yahoo news corpus. They consider all possible pairs of text and image and use clustering with expectation maximization algorithm. In relevance feedback user intervention was used in the process of knowledge acquisition11. Ontology based frameworks for manual image annotation and semantic retrieval include the ones presented in12 and 13. In 14 ontology based information extraction is applied to improve the results of information retrieval in multimedia archives. They used a domain specific ontology, multilingual lexicons and reasoning algorithms to integrate cross-modal content annotations. Ontology has been also applied successfully for handling museum collection in 15. Numerous analysis approaches emerged utilizing the formal semantics and inference capabilities of ontology. Ontology appears to be the right choice for knowledge representation and management among the computer vision community. In 16 the user-assisted approach for automatic image annotation is enhanced by rules on top of domain ontology. Kumar et al.18 use of multilevel classification techniques for combining words and picture for museum information retrieval. 3. Ontology Ontology identifies class of objects that are important to examine a domain under a specific viewpoint and organizes these classes in a subclass/super class hierarchy. Each such class is characterized by properties that all instances in that class share. Important relation between classes or instances of the classes is also part of the ontology. In this work, two different ontology is defined and used: one for domain another for analysis. The two ontologies are integrated appropriately. The domain ontology formalizes the domain semantics, provides conceptualization and vocabulary for visual content annotation and retrieval. The analysis ontology is used to guide the analysis process and supports the detection of certain concepts defined in the domain ontology. Both ontology are expressed in RDF and their integration takes place using the common conceptual class between the two ontology for visual content annotation at semantic level. 3.1. Domain Ontology: There are several partly conflicting goals to keep in mind when designing the ontology. The main goal of knowledge-assisted semantic visual content analysis is to extract semantic descriptions from low level image descriptions. Therefore, domain knowledge needs to include prototypical descriptions of domain concept, objects and events, in term of their visual properties and relationship. As illustrated in the domain ontology snapshot of Fig.1.the super-class of domain ontology is artefact class from which all other classes are derived. The ontology provides understanding of the artefacts by decomposing complex items into their constituent logical categories including independent substances, dependent items such as attribute and properties, temporal items such as events and processes, spatial and temporal regions, context sensitive perspectives and various form of relation like Internal relation, External relation, Grounded relation, Intentional Relation, Existential Relation17. Relations are defined to model additional information regarding the person or artefacts who took part in an event, and the corresponding location as well as the way the various object and event related to each other to the different classes of such artefacts. The main classes of the ontology are as follows:

171

172

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176

Museum Artefacts: It is the super-class, which includes all other classes in the domain. This describes common properties like date and place of the artefacts. Event: This sub-class models variety of events that could be depicted in museum artefacts like political, social events, official, personal, historical, etc., as well as events associated to natural phenomena such as rain, snow, water, sunset, sunrise, etc.,. Object: This sub-class encompasses the main categories of objects found in museum. It has two sub-classes: (i) man-made objects that include among others, painting, building, road, furniture and transport related artefacts etc. (ii) natural objects. Natural objects are further divided into three sub-classes :( i) Biological objects that include the different living organisms such as, person, animal, vegetation. (ii) Geographical objects: that includes various geological formations like land, mountain, volcano, water bodies, etc. And (iii) Celestial objects like star, moon, sun, sky.

Fig.1 Domain Ontology

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176

173

3.2 Analysis Ontology: As illustrated in the analysis ontology snapshots of Fig.2 .The domain ontology is analyzed to detect concepts related to an image. It links each object instance to its visual description in terms of low-level features and spatial characteristics certain object detection may be significant if other objects are detected first. The visual feature being used includes histogram features such as gray scale and color histogram that have been used for different scenario 5. However, any other proprietary descriptor could be used instead, showing another benefit from the formal knowledge representation framework provided by ontology

Fig.2 Analysis ontology 4. The Proposed Framework The architecture in Fig.3 depicts how semantic understanding is supported by the propose framework. In semantic annotation, meaning of images and queries are described based on a combination of concepts defined in ontology. Through semantic annotation, both image and queries can be formalized as xml file. The analysis starts by segmenting the input image and extracting low-level visual descriptors of the segments and their relation according to the domain ontology. The domain knowledge is used in annotation. Since each semantic annotation is a description based on the concepts in ontology, understanding of the corresponding concept is the first step needed to understand for each segments. The concepts extracted from several of segments of an image are combined to yield a set of combined concept entity. The combined concept entity is the fundamental semantic unit in the model for semantic annotation, and is also the basis for similarity comparison between images and queries. In an image, we include our context about objects. This spatial context of object in an image is also stored in knowledge base.

174

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176

Fig.3 Propose framework

Fig.4 Architecture for Semantic retrieval component

5. Dataset and Example 5.1 Data In this section, we describe the dataset designed for evaluation of the proposed framework. The motivation behind the development of this dataset is to provide a realistic scenario for the evaluation of the proposed framework. The dataset consist of 1200 images collected from 11 different categories. These images are organized into a hierarchy. The number of instances/images in each category is summarized in Table 1. The images are preprocessed using Adobe Photoshop CS to standardize their size, adjust contrast and to reduce noise. After normalization, the items of interest in the image are stored in color jpeg format and metadata descriptions is created. Table1: Category of Allahabad Museum Images Category Name No. of Images Terracotta 78 Sumitra Nandan Pant 35 Stone 62 Natural History 80 Nehru 190 Jewellery 37 Gandhi 125 Freedom fighter 70

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176 Early Sculpture Early Medieval Archaeological collection Others

51 73 56 343

5.2 Example Fig.4 elaborates how the framework supports semantic annotation and retrieval. In the example of query “freedom fighter in Indian independence” we can extract the concept freedom fighter and Indian independence, with these two concept we can also extract several attributes used to specify information about instantiation of these two concepts such as name, year, place etc. These concepts can be considered to form a concept entity that represents a semantic unit. Therefore, concepts in the concept set are combined into combined concept entities when one serves as an attribute of another. After the extraction of combined concept entities, the semantic annotation in an xml file is converted to a set of combined concept entities. Based on corresponding sets of combined concept entities, the next step is to compare the semantic similarity between images and queries using concept relation IS-A. 4. Conclusion In this paper, we proposed ontology based framework for semantic understanding of image for retrieval and developed a dataset for its evaluation. The proposed framework addresses semantic image annotation representation. The fundamental semantic entity is combined concept entity. A mechanism is presented for computing the semantic similarity between images and query. The Ontology based framework partially alleviates the limitations entailed by the high cost of manual annotation through automatic generation of semantic annotation thus, enabling a realistic approach to effective image access at semantic level. The textual features include keywords automatically extracted from accompanying textual description and metadata extracted from ontological concepts. Acknowledgements We thank Director, Allahabad Museum for permitting us to collect images to create a dataset. The authors gratefully acknowledges the financial support from Dept. of Science and Technology, Govt. of India under Grant No SR/FTP/ETAY36/2011 References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

Park KW, Jeong JW, Lee DH, Olybia. Ontology-Based Automatic Image Annotation System Using Semantic Inference Rules. In R. Kotagiri et al. (Eds.) LNCS 4443, DASFAA; 2007. p. 485–496. Liu Y, Zhang D, Lu G, Ma W-Y. A survey of content based image retrieval with high level semantics, Pattern recognition 2007; 40:262282. Siddiqui TJ, Tiwary US. Words and Pictures: An HCI Perspective. Proceeding of the First International Conference on Intelligent Human Computer Interaction 2009;59-70. Faloutsos C, Barber R, Flickner M, Hafner J, Niblack W, Petkovic D, Equitz W. Efficient and effective querying by image content. J Intell Inf Syst1994;3(3-4):231-262. Pentland A, Picard RW, Scaroff S. Photobook: content-based manipulation for image database. Int.J.Comput. Vision 1996;18(3):233-254. Smith JR, Chang SF. Visual Seek: a fully automatic content based query system. Proceeding of the fourth ACM International Conference on Multimedia1996:87-98. Eaking J, Graham M. Content-based image retrieval. Technical Report, University of Northumbria at Newcastle, 1999. Barnard K, Duygulu P, Forsyth D, Fretas ND, Blei DM, Jordan MI. Matching Words and Pictures. J Machine Learning Research 2003:3(6):1107-1135. Westerveld T, Gemert JCV, Cornacchia R, Hiemstra D, Vries AD. An Integrated Approach to Text and Image Retrieval. In Proceedings of TRECVID Gaithersburg MD, 2005. Berg TL, Berg AC, Edwards J, Forsyth DA.Who’s in the picture? In Neural Information Processing Systems 2004:137-144. Zhou XS, Huang TS. Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems 2003;8:536–544. Schreiber AT, Dubbeldam B, Wielemaker J, Wielinga BJ. Ontology-Based Photo Annotation. IEEE Intelligent Systems2001;16:66-74. Hollink L, Schreiber AT, Wielemaker J, Wielinga B. Semantic Annotations of Image Collections. Workshop on Knowledge Capture and Semantic Annotation (KCAP), Florida, 2003.

175

176

Manoj Kumar Sharma and Tanveer J Siddiqui / Procedia Computer Science 84 (2016) 169 – 176 14. Reidsma D, Kuper J, Declerck T, Saggion H, Cunningham H. Cross document annotation for multimedia retrieval, 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL)2003. 15. Sinclair PAS, Goodall S, Lewis PH, Martinez K, Addis MJ. Concept browsing for multimedia retrieval in the SCULPTEUR project, Multimedia and the Semantic Web Workshop. Annual European Semantic Web Conference (ESWC) 2005:28-36. 16. Little S, Hunter J. Rules-by-example - a novel approach to semantic indexing and querying of images, Proc. International Semantic Web Conference (ISWC)2004:534–548. 17. Little EG, Rogova GL. Designing ontologies for higher level fusion. J Information Fusion 2009:10 70-80. 18. Kumar A, Tiwary US, Siddqui TJ .Combining words and pictures for Museum Information Retrieval, Proceeding of 4th International Conference on Intelligent Human Computer Interaction(IHCI 2012),kharagpur,India 2012:1-6.