Electronic Letters on Computer Vision and Image Analysis 13(3):81-96; 2014
FMIRS: A Fuzzy indexing and retrieval system of mosaic-image database Maghrebi Wafa1, Khabou Mohamed A2., Alimi Adel M.1 1
REGIM: REsearch Group on Intelligent Machines, University of Sfax, Tunisia Electrical and Computer Engineering Dept, University of West Florida, USA Received 31st January 2014; accepted 27th Oct 2014
2
Abstract This work is dedicated to present a fuzzy-set based system useful for image indexing and retrieval pertaining to historical Roman-mosaics. This exceptional collection of mosaics dates back from the first to fourth centuries AD. Considering the state of these images (i.e. noise, color degradation, etc.) a fuzzy features definition is necessary. Thereby, we use a robust to rotation, scale and translation fuzzy extended curvature scale space (CSS) as shape descriptor. Furthermore, we propose a fuzzy color-quantization approach, applied on mosaics, using HSV color space. The system allows for two user-friendly querying modes: a drawing based mode and the mode that fusion both shape and color features using a unified fuzzy similarity measure. Based on queries of variable complexity, the advanced fuzzy system has managed to achieve interesting recall, precision and F-measure rates. Key words: Fuzzy system, fuzzy color-quantization, fuzzy CSS shape descriptor, Roman mosaics, fuzzy similarity.
1 Introduction A trend of museums creating digital showrooms/archives of the real paintings and artifacts they contain has emerged lately. The digital showrooms usually consist of high-resolution images of the paintings and artifacts devised to preserve the original pieces of work and make them available to the reach of a wider audience via the Internet. In this respect, Van den Broek et al. [1] have designed the C-BAR system (Content Based Art Retrieval) to index and present a digitized version of a 17th-century painting-collection belonging to the Netherlands’ national gallery (the Rijksmuseum http://www.rijksmuseum.nl/). In this context Broek et al.[2] have continued with improving the system’s performance and user interface. The Hermitage Museum, in collaboration with its partner IBM, presented by the QBIC (Query By Image Content) system (Flickner et al.[3]) that operates by applying layout and color queries to retrieve its stored digital-collection pertaining images (http://www.hermitagemuseum.org). As for (Schomaker et al.[4], Vuupijl et al.[5]), the authors have put forward the Vindx system as referential framework which helps index the digitized collection of the Rijksmuseum wide-array of images by allowing the user to specify a textual description of the object(s) he or she is trying to retrieve among the digitized images. The rationale advanced by the authors in favor of the object-based matching process lies in the fact that users are predominantly interested in the object content, rather than in the layout. Within the same line of thought, Chang and Kim [6] have considered, in their elaborated work, setting up a retrieval system useful for application to the digital of Korean-porcelain images. Actually, the authors have proposed introducing a fusion between normalizedcolor histogram and the centroid-distance vector as object-shape descriptors. The system uses high-resolution images consisting of mono-object porcelain artifacts, photographed on a uniform background.
Correspondence to: <
[email protected]> Recommended for acceptance by ELCVIA ISSN: 1577-5097 Published by Computer Vision Center / Universitat Autonoma de Barcelona, Barcelona, Spain
82
W. Maghrebi et al. / Electronic Letters on Computer Vision and Image Analysis 13(3):81-96; 2014
With regard to the Tunisian context, the National Library (http://www.bibliotheque.nat.tn), the National Archives (http://www.archives.tn), and a selection of other museums (http://www.patrimoinedetunisie. com.tn) such as Bardo, El-Jem, Enfidha, Sousse and Sfax, contain a huge selection of ancient documents, mosaics, and artifacts of important historical value. These treasures are carefully photographed and catalogued in such a way to be at the reach of researchers, thus avoiding direct handling of such fragile items. Some researchers have recently tackled the problem of indexing and cataloguing mosaic-related images with all the challenges such image peculiarities could present (Stanco et al.[7], M’hedbi et al.[8], Maghrebi et al.[9]). In this respect, M’hedbi et al.[7] use a CBIR (Content Based Information Retrieval) approach, whereby to retrieve mosaics based on shape descriptors, while Maghrebi et al.[9] present a retrieval system of Roman-mosaic images using drawing queries, involving a robust MPEG7 shape descriptors to index objects. However, users (laypersons and experts) involved in the cultural-heritage domain usually want to describe an object they are looking for. Since we treat historical images, with all the deficiencies they represent (noise, degradation of color, etc.); a fuzzy system is advanced in this paper for the purpose of indexing and retrieving rich and complex mosaics (see figure1). The main contribution of this work is the introduction of color and its fusion with shape to retrieve mosaic-images. Noteworthy, also, the retrieval process proposes a two-querying-modes process designed to be as much user-friendly and simple to implement as possible while guaranteeing a maximum level of ease of use. The remainder of this paper is organized as follows: the second section is dedicated to present the toplevel architecture of our designed scheme, while section 3 presents the mosaic-image database (DB) and the preprocessing step, followed by the section 4 which interests to present the fuzzy color quantization approach. As for section 5, it is devoted to describe the shape fuzzy features. Regarding the query and the fuzzy retrieval process, it is described in section 6, and the achieved experimental results are depicted in section 7 and last section is reserved to highlight the concluding remarks, research perspectives and horizons for future works.
Figure1. Samples of Tunisian historical mosaics
2 The system general architecture It is worth highlighting that our devised system’s general architecture is depicted in figure 2, below. Once a mosaic input image is provided to the FMIRS system, the later would allow the user to specify/extract the relevant objects using an on-line annotation module, which relies on a user’s perception to annotate areas of interest and important shapes/objects in a mosaic image. More details can be found in Maghrebi et al.[10]. By using an object boundary as a guiding outline, we extract, automatically, the objects’ robust shape crisp descriptors. Our shape features are a combination of both global and local features. The global features consist of the circularity and eccentricity. The local ones are the definition of shape concavities and convexities to represent an extended version of the Curvature-Scale Space (CSS). More details regarding these features are depicted in section 5. In addition, the proposed approach suggests color-features extraction based on the use of a fuzzy quantized-HSV-color space from preprocessed mosaic-image DB. To note, fuzzy sets are defined according to the hue (H), saturation (S) and value (V) components of the HSV color space, through which we attempt to provide a fuzzy logic model that aims at following the human intuition for color classification. The five most dominant colors (based on the normalized areas they occupy in respect of the targeted object) are determined and included within the image indexing process. Noteworthy, however, these extracted fuzzy visual features are used in the retrieval process. The user can apply either drawing query or query that simultaneously combines the relevant object’s color and boundary. Using a fuzzy similarity measure the system returns relevant images.
W. Maghrebi et al. / Electronic Letters on Computer Vision and Image Analysis 13(3):81-96, 2014
Mosaic image filtering
Filtered Mosaic DB
Foreground/Background separation
Objects Boundary DB
Extraction of objects crisp values Fuzzy color Quantization
Off line
Mosaic DB
83
fuzzy shape descriptors
On line
Object outline
Mosaic Index
User
Query
Query fuzzification
Fuzzy retrieval process Relevant mosaics
Figure2. The system general architecture
3 Mosaic-images database preprocessing Mosaics are a harmony of marble tesserae. So, mosaics are of natural colors and complex content. These images are resized to 512x512 and have been filtered with a Gaussian and median filters to reduce the noise caused by the mosaic architecture and the brisk intensity variation respectively. Furthermore, we use the Huang and Wang [11] threshold approach for the background/foreground separation to take only relevant information from the image. The Huang and Wang thresholding approach is based on minimizing the measure of fuzziness of an input image. The membership function in the thresholding method is used to denote the characteristic relationship between a pixel and its belonging region (the object or the background). It defines the absolute difference between the pixel gray level and the average gray level of the belonging region. Let I denote an image set of size M x N with L levels and f(x,y) is the gray level of a (x,y) pixel in I. So, the fuzzy membership function is defined by: μI (𝑓 𝑥, 𝑦 ) =
1 1+ 𝑓 𝑥,𝑦 −𝑚 1 (𝑇) /𝐷 1 1+ 𝑓(𝑥,𝑦)−𝑚 2 (𝑇) /𝐷
𝑖𝑓 𝑓(𝑥, 𝑦) ≤ 𝑇 𝑖𝑓 𝑓 𝑥, 𝑦 > 𝑇
(1)
With f(x,y) represents the gray level of the pixel (x,y) , while the measures m1, m2 are defined based on the Threshold T and histogram h(z) of the gray level z. The respective equations of m1 and m2 are as follows: 𝑚1 𝑇 =
𝑇 𝑧=0 𝑧 ℎ 𝑧 𝑇 𝑧=0 ℎ 𝑧
𝑚2 𝑇 =
𝐿−1 𝑧=𝑇+1 𝑧 ℎ 𝑧 𝐿−1 𝑧=𝑇+1 ℎ 𝑧
(2)
D is a constant defined by the equation 3 as follows: 𝐷 = 𝑧𝑚𝑎𝑥 − 𝑧𝑚𝑖𝑛
(3)
The main role is to find a membership degree between 0.5 and 1 in order to use Shannon function of Luca and Termini which define the entropy of fuzzy sets as:
83
84
W. Maghrebi et al. / Electronic Letters on Computer Vision and Image Analysis 13(3):81-96; 2014 1
𝐸 𝑇 = ln 2
𝐿−1 𝑧=0 ℎ
𝑧 𝑆𝑒(𝜇𝐼 (𝑧))
(4)
Where I(z) [0.5,1]. With the Shannon function, Se is defined as follows: 𝑆𝑒(𝜇𝐼 𝑧 ) = − 𝜇𝐼 𝑧 ln 𝜇𝐼 𝑧
− 1 − 𝜇𝐼 𝑧
ln 1 − 𝜇𝐼 𝑧
(5)
We retain the optimal threshold value that corresponds to the minimum measure: 𝑇 ∗ = arg max0≤𝑇≤𝐿 𝐸(𝑇)
(6)
Figure 3 shows examples of background/foreground separation using Huang and Wang approach.
(a)
(b)
(c)
(d)
Figure 3. a) Original image; b) binary image using the Huang and Wang thresholding approach; c) application of morphological operators and d) mosaic-image foreground
4 Presentation of the fuzzy-color naming approach 4.1 Related works Research dealing with color modeling for computational purposes has concentrated on seeking numerical color-representations fit for application to computer graphics and image processing. The problem is that none of these color spaces maps well onto the human perception of colors. Indeed, within a random set of RGB (Red, Green and Blue) or HSV values, even experienced humans can have difficulty determining the exact color representation is being depicted. (e.g what color is represented by the RGB triple (0.5,0.1,0) or by the HSV triple(0.03,1,0.5)). The reverse process is perhaps even more complex as in the case with: what are the exact RGB or HSV triples that best represent the color “Brown”. In this context, several studies have been conducted with the aim of creating color-semantic-labeling systems using a color space quantization (Berk et al.[12], Conway [13], Kelly[14], Shamir [15]). Berk et al.[12], with their devised Color Naming System (CNS), have propose to decipher 627 distinct colors, along with the designed ISCC-NBS system (Kelly[14]) using 267 different linguistic-color labeling based on color centroids, as well as the Conway system (Conway [13]) which proposes a quantized HSL color space distinguishing between 141 distinct colors. In turn, Shamir [15] proposes an image segmentation method based on fuzzy quantization HSV color space approach using ten, five and four fuzzy sets useful for representing the Hue, Saturation and Value fuzzy sets respectively. Benavente et al.[16] present the color-naming experiment developed to obtain a set of color judgments adequate for the fuzzy modeling of the color-naming task. Since, we use the historic mosaics (dates back from the 1st to the 4th century AD) which are in a vulnerable situation and present a pale and degraded color, our purpose is in the sense to apply a fuzzy color-quantization to present the color-fuzziness of mosaics.
W. Maghrebi et al. / Electronic Letters on Computer Vision and Image Analysis 13(3):81-96, 2014
85
4.2 The mosaic color specification It should be noted that for the sake of extracting color descriptor, the HSV color space needs to be applied (rather than RGB space). Indeed, the HSV color space proves to be more intuitive and closer to the human perception than the RGB space. In fact, within an HSV-space, the color is depicted by its hue component (H) with values ranging between 0 and 360, which actually code the color itself, the saturation (S) and value (V) with values comprised between 0 and 100. It is worth recalling that the last two HSV components refer to the color richness and brightness.
70% 60% 50% 40%
Hue
30%
Saturation Value
20% 10% 0%
