Similarity-Based Operators in Image Database Systems Solomon Atnafu1 , Lionel Brunie1 , and Harald Kosch2 1 Information Systems Engineering Laboratory, INSA de Lyon Bat.Blaise Pascal, 20 Avenue Albert Einstein, 69621 Villeurbanne, France {Solomon.Atnafu, Lionel.Brunie}@lisi.insa-lyon.fr 2 Institute of Information Technology, University Klagenfurt Universit¨ atsstr. 65-67, 9020 Klagenfurt, Austria
[email protected]
Abstract. The integration of similarity-based data retrieval techniques into database management systems, in order to efficiently support multimedia data, is currently an active research issue. In this paper, we first demonstrate the necessity of introducing novel similarity-based operations in image databases, with example queries. Then, we introduce our image data repository model that is designed to support similaritybased operations under the object-relational database paradigm. We then define novel similarity-based algebraic operators that are frequently required and study their properties in image database management systems. Finally, we investigate the possibilities of extending the proposed image table model to support salient-based operations on image database systems.
1
Introduction
The importance of using the low level contents of image data for its identification, storage and operation purpose has been one of the active issues of research in the last decade. As a result, a number of research prototypes, applications, and commercial systems that support low-level content manipulation has been developed [12,2,13,8,14]. Thus, the use of low-level contents for similarity-based search has become a promising approach for image and video data management. Thus, in this paper, we focus on the management of content-based1 image databases. A number of works are done to integrate image and other media data with contentbased data retrieval methods using different techniques [8,3,6,7]. What most of the currently available content-based image retrieval systems do in common is that, for a given single query image, they search for the most similar images from a set of or database of images. This in principle can be associated to a content-based selection operation in a multimedia database systems. However, other more complex operations such as ”content-based join operation”2 (i.e. a 1 2
In this paper, the terms ”content-based” and ”similarity-based” are used interchangeably. See Section 4 for more details on this operation.
X.S. Wang, G. Yu, and H. Lu (Eds.): WAIM 2001, LNCS 2118, pp. 14–25, 2001. c Springer-Verlag Berlin Heidelberg 2001
Similarity-Based Operators in Image Database Systems
15
Similarity-based matching of two or more image tables) are not considered in the works done in this area. We believe that there is the need to consider such operations and to develop a formal framework for the similarity-based operations on image databases identical to that of the traditional database operators. Hence, the purpose of this paper is to identify and introduce the most needed similarity-based operations, to study their properties, to formalize the use of these operations, and to see the way of using these similarity-based operations in conjunction with the relational operators under an Object-Rrelational (OR) paradigm. The capability of an OR DBMS to support multimedia data is widely discussed in [11]. To demonstrate the practical significance of our work, consider that an image table consist of the attribute components: image, the feature vector representation 3 of the image, and a component that contains alphanumeric information about the image. Let EMP is an image table of employee of a company and contains the attribute components: photo of the employee, feature vector representation of the corresponding photo, the name of the employee, his/her occupation, etc. Let SI be an image table that contains images of individuals who appeared and entered by the front gate of the company where a surveillance camera is mounted, the corresponding feature vector representations of their images, the date and time at which each of the images was scanned or taken. Suppose now that there is an investigation scenario of an event that is associated to the images in SI. SI alone can not give a complete information about a person whose photo is captured. It is therefore necessary, to perform some operations on the tables in order to get a complete information of a person whose image is in SI. Sinse, operations based on similarity measures do not perform accurate matching, a common practice is to search for a range of the most similar images. An investigator may pose the following query. Query: For pictures of individuals in SI that were scanned yesterday from 4 to 6PM, find their most similar images from EMP, with their corresponding name and address. Processing this query requires a relational selection on the SI table and then a ”similarity-based join” on SI and EMP. This demonstrates the need for a combined use of relational and similarity-based operations on image tables. To facilitate similarity-based system of operations we also need a convienent image data repository model. Thus, to introduce such system of operations in image database systems, we first present in Section 2 the related work in the domain. Section 3 introduces our model for an image data repository. In Section 4, the most commonly required similarity-based operators are introduced and their properties are studied. Section 5 demonstrates how our image table model can be extended to support operations on salient objects of an image. Finally, conclusions are given.
3
That is, a vector representation of an image in terms of its features such as color, texture, shape, etc.
16
2
S. Atnafu, L. Brunie, and H. Kosch
Related Work
A number of content-based image retrieval research prototype systems such as Photobook, Netra, Surfimage, VisualSeek, CAFIIR, STAR etc. [8,15,5,3] are currently in practice. Commercial database systems have also started to integrate content-based image retrieval modules in their systems. QBIC is a content-based image query system available commercially either in standalone form, or as part of other IBM products such as the DB2 Digital Library. It offers retrieval by any combination of color, texture or shape - as well as by text keyword [15]. However, it doesn’t support operations such as the ”similarity-based join”. The VIR Image Engine from Virage is a well-known commercial system. It is available as a series of independent modules, that system developers can integrate in to their own programs. The engine provides the fundamental capability for analyzing and comparing images. But, it has no concept of persistent storage, user interfaces, query processing nor optimization [3]. It is a pure set of algorithms that is designed to be embedded in software and hardware systems where its capabilities are needed. This makes it easy to extend it by building new types of query interface, or additional customized modules to process specialized collections of images. As a result, the VIR Image Engine is available as an add-on to the existing database management systems such as Oracle8i Enterprise Edition [7]. The Excalibur Image DataBlade Module from Excalibur Technologies is a product that offers a variety of image indexing and matching techniques based on the company’s own proprietary pattern recognition technology. This datablade module is incorporated in the Informix database system to support a content-based image storage and retrieval [6]. A common feature of the add-on modules integrated in these DBMSs is that, given a query image, they search its most similar images from a database of images using their respective content-based image retrieval engines. That is, the attempts so far didn’t exceed from these one-to-many content-based image retrieval operations. A positive result of these works is that, one can use SQL based statements to store, and retrieve images using content-based feature representations. However, as said before, these systems are strongly limited in terms of supporting complex similarity-based operations and mixed queries involving both relational and similarity-based operations.
3
A Model for an Image Data Repository
The introduction of images in to a database management system has required different techniques to store, describe, and manipulate image data. W.I. Grosky et al have given a model that describes the information that can be captured by an image data to facilitate its storage and content-based retrieval [4]. According to this model, the information that an image data can posses may be seen as physical view (image matrix and image header) and logival view (global and content-based view). Thus, an image repository or an image table should manage to capture all these information regarding an image.
Similarity-Based Operators in Image Database Systems
17
In this paper we introduce our novel image data repository model (which we also refer it as ”image table”), that is defined under an OR paradigm. Definition 3.1. (Image Table) An object-relational image table is defined as a table of five components M (id, O, F, A, P ). Where: – id is a unique identifier of an instance of M, – O is a reference to the image object itself which can be stored as a BLOB internally in the table or which can be referenced as an external BF ILE, – F is a feature vector representation of the object O. – A is an attribute component that is used to describe the object using keyword like annotations and may be declared as a set of object types, – P is a data structure that is used to capture pointer links to instances of other tables associated by a binary operation. Note that, in an image table M, the item of primary importance for a contentbased retrieval is the image itself. The image is described by its feature vector and all the remaining attributes are associated to the image. The three components O, F , and A can be used to capture sufficient information on an image data. P can be considered as a column whose content is a data structure that can store links to instances of other tables during binary operations such as similarity-based join. P has a value null in the base tables, but a non-null value in the resulting table of a similarity-based binary operation. More formally, when it contains a value, P can be expressed as a set of tuples of the form (table, set of ids), where the component table denotes the associated table by a binary operation and set of ids is the set of its referenced id elements. After a similarity-based binary operation like a similarity-based join of M1 and M2 (see Section 4 for the Definitions), the P component of the resulting image table, M 0 , holds a link to a table M2 . Then, for each instance of M 0 , P will contain elements of the form (M2 , {id12 , ..., idh2 }), where h is the number of instances of M2 associated by the operation. With the help of this image table model, we can manage the requirements to integrate a content-based image data in an OR data management system more convienently. Furthermore, this model can be extended to support the storage of segmented objects of an image to enable us perform salient-based operations on image tables.
4
The Similarity-Based Operators
To define the novel similarity-based operators on image tables we use the method of content-based range query [9]. A content-based range query on a set of images S returns those image objects that are within distance4 of ε from the query 4
Here, distance is defined as the distance between the feature vectors representing the images in a feature space
18
S. Atnafu, L. Brunie, and H. Kosch
image q. Thus, we define our similarity-based operators as given below. At the end of this section, we will show how the range query is advantageous over the k-Nearest Neighbour method [10] for similarity-based opearations in image database systems. 4.1
The Similarity-Based Selection Operator
The similarity-based selection operator is a unary operator on an image table performed on the component F based on the following definition. Definition 4.1. (Similarity-Based Selection) Given an image query object o with its feature vector representation, an image table M , and a positive real number ε; a similarity-based selection operation denoted by σoε (M ), is given by: σoε (M ) = {(id, o0 , f, a, p) ∈ M |o0 ∈ N N ε (M, o)}, where N N ε (M, o) denotes the range query with respect to ε for query image object o and an image table M . The similarity-based selection operation first uses the range query search method to select the image objects that are most similar to o from the objects in M (which is also expressed by the notation N N ε (M, o)). Then, it identifies the instances of M whose image objects are selected to be similar to o. 4.2
The Similarity-Based Join Operator
A similarity-based join is a binary operator on image tables performed on the feature vector components as defined formally below. Definition 4.2. (Similarity-Based Join Operation) Let M1 (id1 , O1 , F1 , A1 , P1 ) and M2 (id2 , O2 , F2 , A2 , P2 ) be two image tables and let ε be a positive real number. The similarity-based join operation on M1 and M2 , denoted by M1 ⊗ε M2 , is given by: 0 M1 ⊗ε M2 = {(id1 , o1 , f1 , a1 , p01 )|(id1 , o1 , f1 , a1 , p1 ) ∈ M1 and pQ 1 = p1 ∪ ε 0 ε (M2 , s id (M2 , o1 )) and p1 6= null}, where s id (M2 , o1 ) = M2 .id (σoε1 (M2 )) (i.e. the ids contained by the projection on the id component of the associated instances of M2 ). Each instance of an image table and a resulting table of a multimedia join 5 is identified by its unique identifier, id. Note that if the content of p1 is null (not pointing to M2 ), then its instance will not be contained in the resulting table. This definition of a similarity-based join reflects the practical needs of content-based image retrieval. Figure 1. illustrates, the resulting tables of two similarity-based joins. The similarity-based operators defined above depend on ”relative” measures. Due to this, the similarity-based operators possess different 5
In this paper and in [1], the terms ”similarity-based join” and ”multimedia join” are given interchangably.
Similarity-Based Operators in Image Database Systems M
M1
1
id 1 O 1 F 1 1 id 11 o1 f11
.. .
i
id 1
. . .
n
id 1
.
19
A1 a 11
.. .
.. .
o1i
f1 i
a 1i
n
a1n
. . .
. . .
o1n
f1
.. . . . .
P1 p11
id 1 O 1 F 1 1 f11 id 11 o1
.. .
. ..
p1i
. . .
1
k
2 (M 2 , { id 2 , id 2 , . . . , id 2 })
p1n
i
id 1
.. .
n
id 1
A1 a 11
P1 p11
p1i
.. .
.. .
o1i
f1 i
a 1i
n
a1n
.. .
.. .
o1n
f1
.. . .. .
. .. .. .
1
k
2 (M 2 , { id 2 , id 2 , . . . , id 2 }) 1
2
h
(M 3 , { id 3 , id 3 , . . . , id 3 })
p1n
Fig. 1. The Similarity-based joins M1 ⊗ε M2 and (M1 ⊗ε M2 ) ⊗ε M3 respectively
algebraic properties than that of the relational ones. For example, contrary to the relational join operator, we observed that the similarity-based join operator ⊗ε is not commutative. This shows that in similarity-based join operation, the order of the image tables is meaningful. If an image table appears at the left of a similarity-based join, then its objects are taken as reference for the similarity operation. In the example Query of Section 1, the image table SI should appear at the left if we are interested to know about the individuals who entered the gate. Furthermore, the similarity-based join operator ⊗ε is clearly not associative. However, an operator without these properties of commutativity and associativity is difficult to be exploited for query optimization. We thus need to see possibilities of extending this operator so that it satisfies these useful properties. 4.3
A Symmetric Similarity-Based Join Operator
In view of the current practices of content-based query systems (where for a given image query object, we search for its most similar objects from a database of image objects), the similarity-based join defined above is what may be needed for many applications. However, its non-commutativeness and non-associativeness makes it lose the good properties that the relational join operator has. To make the similarity-based join operator suitable for similaritybased query optimization, we extend the similarity-based join operator to a Symmetric M ultimedia Join operator in such a way that it satisfies the useful properties for query optimization. To facilitate this, let us first define the following basic operator, the Additive Union. Definition 4.3. (Additive Union) Let M1 and M2 be two image tables, the Additive Union of M1 and M2 denoted by M1 ] M2 is an image table that merges all the records of M2 to M1 or vice versa. The additive union contains all the instances that are either in M1 or in M2 , without excluding none of the instances of M1 or M2 . To perform a similarity-based binary operation on two image tables, we assume that their feature vector component F are extracted identically in such
20
S. Atnafu, L. Brunie, and H. Kosch
a way that it permits a meaningful computation of range query. Moreover, when we perform additive union, if the A components do not have the same structure we take the union of the A components of the two operand image tables. Here it is important to note that additive union is commutative. Below we define a symmetric similarity-based join that makes use of the similarity-based join operation and the additive union operator. Definition 4.4. (Symmetric Similarity-Based Join) Let M1 and M2 be two image tables, the symmetric similarity-based join of M1 and M2 denoted by M1 ⊕ε M2 is formally defined as: M1 ⊕ε M2 = (M1 ⊗ε M2 ) ] (M2 ⊗ε M1 ). Hence, a symmetric similarity-based join ⊕ε possesses the property of commutativity. i.e. M1 ⊕ε M2 = M2 ⊕ε M1 . This follows directly from the commutativity of the additive union. We can now generalize the symmetric symilarity-based join on more than two image tables and define a Multi Symmetric Similarity-Based Join. This definition reflects the characteristics of similarity-based operations and maintains useful properties for similarity-based query optimization. Definition 4.5.(The Multi Symmetric Similarity-Based Join) Let M1 , M2 , ... Mn be n image tables. The Multi Symmetric SimilarityBased Join, denoted by M1 ⊕ M2 ⊕ ... ⊕ Mn is defined as: ] Mi ⊕ Mj M1 ⊕ M2 ⊕ ... ⊕ Mn = i
