MUSTER: Multi-platform SysTem for Efficient Retrieval from multimedia databases T. Sinan Ingec*, Bulent Tavli**, Kadriye Ozbas*and Gozde Bozdagi** * Baskent University, Dept ofComputer Eng., Ankara, Turkey
** Baskent
University, Dept. ofElec. And Electronics Eng., Ankara, Turkey ingec, tavli,
[email protected], g.bozdagiieee.org ABSTRACT
In this paper we propose a tool for performing queries on image databases based on image content. The image content is saved as a feature set belonging to each image. The feature set is composed of color and location features but can easily be
extended to include features like, texture, shape etc. The features are extracted by a new method which processes the histogram of each color component at different scales and merge the results to fmd dominant colors in the image. The database access and user interface is written using VBScript, ASP and the gateway to the database is achieved using ADO. One of the main advantages of this tool is the usage ofActive Server Pages (ASP) rather than Cgi-bin or JAVA. Keywords: Multimedia databases, ASP, color segmentation, VRML
1. INTRODUCTION Due to the recent developments in hardware and software, it becomes much more easier to generate, process and store electronic information. Being only text at the beginning, this electronic information now includes graphics, images, animations, video, audio and other multimedia data at an increasing rate every year. Most of this information can be accessed through WEB by the help of high speed networks, search engines and browsing tools. However, most queries result in many
retrieved documents, only some of which are relevant. In case of multimedia data the problem is even worse. Since traditional methods of retrieving multimedia information depend on a number of descriptive keywords given to the data by the person indexing it, these methods are inefficient. Due to this fact, content based retrieval of digital imagery became an active area ofresearch in the field ofdatabase management [1].
Another issue in search of information from large multimedia databases is that, the interface should be user friendly and highly interactive. Also the system should be able to work on different platforms for wide accessibility. The tools that are found in the literature for content based indexing and retrieval are mostly based on JAVA and CGI-BIN applications [2-4]. Hovewer these technologies have drawbacks which make the tools either slow or reach a limited group of users. The disadvantage of CGI-BIN applications is that the executable codes must be put in a specified place where every user does not have the write permission wheras JAVA has the disadvantage of slow execution time because of applet's execution in the client side. Also the transfer of extra code of JAVA Applet requires extra time. We can see JAVA Servlet as a new alternative. The logic of serviets is very similar to ASP logic. The code is executed at the server side. The class codes are loaded into memory and the requests are answered via these classes. However the typical problem of JAVA is abs valid for JAVA servlets. Java serviets are slow compared to the ASP systems especially when ActiveX DLLs are used. Not only ASP is a more mature technology —more than 2 years in deployment — but it benefits from other developments at Microsoft, such as the recent decision to meld COM+, MTS and Microsoft Message Queuing into a full enterprise middleware platform. ASP has a component based structure and as components behind ASP get stronger, so does ASP [10]. In this paper, we propose a new database access, search and browsing tool based on ASP (Active Server Pages) where the gateway to database is achieved using ADO (ActiveX Data Objects). ASP is a new technology developed by Microsoft which brings a revolutionary vision for the WEB programming and solves the above mentioned problems [5]. The main idea is that the code runs on the server machine and the resulting HTML coded page is sent to the client machine. It also provides
Part of the SPIE Conference on Multimedia Storage and Archiving Systems IV Boston, Massachusetts • September 1999 SPIE Vol. 3846 • 0277-786X/99/$1 0.00
165
an ODBC connection to any database system which is available on the server PC. Also, since a simple HTML, coded page is sent to the client side, any browser can be used to see the results of the retrieval process.
Ihe database search is done by using color based features in the proposed system. Each image has an associated Iature vector composed of mean, area and center of mass of the dominant color regions. These feature vectors arc processed by the asm code where comparison is done based on a library of matching algorithm the tool provides. The rest of the paper is organized as follows: Section 2 presents the algorithm used for color extraction. Section 3 gives an overview of ASP and its use in MUSTER. Section 4 covers future extensions of the system. Conclusions arc given in section
2. DOMINANT COLOR EXTRACTION Current research on image indexing focuses on the choice of the suitable features for image representation. Among these featues. texture, shape. color, location are the most frequently used ones [6]. The comparison of these features used in the literature is based on the complexity of search and the feature vectors. In this paper, we propose a simple method for color based indexing that provides efficient indexing and fast search. Since color is a vital part of our everyday experience, it is useful for powerful visual processing in the world around us. Other than exact color distribution within a digitized image, people are usually interested in the most dominant color components. Due to this fact, we try to extract the dominant colors together with their locations and the area from the images to be used in indexing and retrieval. The proposed method is based on coarse segmentation of color images using a scale-space filter. Application of scale space filters for determining salient features is first proposed by Witkin [7]. Later on, Lim and Lee combined it with a Fuzzy clustering method to have a more accurate color segmentation [8]. In this paper we use a similar idea however reduce the complexity by modifying the clustering method.
Fig. I. The original image. Ihe proposed algorithm is composed of the following steps: 1. Obtain the histograms of R, G, B components of the image seperately. 2. Obtain the scale-space representation of each histogram. In order to do this, convolve the histograms by a Gaussian filter given as
h(x,r) exp[-x2I(2a2)]/(o(2m)), where r changes at each scale and take the second order derivative. A sample image is given in Fig. I, its R-histograni and scale-space representation obtained by this method is shown in Fig. 2.
166
::i 8O® 7000
3000F
LI5
Vu.*yIsv
Fig. 2. (a) Red image histogram, (b) Scale-space representation of(a).
3.
Determine the peak locations from the scale-space representation. The number of locations depends on the number of
dominant regions that is wanted to be extracted from the image. The location of peaks are shown in Fig. 2 based on the fact that the 3 dominant regions are going to be extracted. 4. Trace the peak locations to the lower scale in order to fmd thresholds for each region. 5. Repeat same process (3-4) for each color component. 6. Take the intersection of the extracted regions and fmd the overall thresholds. 7. Segment the image based on these thresholds. 8. Filter the segmented image to get rid of small regions. The three dominant colors extracted by the proposed method is shown in Fig. 3.
The next step for color indexing is to compute the mean value, area and the center of mass of the extracted regions as the feature vector. If more than one region is found to be belonging to the same cluster, the area and center of mass of the largest one is stored for the indexing. The feature vector for the image given in Fig.l is shown in Table 1.
Fig.3. Three detected objects in the image.
167
Object
Norm. Area
Mean
1
0.437
(69,43,173)
(0.359,0.508)
2
0.185
(90,39,157)
(0.792,0.218)
3
0.105
(133,26,135)
(0.767,0.597)
Norm. Center of mass
Table 1. Feature values for image in Fig. 1.
3. ACCESSING DATA USING ADO AND ASP ASP is a new technology developed by Microsoft which brings revolutionary vision for WEB programming. The main idea is that the code runs on the server machine and the resulting HTML coded page is sent to the client machine. One can write the code directly into the WEB page using VBscript syntax or the code can be compiled as DLL and can be reached by creating objects and components. ASP objects and components are nothing more than ActiveX components. The difference between objects and components lies in the way they are packaged. ASP objects are ActiveX elements (OLE objects) that are always available to Vbscript. ASP supports the Application, Session, Request, Response and Server objects. The Application object is active from the beginning of the web publishing to the end of it. One can create variables of Application object and these variables are kept to the end. The programmer can use the last modified value of an Application variable at any time. The
Session object is created for every user accessing to the page. One can also create variables of Session objects. These variables remain alive to the end ofthe session. When the connection is closed, or when a specified time period, which can be set by the programmer, has passed, the session variables are dropped and cleared from the memory. The good thing about the session variables is that, you keep the information in the memory until you do not need it anymore. No resource consumption is done and you reach the most important information easily. There is no way to do this by using cgi-bin applications. ASP components on the other hand, are DLLs that exist outside the ASP framework. These components can be created in any language. When these components are created, they are loaded into memory only once whatever the client access number is. As they are loaded into memory, they use a small part ofthe memory. When we think of cgi-bin applications, we see that cgibin applications are very weak in this concept. For every client access, a new instance of the application is loaded into the memory. This results in the bottleneck at the server side.
ASP supports the Database access, Browser Capabilities, Ad Rotator and Content Linking components. The most useful of
all the ASP components is the Database access component, also called the ActiveX Data Objects (ADO). Database publishing on the Web utilizes this component and the objects contained in it, to read and write to Open Database Connectivity (ODBC) data sources through OLE DB's ODBC data provider. The connection object is used to establish a connection between the software application and the database. When the data source connection is open, one can use the Recordset object to assign a query, execute it and retrieve the records.
3.1 The Structure of MUSTER In MUSTER, the images are retrieved based on the similarities between the dominant colors, areas of the regions having these colors and their center of masses in the query image and the images in the database. Initially the number of dominant regions is set to be 3 but in future versions, a hierarchical object description will be added for further flexibility. These features can either be extracted offline and stored in a file to be used in the retrieval process or be extracted on line by calling the segmentation program. The user has the flexibility to search based on all these 3 feature vectors or any combination of them as shown in Fig. 4. If all three features are chosen then color has the highest and location has the lowest priority. The user can also make a query based on a set ofselected features ifthere is no image to start with as shown in Fig. 5.
To send data to a WEB server, a client should utilize a form, which contain data input fields such as text boxes. In MUSTER, when the user choses the features that are going to be used in the retrieval process, respective elements of the feature vector of the query image (Table 1) is automatically inserted in a Request object in FORM format. This data is then processed by
168
ASP where comparison is done based on a library of matching algorithms and the content is returned to the browser as a data containing the links to the retrieved images and their feature vectors as shown in Fig. 6. ' c:MMProjectSecondasp - Microsoft Internet Expli
1 COLOR r AREA r LOCATION AVAIlABLE IMAGES
;11.Object C 2.Object C' 3.Object
C 1.Object
C'
2.Object C 3.Object
Fig. 4. The query page.
QUERY BUILDER FieIdName JFitstObjoctA,ee
keegeNeate
I!regeNeeto
krageNatee
Operator
Value
Coautector
j T F ____ J
[TJ _____
j
J FT1
Fig. 5. The stand-alone query builder.
169
Ths malles thaI & your csotersa are as follows.
173
115
60
127
Fig. 6. The result of the query based on the dominant color in the first image.
4. FUTURE WORK Right now, the proposed system assumes that there are only three objects in an image. The database structure is suitable for that assumption. However, we will bring a hierarchical level understanding in the database stucture in the future. In the upper layer, the main image object will reside, and in the lower layer the objects in the image will reside as shown in Fig. 7.
First object
First image
Second object Third object
IMAGE TABLE
OBJECT TABLE
Fig. 7. The hierarchical layer. Another extension ofthe proposed work will be the incorporation of VRML to MUSTER. VRML (Virtual Reality Modelling Language) is a file format that is used to describe interactive 3D objects and virtual worlds to be experienced by the users.
VRML's technology has very broad applicability, including web-based entertainment, distributed visualization, 3D user
170
interfaces, interactive simulations for education, virtual museums, virtual retail spaces and more [9]. In most of these applications the necessaity of databases can not be ignored. For example, if one wants to shop for furniture for the house, he/she may extract the body ofthe furniture from the database and put any texture/color on it again chosen from the database. It would also be better if the customer is not only limited to the textures/colors found in the database. In other words, it would be nice if the system offers the customer to see a texture the customer likes in a picture on a different style. This is what we are trying to achieve in our proposed system. Once the images the user looks for are extracted, the whole picture , part of it or a specific color/texture within the image can be pasted on any object in a VRML scene. The objects can also be animated, deleted or new objects can be created. As an initial demo of this system, we only use the color components extracted from the database to paste onto objects. Future study will include other scenerios described above.
5. CONCLUSION in this paper, we propose a new database access, search and browsing tOOl based on ASP (Active Server Pages) where the gateway to database is achieved using ADO (ActiveX Data Objects). The main idea is that the code runs on the server machine and the resulting HTML coded page is sent to the client machine. It also provides an ODBC connection to any database system which is available on the server PC. Also, since a simple HTML coded page is sent to the client side, any browser can be used to see the results of the retrieval process. The database search is done by using color based features in the proposed system which can easily be extracted from the images. The proposed tool is also combined with a VRML based visualization interface in order to use the retrieved images to create user defined scenes.
REFERENCES 1.
2. 3.
V. N. Gudivada and V. V. Raghavan (Guest editors), "Special Issue on Content-Based Image Retrieval Systems," IEEE Computer, Sept. 1995. S. Sclaroff, L. Taycher and M. La Cascia, "Image Rover: A Content-Based Image Browser for the World Wide Web," Proc. IEEE Workshop on Content-Based Access oflmage and Video Libraries, June 1997. J. Smith and S. Chang, "An Image and Video Search Engine for the WWW," SPIE Vol. 3022, 1997.
4. M. Popescu and P. Gader, "Image Content Retrieval from Image Databases Using Feature Integration by Choquet Integral," SPIE Vol. 3656, Jan. 1999. 5.
htttp://www.microsoft.com
6. htttp://drogo.cselt.stet.it/mpeg/standards/mpeg-7/mpeg-7.htm 7. A. P. Witkin, Scale Space Filtering: A New Approach to Multi-scale Description," Image Understanding, S. Ullman and W. Richards, Eds., pp. 79-95, Ablex Publishing, N.J., 1984. 8. Y. W. Lim and S. U. Lee, "On the Color Image Segmentation Algorithm Based on the Thresholding and the Fuzzy cMeans Techniques," Pattern Recognition, Vol. 23, No. 9, pp. 935-952, 1990.
9. http://www.vrm1g 10. http://www.commweek.comlnews/newsO626-l0.htm
171
