Image segmentation search engine applied to a distributed archiving architecture for content retrieval system to educational products Sam H. Minelli, Andrea de Polo Fratelli Alinari Photo Archives, Largo Alinari 15, 50123 Florence, Italy {
[email protected];
[email protected]}
Abstract The advanced access to different databases by means of one single interface has become a critical and important requirement for most archives and content providers. This paper will describe an application case for a system architecture capable of retrieving large and diversified contents by means of innovative networking and search engines based on the Qimera platform. The system is supposed to automatically select contents from different repositories, and making them available trough an interface which has editing tools for direct usage of the contents. The application is studied to be integrated into educational products delivering dynamic composite content by means of a simple and transparent user interface1.
Keywords — distributed database, IPR management, content management, digital image archiving, image segmentation, Qimera, JPEG 2000.
1. Introduction Image archives and generally speaking, digital content providers, have created their web sites getting in contact with new market segments both for business and for educational purposes. At this time an aggregation process has been revealed: small and medium sized content providers are going to create partnerships and share knowledge and market. A netarchive can be seen somehow as a big store where the customer can find lots of different products in the same place. In this sense the customer will be able to find both a fashion image or a movie. We refer to the cultural heritage domain, where the end user can search for an image of the royal Habsburg family both inside Alinari’s archive and inside the Austrian National Library’s using the same interface and even not knowing to access different databases. This involves the developing of distributed processes not only for the delivery 1
This work was supported in part by the IST programme of the European Community in the SCHEMA project ( IST-2001-32795) ; by the e-Content programme TNT (EDC-33015 TNT/27966); by the e-TEN programme ORPHEUS project (e-TEN: C27983). Sam Habibi Minelli is with the Electrical Engineering R&D Unit, Alinari (email:
[email protected]). Andrea de Polo, is with the European Project, Alinari (e-mail:
[email protected]).
but also for the content management, from the image generating to the cataloguing or information adding. One of the most important consequences of this net-partnership is the creation of new IPR management and content protection procedures, such as the creation of trusted networks or trusted nodes. This paper describes a possible implementation where the content providers are at the center of the architecture and can modify and personalize the access to their data, enhancing the services by means of advanced search engines (such as image segmentation engines) that could be integrated into educational products.
2. From local to distributed The actual state of the art, in the domain of content providers (mainly archives and cultural institutions), express the flourishing of distributed configurations of contents and services. Networks of contents seem to be able substituting the centralized configurations. The idea of a commune interface to retrieve information from different sources is not new (some research project with this goal have been explored by the REGNET European project [8] and TNT EDC-33015 TNT/27966), but in most cases the end nodes are expected to have commune databases structure and protocols with same services while the manager-node is not the content provider itself with the generation of conflicts. The main reasons that support the distributed archiving are: • • • • • •
Scalability: nodes can be added or removed without reconfiguring the architecture. Lower costs for later starts up: after the base system has been realized, any new add of a node has lower costs for the start up. Robustness : if any node fails, the network will supply the node that is missing. Speed: a distributed network of nodes guarantees quick connections. Security: the data are stored locally by the owners (the central repository is virtual). IPR distributed management.
3. Form service-centres to content-centres
all the Content Providers are cut off from the network.
During the last few years many content providers (museums, archives, libraries) have started their business giving access trough internet to their contents. The content providers could be divided referring their size of data, contents etc., but a great difference for the access to internet trading is the budget and the technical infrastructure they have. There are a lot of CPs (Content Providers) operating in very specific domains without any internal IT department (knowledge referring to thesauri, data bases, internet services). Moreover, these content owners wont even have locally all the content required by the market (images, metadata, music, text, footage). The external portals offer good services but CPs hardly let out their contents on remote servers and the lifetime of contents is very different from the lifetime of technology used (usually the second is much shorter). As soon as more efficient DB and new search functionalities are requested by the market, they should migrate from one technology to another. Lastly, the CPs usually would be able to become more independent personalizing their contents, the tools and functionalities, and the services. Following the previous general requirements the architecture lets the possibility to isolate a cell if out of order without compromising the correct functioning of all the netarchive. An example of image retrieval and payment: the user searches in the DA (Distributed Archive) the images of an actor or personage. The system automatically (and transparently) searches inside Alinari’s, inside Austrian National Library’s, Italian Catalogue Institute’s, etc. and shows the results in one commune page to the user. When the user selects 3 images and asks for the rate, the DA system queries in the local servers and shows the results. Then if the user asks to pay, the DA system connects the user to the DABank-server which provides the distributions of benefits to the content owners.
Figure 2 : Distributed configuration topology. In this configuration at any time one DA-server fails, another DA-server will still work and can supply the services or contents of the broken node.
4. Qimera Before describing the application to the educational domain, few words must be said on the Qimera platform. The Qimera software uses a graphical user interface that is platform independent and through the communication interface lets the user activate different engines. The image analyser engines have been developed by the SCHEMA project technical partners who have realized different operative modules. The analysis techniques that are under testing, use two approaches to the segmentation: the regionbased segmentation and the object-based segmentation. The region-based segmentation module can activate the following algorithms: •
Recursive Shortest Spanning Tree (RSST) algorithm.
•
K-Means-with-connectivity-constraint (KMCC).
•
EM-based segmentation in 6D colour/texture space.
•
Pseudo Flat Zone Loop Algorithm .
The object-based segmentation module can activate the following algorithms:
Figure 1: Partially distributed configuration topology. Notice that if the main DA-server web fails,
•
Semi-automatic segmentation via modified RSST.
•
Level-set based snake segmentation .
Distributing and aggregating is a process that needs protocols and standardizations. Lots of small/medium content repositories have been stored using different data bases and different procedures (i.e. universities, archives, hospitals, museums, etc.) and the aggregating process involves digital
content owners, trying to make available different contents such as: audio, video, image and text format. The DA provides a proper XSLT style-sheet if the new partner’s database can be interfaced by XML data, otherwise it provides a specific layer to adapt and reach the local data to the commune outcome. The search engines can be then integrated with the traditional searching modes (by keyword; advanced; by list from some thesauri, etc.) and with the new one Qimera that could also allow the search in the contents that have not been catalogued yet with metadata. The semi-automatic object segmentation offered by Qimera, allows the interaction by the user who initially defines what objects to be segmented on the input image. The user selection will be then processed and eventually again refined interactively. The semi-automatic segmentation feature will have important application as up to now, all the users that were interviewed, showed real needs for interaction and refinement of their searches.
Figure 3: Query workflow with evidence on the Qimera process.
5. The application to the educational domain The distributed architecture is under development in order to aggregate image archives; libraries; hospital repositories; educational institute repositories. Innovative educational products will be then implemented. A researcher could need to study the human body images (retrieved from the hospital archive) and the drawings taken from an old manuscript (retrieved from the National Library’s database). The user could also require advanced image analysis and delivery tools: the student visiting a town navigates the map using a palm device, the map is retrieved from the geographical archive and could have links to cultural services such as guided tours (retrieved from the museum network). In some cases the segmentation engine is the only way to retrieve data (typically when the image repository has not cataloguing metadata at all). An integrated educational application for both universities and high schools is expected to allow the professors and teachers to create specific course modules and tests for examining the students. Three examples: • History domain application: the professor requires the students to find images referring historical personages and reporting on the places, events, chronicle. • Art domain application: the professor requires the students to analyse a painting reproduction and to find similarities and correspondences with similar artists. • Science domain application: the student preparing an exam on astronomy needs images on eclipses and planets from an image database that has not been catalogued by keywords (rough image database).
Figure 4: Query workflow with evidence on the archive data base interface block diagram. Recent market analysis have shown the interest from the university decision makers to acquire tools that integrate general functionalities (i.e. mailing, organizing, institution calendar and deadlines, courses,…) with course tailored features. An advanced search engine that could combine images, text, and other multimedia contents trough one educational interface which manages the IPR references can answer to the need of accessing contents and use them immediately and directly inside the same framework. The image segmentation software has the capability to generate information from a set of images, this information could constitute a new vocabulary that can be interfaced with the text database. The distributed image-archive will need a specific search agent able to “retrieve images by images”. This means that a customer could be interested to find all the images that are someway similar to one specific image he
chooses (i.e. a personage photo). Then the search result can be edited by means of the same interface and enriched with comments and other information and lastly sent to the students (or any other user).
providers. In fact, some content providers use to change their copyright requirements frequently and a centralized management would not allow to contemplate all the different domain needs.
6. Integration of new technologies
8. Conclusions
Lots of digital archives already offer their services trough the Web, the mobile technologies have been explored (as example, image archives providing Multimedia Messaging Service contents: images with simple text messages and jingles) but the possibility to access the content is still considered by the users mainly as opening the browser and searching images or musical excerpts trough some search engine (and the search engines usually give no added value nor stimulate to interaction). As long as innovative channels are going to be accessible, the digital archives need to integrate new technologies and services for digital image processing and delivery. The demand for services generates also new problems to be solved as security, migration from previous file formats and closed architectures, and integration into new systems. The possibility to search images in a fast way and to be able to use content retrieved inside an interactive architecture, transparently, seems to be an important challenge. The market demand pushes towards products that can be easily searched and immediately integrated in other products and services. New file formats such as JPEG 2000, has been tested for its interesting features even if the Qimera segmentation platform does not at this time support the JP2 file format. The JPEG 2000 standard will include most of the features required by image archives (JPWL for wireless; JPSEC for security and access control; JPIP for interactivity and interoperability; JP3D for volumetric and 3D imaging) and could be considered also for performance improvement of Qimera as the JP2 file contains metadata.
7. Distributed intellectual property right management When considering contents such as images or music excerpts or texts, there must be a strong IPR management system. In the case of a number of content providers the IPR management must adapt to each single content owner requirements and be compliant to the local IPR legislation. For the first prototype, encryption and security techniques have been implemented and tested (such as MD5 digital signature; JPEG 2000 Part 4 Registration process; Key generation and management; Secured DES encrypted archive [6]). The Digital Right Manager System is expected to let on line registration and delivery of a License Plate as unique identifier; the creation of an archive containing digital signatures of registered contents; to document encryption for undisclosed content archiving and to create multiple versions of a same document with the same identifier. The DRMS must be distributed allowing personalization by the content
The image segmentation search engine called Qimera, can be integrated into a distributed archiving architecture for content retrieval system to realize educational products and support the teaching workflow. Both teachers and students could access much higher volume of digital contents (images, text, video) with the possibility to interact inside a commune platform. The distributed architecture guarantees IPR management and reduces the general costs.
9. References [1]
Suhayl Masud – Building a real-world web service- RosettaNet-XML Journal, volume 4, issue2, pages10-15, February 2003. [2] Greg Colyer, James Holwell - Elysium Ltd (UK) WG 1 N2665, 11 July 2002. [3] C. Serrão , J.M. Salles Dias, J. Messens, JM. Boucqueau, ISO/IEC JTC 1/SC 29/WG 1- ITU-SG8- Coding of Still Pictures WG1N2824 -Use case - remote sensing. [4] N. O’Connor, S. Sav, T. Adamek, V. Mezaris, I. Kompatsiaris, T. Y. Lui, E. Izquierdo, C. F. Bennström, J. R Casas, “Region and object segmentation algorithms in the QIMERA segmentation platform.” Paper to appear in CBMI 2003. [5] S. H. Minelli, “Applicazione della Tecnologia MMS: Archivio Digitale Alinari.” Alinari’s document, November 2002. [6] Jean Barda -CAVEAT:” Controlling Authenticity and Versioning of an Electronic document by Accessing a Trusted third party”, Netimage, October 2003. [7] JPIP Editors, “ISO/IEC JTC 1/SC 29/WG 1 - (ITU-T SG16) Coding of Still Pictures”. [8] S. H. Minelli- Content Databases Management and System Standards for a Digital Image Archive - EC Project IST-2000-26336 REGNET: Regional Networks of Culture Heritage – Proceedings. Institute of Computer and Communication Systems, Bulgarian Academy of Sciences-Sofia BULGARIA-(Nov. 2002). [9] E. Tuncel, L. Onural, “Utilization of the recursive shortest spanning tree algorithm for video-object segmentation by 2-D affine motion modelling”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, no.5, August 2000 [10] I. Kompatsiaris and M. G. Strintzis, “Spatiotemporal Segmentation and Tracking of Objects for Visualization of Videoconference Image Sequences”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 10, no. 8, Dec. 2000. [11] N. O’Connor, T. Adamek, S. Sav, N. Murphy, S. Marlow, “Qimera: a software platform for video object segmentation and tracking”, to appear in Proc. WIAMIS 2003. Sam H. Minelli received his M.S. degree in Electronic Engineering from the University of Padua (Italy). His research interests include segmentation and compression algorithms and quality assessments. He is currently involved with the testing group for the JPEG2000 compression algorithms and webarchives at the R&D Lab at Alinari photo archive in Florence (Italy). Andrea de Polo is senior project manager at the Alinari company in Florence (Italy). His research interests include intellectual property rights and watermarking. He is an I3A member since 2002, member of The Photo Marketing Association (PMA). This author became a Member (grade A) of IEEE in 2003. He is currently involved with the art-on-demand and educational projects.
