Dynamic Hyperlinker for 3D Content Search and ...

mm13-119

1

Dynamic Hyperlinker for 3D Content Search and Retrieval M.R Swash, A. Aggoun, O. Abdulfatah, B. Li, J. C. Fernández, and E. Tsekleves

Abstract - Recently, 3D display technology, and content preparation and creation tools have been under vigorous development. Consequently, they are also widely adopted by home and professional users. 3D digital repositories are increasing and becoming available ubiquitously. However, there is not yet a single 3D content search and retrieval platform. In this paper, we propose and present the development of a novel approach for 3D content search and retrieval which is called Dynamic hyperlinker for 3D content search and retrieval. It advances 3D multimedia navigability and searchability by creating dynamic links for selectable and clickable objects in the scene while the 3D video clip is being replayed. The proposed system involves 3D video processing, such as detecting/tracking clickable objects, annotating objects, and metadata engineering. Such system attracts the attention from both home and professional users such as broadcasters and digital content providers. The experiment is conducted on Holoscopic 3D images “also known as integral images”. Index Terms— Holoscopic, Integral image, 3D, Display, Omnidirectional, 3D Search, Retrieval, Content, 3D Content

I. INTRODUCTION

D

igital information processing, especially search and retrieval, has been a hot research topic for the past few years. A number of EU funded projects have been completed that aimed to develop frameworks for multimodal processing, unedited multimedia indexing/annotating, and information extraction [1][2][3][4][5]. There has been some research focus on vertical search solutions for semantic-based search and retrieval [6]. A framework for content publishing across multiple platforms has also been developed [7]. 2D multimedia search and retrieval faces great challenges Manuscript received October 30, 2013. This work was supported by European Commission under FP7-ICT-2009-4 (3DVIVANT). “Holoscopic 3D Image Rendering For Autostereoscopic Multiview 3D Display” M.R. Swash is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK ([email protected]) A. Aggoun is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK. He is a Reader in 3D Imaging Technologies / Course Director for BMTD / Digital Media Research Coordinator ([email protected]) O. Abdulfatah is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK ([email protected]) B. Li is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK ([email protected]) J.C. Fernandez is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK ([email protected]) E. Tsekleves is with the Electronic and Computer Engineering, Brunel University London, UB8 3PH, UK. He is Lecturer and Course Director in Multimedia Technology and Design ([email protected])

considering reliability and accuracy. Thanks to the 3D imaging technology that overcomes some of challenges in 2D multimedia processing and also increasing demand that accompanies many new practical applications such as multimedia search on media asset management systems in corporations. Diverse requirements derived from these applications impose great challenges and incentives for research in the field. In this paper, we propose and develop a dynamic Hyperlinker tool for 3D content search and retrieval that easies the multimedia search and retrieval by creating hot links for selectable objects in the scene. There are a number of work steps and related components involved in creating a hypervideo. The hyperlinking parts of a video requires the detection of linkable objects, search and retrieval for relevant link targets, the specification of linking information and the transfer of metadata between the components. The link-related information are encoded using a 3D content descriptive protocol that is used to link 3D objects and push alerts to the screen/users in real time. II. THE PROPOSED HYPERLINKER TOOL The Dynamic Hyperlinker is a holoscopic [8] 3D video player with advance hyperlinking capability to ease the search and retrieval. It links objects in the scene to a similar / matching objects in the repository while the video clip is being played. The user clicks on an object in the scene to search for similar objects in the repository. It requires pre-processing for creating hot links dynamically that is analyzing and annotating 3D media content in following steps: ƒ Generate a centric viewpoint image of holoscopic 3D images for object segmentation module ƒ Generate a depth map of holoscopic 3D images for content based search and retrieval ƒ Generate Bounding Box and object masks from centric viewpoint images for Dynamic hyperlinker and content based search and retrieval module using segmentation module ƒ Generate search results “Indexed files for a holoscopic 3D image/video” from holoscopic 3D images, depth map images, centric viewpoint images, segmented object masks in content based search and retrieval module The above processes produces indexed files using 3D content descriptive protocol as shown in Figure 1. The components shown in Fig.1 communicate and exchange meta-

mm13-119 messages using the 3D content descriptive protocol. On hyper-video playback, the indexed files are used to create hot links, search and retrieval identical objects on object clicks in the scene.

2 ƒ User feedback adding/deleting search results

Fig. 3. Dynamic hyperlinking action flow diagram Fig. 1. Architecture of Hyperlinker tools for 3D Content search and retrieval

As shown in Figure 2, a hyper-video is a holoscopic 3D video clip with an associated 3D content descriptive file (Indexed file) and these two files are feed to the hyperlinker tool which creates hot links for selectable objects in real time.

Fig. 2. Block diagram of Dynamic hyperlinker tool

The Dynamic hyperlinker player loads the indexed file with a selected holoscopic 3D video during the video playback as shown in Figure 3. It monitors and tracks the computer mouse movements and when the mouse cursor hovers over the region of selectable objects, the mouse cursor icon is changed from default to hand icon to alert the user that the object is clickable or hot link. On mouse click event, it captures the mouse cursor location (x,y) and matches it with bound box in the index file and then it performs a search for identical objects in the repository passing the bounding box region to the content based search and retrieval which responds with list of search result for the region “clicked bounding box” and the result results are shown in a list box. Users can select a search result item to jump and play the scene. The hyperlinker tool supports user feedback in other word it supports a semi-automatic solution for annotating Holoscopic 3D video clips. It enables users to create a new hot link to an external source or remove an existing hot link if it picks up irrelevant result. The basic functionality of the proposed system is as follows: ƒ Import and play file to be linked ƒ Play/pause/stop a 3D video clip ƒ Highlight selectable objects ƒ Show search and retrieval results ƒ Link selectable objects to any destination resource ƒ Play target media ƒ Preview hyperlinked video

The 3D video gets paused automatically on a selectable object click as the system invokes content based search and retrieval and the found results appear in the list box. At this stage, the user can remove any irrelevant result from the result list box if necessary. Also a new source can be linked to the selected object and the system re-indexes the whole sequence frames. The re-index works only objects in the scene. It is valid until the object gets disappears. If the object goes away from scene and then it comes back, the system treads it a new object because the content based search and retrieval module retrieves objects based on its visual information. Yet there is not an attention module to detect previously appeared objects however this will be done in future work. Table 1 shows the time taken to perform various actions such as playing, linking, loading, searching, listing, adding and removing hot links. This means the proposed system can be executed on any today’s PCs/laptops without having a special computer. Table 1. The performance sheet of the hyperlinker tool

Description Bounding Box Loading time A single object highlighting time Adding new item – re-indexing time Removing an item – re-indexing time Loading/playing first frame play with highlighting disabled Loading/playing first frame play with highlighting enabled Search and retrieval time on an object click @20 research result items

Performance (seconds) 0.0009766 0.0019531 0.0742187 0.0019532 0.2792969 0.2841797 0.0175782

III. 3D CONTENT DESCRIPTIVE PROTOCOL A 3D video content descriptive protocol is developed that is based on Extensible Markup Language “XML”. It is used for 3D video content indexing including tagging selectable objects in the video clip. It is proposed for annotating 3D content as well as meta-information exchange between 3D search and retrieval components as this is the metadata schema/format between the system components. The proposed 3DCDP describes 3D video contents in a structured manner using xml which eases the system integration and improves 3D search and retrieval performance

mm13-119 because it makes it easy to lookup for scene descriptions, annotations, bounding boxes and search results. Also it is applicable for both still and video images As shown in Figure 3, CDP “Content descriptive protocol” is the root that is very structured and segmented in terms of describing a video scene including objects. As seen, a video clip contains one or more scene which has one or more annotations and in each annotation, there are one or more operators or classifiers which detect, recognize and tag selectable objects with a bounding box. A Bounding box has coordinator points “MIN/MAX X and Y values of a zone which contains a clickable object and also it has the search results embedded in it.

3 IV. 3D CONTENT VISUALIZATION The proposed tool supports holoscopic 3D videos and Figure 5 shows a video clip playback result. As seen, the selectable objects are highlighted in red rectangles and this means the objects are hot links. The user can click on any of the objects to search for similar objects in the repository. The search results are listed in the list box. The user can view/play the result by selecting it from the list box.

Fig.5. The proposed Dynamic hyperlinker player screen

V. CONCLUSION In this paper we have proposed and developed a Dynamic hyperlinker including 3D content descriptive protocol for 3D search and retrieval that enable users to search, retrieve and visualize holoscopic 3D video clips by clicking on selectable object in the scene while the video clip is being played. At this stage, the holoscopic 3D videos are preprocessed by 3D operators such as centric viewpoint extraction, depth map creation, segmentation and content based search and retrieval module. These 3D operators use a 3D content descriptive protocol to exchange meta-message as well as annotating the media content. The proposed system advances user interaction and easies the multimedia content search and retrieval. The experiment is conducted on Holoscopic 3D content and it is applicable and scalable to any other 3D content. It is worthwhile mentioning that dynamic hyperlinker performed well on 3D video sequences therefore it is an ideal tool for 3D content search and retrieval. ACKNOWLEDGMENT

Fig 4. Block diagram of the proposed 3D Video Content Descriptive Protocol

The proposed approach is effective for indexing, searching and navigating a video content and also this metadata can be used for creating statistical reports for the video clip e.g. number of selectable objects, number of scene, and fast/slow motions.

This work has been supported by European Commission under Grant FP7-ICT-2009-4 (3DVIVANT). The authors wish to express their gratitude and thanks for the support given throughout the project. REFERENCES [1] Accessed date: January, 2013. “RUSHES - Retrieval of mUltimedia Semantic units for enHanced rEuSability”, http://cordis.europa.eu/ist/kct/rushes_synopsis.htm

mm13-119 [2] Accessed date: January, 2013. “BOEMIE – Bootstrapping Ontology Evolution with Multimedia Information Extraction”, http://cordis.europa.eu/ist/kct/boemie_synopsis.htm [3] Accessed date: January, 2013. “aceMedia - create, communicate, find, consume, re-use”, http://cordis.europa.eu/ist/kct/acemedia_synopsis.htm [4] Accessed date: January, 2013. “CARETAKER - Content Analysis and REtrieval Technologies to Apply Knowledge Extraction to massive Recording”, http://cordis.europa.eu/ist/kct/caretaker_synopsis.htm [5] Accessed date: January, 2013. “K-Space - Knowledge Space of Semantic inference for automatic annotation and retrieval of multimedia content”, http://cordis.europa.eu/ist/kct/kspace-synopsis.htm [6] Accessed date: January, 2013. “Superpeer Semantic Search Engine”, http://cordis.europa.eu/ist/kct/alvis_synopsis.htm [7] Accessed date: January, 2013, “Content4All - Cross Platform Tools for Community Content Publishing”, http://cordis.europa.eu/ist/kct/content4all_synopsis.htm [8] G. Lippmann, “Epreuves reversible”, Comptes rendus hebdomadaires des Seances de l'Academie des Sciences 146, 446-451, March 1908.

4