Metadata Integration and Querying- A Case of MPEG-7

Metadata Integration and Querying- A Case of MPEG-7 Muhammad Javed School of Computing, Dublin City University, Ireland [email protected] Abstract Metadata management is an emergent field attracting researcher’s considerable attention recently. This paper describes how metadata of audiovisual content is transformed to mp4 file format and then connected to an XML knowledgebase for performing various queries on the saved metadata. This resulted in delivery of media as well as MPEG-7 metadata in one file for content searching, content retrieval and overall management. This paper also describes the additions and changes made to MPEG4IP, an open source tool provided by Motion Picture Expert Group (MPEG).

1. Introduction

M

PEG-7, formally known as Multimedia Content Description Interface, is a standardization effort by the International Standards Organization (ISO) with active collaboration with other stakeholders(broadcasting companies, electronic goods manufacturers, telecommunication service providers etc.) This is a metadata standard for description of multimedia content, used for a broad spectrum of applications like multimedia archives, search engines, media production supports, education, and entertainment. It does not deal with the actual encoding of audio or video, which makes it different from its predecessors (MPEG-1, MPEG-2, and MPEG-4). It stores metadata to search and tag particular events and deals with all aspects of multimedia including audio, video, images and 3-D graphics[4], [5]. The standardization efforts started in 1996 when preceding standards like MPEG-1, MPEG-2 and MPEG-4 helped to realize the possibilities of interactive video on CD-ROM and Digital Television. These standards did not provide information about the multimedia content; they only provided presentation of content for media players. Hence the issues related to data management, data selection, data search and data filtering were not adequately sorted out, which necessitated MPEG-7 standard [4], [5].

The information about the multimedia content, gener1ically called as metadata, is a critical factor for effective data management. Metadata management refers to the process of applying administrative policies on storage of metadata to improve content management [9]. In MPEG-7, it is stored as an XML file [6], [7], [8]. The information about the multimedia content is described using its Multimedia Content Description Interface [1], [4], [5]. Such information can refer to media content (e.g. lyrics of a song, a person in the video clip), administrative information (e.g. copyright information, location of media) preservation aspects (e.g. tools used to create media), technical aspects (e.g. applied coding formats and parameters) and usage aspects (e.g. user tracking) [5]. MPEG-7 consists of number of components including Descriptors (D), Description Schemes (DS) and Description Definition Language (DDL). A descriptor is a demonstration of any characteristic or a description of a feature and defines the syntax and semantics of its representation. It is also possible to have multiple descriptors defining a single feature. The standard provides standardized ‘audio only’ and ‘visual only’ descriptors. The Multimedia Description Schemes (MDS) component provides standardized description schemes involving both audio and visual descriptors. DDL is a schema language and used for defining the composition of an MPEG-7 document. It provides a standardized language to express description schemes [4], [5]. Another component associated with MPEG is the mp4 file. The mp4 file can contain audio, video, text information or combination of them. Text could be the information about the audio/video streams, known as metadata. The metadata used in our case was MPEG-7 standard metadata. Most of the popular audio/video tools such as iTunes, QuickTime Player, Real Player, Nero Media Player, Media Player Classis etc, support the mp4 file format. It is designed to contain media information for presentation in a flexible and extensible format that facilitates interchange, management, editing and presentation. The filename is denoted by .mp4 extension and its organization is based on the ISO File Format. A 1

At time of research, author was working as Internee at T-Systems Nova GmbH. Germany.

simple ISO file is described in Figure 1. The file can normally be decomposed into a number of constituent objects (boxes). Each object is denoted by an object descriptor which describe the streams that makeup those objects. The MPEG-4 file format contains MPEG-4 encoded video and Advanced Audio Coding (AAC)-encoded audio content [10, 11]. ISO File moov ..other boxes

telecommunication industry. Metadata streaming is required for the events taking place in real time e.g. sports and news. Metadata streaming can also be used for other interactive applications, for example in the education sector. As MPEG-4 applications are used widely through the use of Internet and on mobile devices, it requires contents search and its manipulation and management. Thus it is a worthwhile aim to combine MPEG-4 with metadata for contents description [1].

mdat track (Video) track (Audio)

Interleaved, timeordered, Video and audio frames

Figure 1: Simple ISO file

In our current study, MPEG-7 metadata is added to the ISO based mp4 file for distribution on different channels. This integration was done at the time of creation of the mp4 file. At later stage this MPEG-7 metadata can also be retrieved from the mp4 file and can be saved as an xml file into the hard disk or into the XML Database. In our project, the xml database used was Apache Xindice. It is an open source native xml database. In Xindice, the xml data is stored into a number of document collections. One can also create his own document collections, however the default document collection is named as /db. The creation of user defined document collections creates hierarchy structure and makes it easier to search for particular metadata information. The search of any metadata information can be done on document as well as on the collection level. The xml documents saved into the database are tagged with date stamp which makes more convenience in search process [14]. As mentioned before, an mp4 file is structured as a sequence of boxes. Each box has a header, which provides information about the size and the type of the box. The standard size of the header is 32 bit. The major boxes are File Box, Movie Box, Track Box, Media Data Box and Metadata Box. The metadata can be integrated either in File Box, Movie Box or in the Track Box [13]. Present work is part of a much bigger project whose objective is: 1) to create the textual and audio/visual descriptors, 2) to save textual and audio/visual metadata in an XML file and 3) finally to integrate it into the mp4 file at the time of creation. This integrated metadata, if required, can also be retrieved from the mp4 file.

2. Problem Definition MPEG-4 standard provides a great encoding competence by an object-base encoding method and makes it feasible to present and administer content of each media data. This characteristic provides a useful application for interactive broadcasting as well as to

3. Development Environment In order to evaluate the results of current study we did the implementation using an open source programming system called MPEG4IP [13]. This programming system is based on the ISO File Format standards and used for streaming, encoding and playing MPEG-4 encoded audio/video files as well as for video and other multimedia streaming over the Internet. This software provides a library “libmp4v2_st” which is an important part of the program set. It is structured according to the ISO Base Media File Format and used for implementing and providing Application Programming Interface (API) for mp4 file format. In order to get familiarity of development environment, it is important to understand the structure of the library, which is shown in Figure 2 and described as follows: MP4Atom

MP4XmlAtom MP4RootAtom MP4MetaAtom MP4MoovAtom

MP4TrakAtom

Figure 2: MP4Atom and its derived classes

As most of the files are organized as a series of boxes, we have a separate class for each box. The class name starts with ‘MP4’ and ends with ‘Atom’ the middle part of the filename have the box’s name, e.g. for Movie Box we have “MP4MoovAtom” class and for the Handler part we have “MP4HdlrAtom” class. There are individual classes for different properties, e.g. “MP4StringProperty” and similarly for different descriptors, there are individual classes. e.g. “MP4ESDescriptor”, “MP4ESUpdateDescriptor”. A class named “MP4File” consists of functions (such as “AddChildAtom”, “GetStringProperty” etc.) which

are used during creation, reading and manipulation of mp4 file. Another class named “MP4Atom” is the parent class of all atom classes (e.g. MP4MoovAtom, MP4MetaAtom) and consists of certain basic functions which are used for an Atom, e.g. “Create Atom”, “ReadChildAtoms” etc. Class named “MP4Error” is used for displaying error messages [1].

4. System Design The MPEG4IP software provides a utility called “mp4creator”, mainly to create and access mp4 file. During the file creation process, “mp4creator” utilizes methods from the mp4 API. The API functions for adding, reading and writing MPEG-7 metadata has been added to the library in the scope of this project. Some of the important API functions which are used to save and retrieve MPEG-7 metadata are MP4Create, MP4Read, MP4AddMP7Meta, MP4WriteMP7Meta, and MP4ReadMP7Meta. MP4Create is used to call a function named as “create ()” in class “MP4file” to create a new mp4 file. MP4Read is used to call a function named as “read ()” in class “MP4file” to read an mp4 file. During retrieval of MPEG-7 metadata this method is used. MP4AddMp7Meta is used to call a function named as “AddMP7Meta ()” in class “MP4file” to create MPEG-7 metadata boxes. It also checks whether data is in textual or binary format and returns the path where to save metadata. MP4WriteMeta is used to call a function named as “WriteMeta ()” in class “MP4file” to create MPEG-7 metadata boxes. This method gets the path where to save metadata from function “AddMP7Meta” as a parameter and then saves the metadata accordingly. MP4ReadMeta is used to call a function named as “ReadMeta ()” in class “MP4file” to retrieve metadata from an mp4 file. First it calls function “Read ()” to read mp4 file and then retrieves MPEG-7 metadata using function “ReadMeta ()”. For the creation of mp4 file, which includes MPEG-7 metadata in addition, changes were necessary in the library. The following sections provide more details about the work done for amendments of library [1].

4.1. Creation of Additional MPEG-7 Metadata Atoms:

Classes

for

As each box has its own associated class and that MPEG-7 metadata is saved in a meta box named ‘meta’. It was, therefore, necessary to create some additional classes for saving MPEG-7 metadata in mp4 file. In the current project, MP4MetaAtom, MP4IlocAtom, MP4XmlAtom and MP4BxmlAtom classes were created.

4.2. Modification “MP4TrakAtom” Classes:

to “MP4MoovAtom”, and “MP4RootAtom”

Metadata can be saved in either File Box, Movie Box or in a Track Box. The metadata box containing the metadata is created at file level, if this document is planned to be mainly a metadata document. When the metadata annotate one stream, then the metadata is saved at the track level. However, if the metadata annotates an entire presentation, then the metadata is saved at the movie level. Thus it was felt necessary to make amendments to certain existing classes, including: MP4MoovAtom, MP4TrakAtom and MP4RootAtom. By adding ‘meta’ atom as an expected child atom, ‘meta’ atom and its expected child atom (i.e. ‘hdlr’, ‘dinf’, ‘iloc’, ’xml ’, ’bxml’) will automatically be created during the creation of an mp4 file.

4.3. Addition of New “MP4File” Class:

Functions

in

the

If the XML document is in textual format, this data will be saved in the “xml” box. If the XML document is in the binary format, data will be saved in the “bxml” box. For setting the path to where metadata is to be saved in an mp4 file, a new function AddMP7Meta() was created. This function creates the MPEG-7 metadata boxes, checks whether data is in the textual or binary format and returns the pathname where the metadata is to be saved. Refer to Figure 3. mp4creator MP4AddMP7Meta( )

creates the meta box and its child boxes

Global Functions

m p4file

AddMP7Mata( )

returns the path where to save m etadata

Figure 3: Program flow of creating meta boxes and setting path for saving metadata

For saving MPEG-7 metadata in an mp4 file, a new function named WriteMP7Meta() was created. As metadata boxes already exist (created by the AddMP7Meta() method), this function decides where metadata is to be saved and then saves it according to the scheme shown in Figure 4. Similarly, for retrieving the metadata from the mp4 file, a function named as “ReadMP7Meta()” was created. The

purpose of this method is to get the selected mp4 file and retrieve the required metadata. Global Functions

mp4creator

W riteMP7Mata( )

MP4W riteMP7Meta( )

Checks where to save MPEG-7 metadata

m p4file

Saves the MPEG-7 m etadata in MP4 file.

Figure 4: Program flow for saving metadata

5. System Deployment The execution of MPEG4IP reference software (as modified as mentioned above) required a command line input, however, to enhance the usability of the package, a front end interface was developed using Java which provided linkage to the reference software built in C++. Java Native Interface was used for interfacing. The software is connected to the XML database for saving, retrieving and executing different type of queries on saved metadata [2], [3]. There are a variety of XML database solutions provided by commercial products, open source projects, and research prototypes, [c. f. 5, 15, 16] database selected for this project was Apache Xindice [12]. The decision to use this system was based on the fact that in many database systems XPath (query language for XML) is implemented on each document individually. However in Xindice, queries can be made either on a single document or on a collection of documents. This facility was found useful as it means that a query can run against multiple documents. For Java programmers, Xindice also provides an implementation of the API. To help the end users, a graphical user interface was also constructed which is used to access the XML database as well as the MPEG4IP reference software for saving and retrieving MPEG-7 metadata to and from mp4 files. The component design of the system is shown in Figure 5. DATABASE (XML)

Passing parameters to C++ Code

6. Summary and Future Work The recent advancement of digitalization of television has enhanced the importance of MPEG series and this family of standards is going to play a leading role in the world of widely available multimedia. In our project, we have modified the MPEG4IP open source software to enhance the MPEG-4 based mp4 files and made it possible to deliver the audio/visual data along with MPEG-7 metadata in one file for content searching and management. This functionality was unavailable before, in the original MPEG4IP code. The application was tested successfully to convert a number of audio/video files of different format such as Avi, Wav, mp3, AAC3 etc, into mp4 files and addition of MPEG7 metadata. It is hoped that the software community will find our contribution useful and advantageous.

7. Acknowledgment This work is supported by T-System Nova GmbH, Darmstadt Germany and this could have not realized without the constant help and encouragement from all colleagues.

8. References [1] Javed, M. Realization of Metadata Integration within the MPEG-4 File Format and Storage within an XML Database, Master Thesis, FH Darmstadt, Nov 2003.

[2] Saeed, S.; Kunz, C., Loading arbitrary knowledge bases in Matrix Browser, Proceedings of INMIC 2004, 8th International Multi Topic Conference, 2004, pp. 631-634, Lahore, Pakistan 24-26 Dec. 2004.

[3] Saeed, S.; Kunz, C., A Framework for Mining Ontological Information Nets In Journal of Applied Sciences Volume 6(3) pp 595-597 2006.

[4] ISO/IEC JTC 1/SC 29/WG 11. 1999. MPEG-7: Context, Objectives and Technical Roadmap, V.12. ISO/IEC Document N2861. July. International Organization for Standardization/ International Electro technical Commission (ISO/IEC), Geneva, Switzerland. Web site: www.iso.ch.

[5] Westermann, U. and Klas, W. 2003. An analysis of XML database solutions for the management of MPEG-7 media descriptions. ACM Comput. Surv. Vol35(4) (Dec. 2003), pp. 331-373.

[6] Curtis, K., Foster, P., and Stentiford, F. 1999. GUI (JAVA)

Java Native Interface

MPEG4IP (C/C++)

Figure 5: Component design of system

Metadata—the key to content management services. In Proceedings of the 3rd IEEE Meta-Data Conference (Bethesda, MD). [7] Sheth, A. and Klas, W., Eds. 1998. Multimedia Data Management: Using Metadata To Integrate and Apply Digital Media. McGraw-Hill, New York, NY.

[8] Kashyap, V., Shah, K., and Sheth, A. 1995. Metadata for building the multimedia patch quilt. In Multimedia Database Systems: Issues and Research Directions, S. Jajodia and V. Subrahmanian, Eds. Springer Verlag, London, UK.

[9] Metadata management: An essential ingredient for Information Lifecycle Management White paper October 2005. Available http://www.sun.com/storagetek/whitepapers /Metadata_Management.pdf

[10] MP4 File Format (14496-14) white paper April 2006. Available: http://www.chiariglione.org/mpeg/technologies/ mp04-mp4-ff/

[11] MP4, Information about the Multimedia files Type http://support.microsoft.com/kb/316992 [12] Introduction to Xindice; An open source native XML database system. Available: http://www-106.ibm.com/ developerworks/opensource/library/wa-xindice.html

[13] MPEG4IP: Open Source, Open Standards, Open Streaming. Available: http://mpeg4ip.sourceforge.net/

[14] Apache Xindice ; Xindice (Zeen-Dee-Chay) Available : http://xml.apache.org/xindice/index.html

[15] Khan, R. A.; Qureshi, A. and Saeed, S. “Vision Based Navigation for a Mobile Robot with Different Field of Views” in Proceedings of Pak-US International Symposium on High Optical Networks and Enabling Technologies (HONET 2005) Islamabad, Pakistan 19th – 21st December 2005.

[16] Safdar, S.; Ahmad, J.; Ahmed, S.; Asghar, M. T. and Saeed, S. “Agent based Model for providing Optimized, Synchronized and Failure free Execution of Workflow Process” in Proceedings of 4th International Conference on New Exploratory Technology (NEXT 2007) Seoul, Korea 25th – 27th October 2007.