A MPEG-21 Based Architecture for Data Visualization in Multimedia Web Applications Adriana Reveiu, Marian Dardala, Ion Smeureanu Academy of Economic Studies Bucharest, Romania
[email protected],
[email protected],
[email protected]
Abstract The development of an open architecture for multimedia data integration and visualization in template based distributed applications is a challenge for the scientific researches. Because of characteristics of multimedia data, their management and querying techniques are unlike than those of traditional data. We can increase the potential use of multimedia data across various applications by storing and managing data using multimedia databases. A standard solution to define the architecture of multimedia systems assures the integration of multimedia web application with the other applications. The goal of the paper is to propose an architecture based on MPEG-21 (Moving Picture Experts Group) standard for various multimedia data types integration, managing and visualization into web applications. Keywords: Data Integration, MPEG-21 Standard, Multimedia Databases, Multimedia Web Application
1. Introduction The development of multimedia applications, from the last years, due to the exponential growth of the Internet had as consequence a great usage of multimedia data. As a result, the importance of the researches in multimedia technologies field had been increasing. Multimedia data is heterogeneous from many points of view: • some data is time dependent and the other is time independent, • multimedia data uses different formats for data representing, • some data is structured and some data is represented as unstructured or as semi-structured streams of data, • some data can be transferred remotely in a short time and the others needs a large period to be transferred. The paper presents the architecture of multimedia system based on MPEG-21 standard and the implementation of this architecture as a distributed multimedia web system.
The first parts of the paper present the main elements, concepts and standards used to define the system’s architecture and in the last part, we present a technical solution based on the proposed architecture. We use the MPEG-21 standard to define template-based architecture and to generate automatically multimedia web applications because this standard allows defining a robust and flexible multimedia opened framework. We used the MPEG-21 standard as support to build a distributed architecture and to implement the multimedia elements interactions and visualization. The paper uses standardized solutions to build dynamically multimedia web applications because these facilitate data exchange with other applications.
2. Developing Template Based Multimedia Web Applications We can think about a multimedia application like a collection of multimedia elements: pictures, texts, video and audio sequences and a range of spatial and temporal information used to describe the places of multimedia elements within the application. If we choose to create multimedia presentations in a manual authoring way, it is necessary to explicitly specified, for each multimedia component, the place and the time in which it must appear in the presentation. In this situation, it is necessary to modify each time the presentation or to build different versions for the same presentation, whenever the context of the presentation changes. If it is necessary to modify some elements of multimedia presentation, or if we want to deliver the same application on different kinds of platforms, like the Internet environment, on mobile devices with various capabilities, or if the presentation is to be adapted to the user preferences, it is necessary to create different versions for the same multimedia presentation. One solution is to build a template for the presentations, and to generate automatically a new presentation for each particular situation or to recreate automatically the presentation when it is necessary to update or to replace the multimedia elements. To do this, it is necessary to build a semantic system that must contain the relationships between multimedia elements and the semantic characteristics of each element. We can use this kind of system to build automatically complex
multimedia presentations, adapted to some criteria, but keeping unchanged the messages of the presentation. The main stages to create a template based multimedia applications are: • Multimedia content description, • Metadata definition, • The management of multimedia data and multimedia descriptions, • Multimedia data integration and usage.
2.1. Multimedia content description Metadata definition is the base for multimedia content description. Metadata is data about data. Metadata is used to describe and to store some characteristics of multimedia resources and is used to decide quickly how can be processed the multimedia content. The usage of metadata is suitable for multimedia resources. There are a lot of standards and research projects having as a goal the multimedia content description, the most successful being: Dublin Core Standard, Metadata Dictionary SMPTE (Society of Motion Picture and Television Engineers), MPEG-7 and MPEG-21. These are general standards; there are not dedicated to some application topologies, with applicability in various areas and having the support of some prestigious international organizations. [1] Dublin Core standard uses as support for multimedia resources description a set of metadata. This metadata facilitates the electronic resources detection. The goal of Dublin Core standard is to describe multimedia resources for some TV archives. Metadata Dictionary SMPTE project defined a large collection of names and data types developed for TV and video industry. The aim of this project was to propose a universal method for resources management and a hierarchical solution for data structure. It uses metadata to describe attributes for environment description. The main disadvantage of this system is that it does not use the semantic annotations. MPEG-7 is a standard for multimedia metadata description, based on XML opened standard. MPEG-7 proposes descriptive elements for the whole succession of multimedia processing: from multimedia data acquisition, i.e. devices classifications, devices characteristics description, to data analyze and filtering, to delivery, i.e. environment descriptors, and till the interface with the final users, i.e. descriptors for users’ preferences. [2] MPEG-21 standard defines a robust and flexible multimedia opened framework, which identify a distributed architecture for multimedia elements interactions. The aim of the MPEG-21 standard is to define an opened framework that allows transparent usage and multimedia resources adaptation, using various networks and terminal devices
2.2. Metadata definition The efficient usage of metadata in a multimedia distributed system needs a multimedia database for metadata storage, management, querying and delivery. From metadata point of view, the database must contain information about multimedia data location, about metadata storage location and the specific requirements of multimedia resources. To generalize the multimedia data and metadata usage in a dynamic, distributed system, it is necessary to resolve two types of problems. The first problem is about the multimedia data: the development or the usage of scalable and compression technologies, adaptable to the changing environments. The second problem is about the models used for multimedia data representation, for multimedia content adaptation, to extract the information about multimedia data and to associate the metadata with the multimedia data. The processing of a large amount of data, the multimedia resources, is time consuming, and in some cases, it is not possible to process multimedia resources in a right time. Because metadata has a small size, it is a text description; the extraction and the processing of metadata allow processing quickly information about multimedia resources and about multimedia content. The usage of metadata in distributed multimedia systems has many advantages: this allows querying the multimedia data function on multimedia content and is based not only on the textual characteristics associated with multimedia data but also on the multimedia content. Before querying the multimedia data based on multimedia content, it is necessary to index the multimedia data. This supposes to extract automatically the metadata associated with multimedia data or to make manually annotation for multimedia data. We can use metadata to describe: • the characteristics of environment, • the users’ preferences regarding the content, • the presentation form for multimedia data, • the speed of networks, and • terminals constrains.
3. MPEG-21 standard presentation MPEG-21 is an ISO/IEC 21000 standard of Moving Picture Experts Group (MPEG) that defines an open framework for multimedia. The power of MPEG-21 is demonstrate in the following situation: there are many multimedia resources that could be used to develop an infrastructure for multimedia content delivery and consume, but there is no architecture to describe the interaction way from theses elements. The goal of MPEG-21 standard is to define an open framework for multimedia, which allows the transparent usage of multimedia resources, networks and peripheral devices, assuring good performances in the multimedia
resources controlling process. MPEG-21 offers an open support to deliver and to use the multimedia data. [1] MPEG-21 covers the whole flow of multimedia content processing from the delivery channels point of view. MPEG-21 refers to the solutions to produce, to consume, to present and to sell the multimedia content the multimedia content, having the goal to personalize the multimedia applications. MPEG-21 standardizes the stream of multimedia information and services from the content development until the delivery to the final users. To realize these stages is necessary to identify, to describe, to manage and to protect the multimedia content. The multimedia content transport and delivery could be made throw various networks and between different types of terminals. MPEG-21 is based on two important concepts: the definition of a fundamental unit for distribution and transaction, namely the digital element, and the users that interact with digital elements. The digital element is a digital object with a standard structure that has metadata associated with it. The digital elements contain both multimedia resources, the content, and metadata associated with the resources or with the digital element. In the MPEG-21 standard, a user is any entity that interacts with the MPEG-21 environment or that uses a digital element, like a multimedia data consumer, an organization or another standard that use multimedia resources. A user can consume multimedia content by publishing or by delivering and can have specific rights and responsibilities. The main elements that define the MPEG-21 architecture are: [5] • The digital element is a hierarchical container for heterogeneous resources like video, audio, text and so on, metadata and other digital elements. A digital element is an elementary unit for delivering and transitioning; • Digital element declaration defines a set of terms used to declare the digital elements; • The identification and the declaration of the digital element supposes to identify and to declare the digital element, it’s nature, it’s type or it’s granularity; • Content management and usage offers the interfaces and protocols that allow to create, to manipulate, to search, to access, to store, to deliver and to reuse the multimedia content over the distribution and consume channels; • The intellectual property protection and management allows the multimedia content management and the digital elements protection inside networks and devices; • Terminals and networks offer the tools that allow the transparent access to the content throw networks and terminals, with the service quality control; • Content representation is responsible for media resources representation;
•
Events reporting contain metrics and interfaces that allow users to evaluate the performances of all events reported inside the system.
4. A MPEG-21 architecture for a distributed multimedia system In the Figure 1, we present a generic architecture of a distributed multimedia system based on MPEG-21 standard. The main components of the system are: •
•
•
•
•
Multimedia Database that contains a collection of MPEG-21 digital elements, as following: ♦ Digital element declaration (DED) identify the digital elements and is used to describe the multimedia content, ♦ Digital element adaptation (DEA) is used to adapt the multimedia content to the characteristics of available services; ♦ Descriptors to presents the rights for multimedia content usage. Multimedia resources contain the media streams, like video and audio streams and static data like images, animations, text. The digital elements descriptions contain the references to the multimedia resources. The Web Server stores the users’ interfaces components. The Web server has the following functions: ♦ To upload and to download the multimedia data, ♦ To present the media components and to assure the human-computer interaction, ♦ To control the media streaming. MPEG-21 terminals are devices used by the final users. The MPEG-21 terminal allows using the delivered multimedia content like a digital element. The digital element can be a Web application, a media player or a media recorder. The terminal contains software components for uploading and downloading the multimedia content, for media stream activation or for transmission adjustment in a broadcasting session. A media streamer is a device that transmits the content to a user for immediate handling. We can obtain the data-streaming throw a one-to-one connection, like in the upon-request applications, or data could be broadcasted.
5. The technical solution for implementing the proposed architecture The multimedia content description supposes to make the abstraction of low-level details of the presentation and to indicate the semantic relations between multimedia elements. Using this information it is possible to generate the presentation from semantic
Multimedia database Digital elements declaration
DED DEA
Descriptors
Digital element storage
Multimedia database management system
Server for multimedia data storage and data transmission
Upload/ download
Web Server
Querying
Digital element processing Media player/ recorder
Legend:
Stream control Media stream
MPEG-21Terminal
Figure 1. The architecture of multimedia system based on MPEG-21 relations that indicate how could be suggested each semantic relation, using multimedia constructions. We can have different multimedia presentations automatically generated, function on software or hardware platform used to manage the presentation, function on users’ preferences and function on the type of communication network. Therefore, we can automate the presentation generation using some criteria but keeping unchanged the main message of the presentation. This issue supposes to use some transforming rules and a set of constraints that provide us the final multimedia presentation. To implement these issues we used a XSLT (eXtensible Stylesheet Transformations) document to transform the MPEG-21 document, based on XML (eXtensible Markup Language), in a HTML (HyperText Markup Language) document, which allows to generate and to adapt the final presentation depending on the available resources or depending on the users’ preferences. We implemented an integrated system for various kinds of multimedia data acquisition, visualization,
management and integration based on the proposed architecture. The main goal of the integrated system is the visualization of various kinds of multimedia data acquired in real-time, to manage and to dynamically generate adapted web presentation function on the content and on the structure of multimedia data. In the Figure 2, we present a distributed multimedia system to capture, to manage multimedia data, and to generate multimedia web solution for a car presentation. The processing level allows the web browser to know which are the video streams, the audio sequences and the static images displayed to the final user and the timeline for each multimedia element. Therefore, the processing is not at the server level, and is at the client level. We can minimize the negative influence because of the dynamic feature of the communication networks and we can assure a good quality for the final presentation. We choose to use MPEG-21 standard for digital elements declaration because this standard offers a generic tool for multimedia content description and for standardization in the presentation development.
Web Client Synchronised presentation
Video stream
Windows Client Video Stream Web Cam .NET Remoting Video Stream Media Server
Multimedia Web Presentation Generation
MPEG-21
Integration Multimedia Data base querying by a multimedia with systems DataBase characteristics via Internet Data Video Stream & Base Key frames
Web Page
Data Adnotation
Figure 2. The general architecture of the solution for multimedia data visualization MPEG-21 components used for digital elements description have the following signification: • The Item is the declarative form of the MPEG-21 digital element. An item is composed from a group of sub-items and/or elements with relevant descriptors; • The Descriptor contains information about the items, like the activity; • The Component realizes the link between a resource and all its relevant descriptors and contains information about all parts of one resource; • The Resource is an individual element identifiable by an address and it is use to identify the video sequences and the static images of one presentation. We dynamically generate the MPEG-21 document in XML format, at run time, function on user’s preferences or function on the technical conditions, using content-based querying. Using these elements, we automatically generate the MPEG-21 document, in XML format, to describe the digital elements used inside the multimedia web presentation. We use a C# solution to build and to implement the system. To generate and to format the MPEG-21 document, we used XmlTextWriter class, in the following way: XmlTextWriter xw=new XmlTextWriter( Server.MapPath ("Presentation.xml"), System.Text.Encoding.UTF8); xw.WriteStartDocument(true);
xw.Formatting=System.Xml.Formatting.Indented; We transform MPEG-21 document in an HTML document based on a schema attached to the XML document. We defined a container that stores information about the multimedia elements, the time interval in which the presentation will display each multimedia element. We use this container for synchronization the multimedia elements during the presentation The link between the name of elements and the corresponding multimedia resources is made by digital elements description presented in MPEG-21 format. The MPEG-21 descriptions for images components are presented in the following example:
…………………….. Multimedia data and metadata management solution consist in the usage of an Oracle 10g database. We choose this DBMS to achieve a structured storage of multimedia data. Oracle InterMedia component of the Oracle 10g DBMS has some specific features that enables the DBMS to handle Internet multimedia content, images, audio and video together with relational data and text documents. We used this component to store both multimedia content and MPEG-21 digital items descriptors too. Oracle’s interMedia management system and Oracle 10g database support contain some media object-oriented types. InterMedia system automatically extracts and record media specific characteristics as metadata structures in types’ attributes and provides sime media type specific methods for media content management. Oracle interMedia cannot control the capture devices and has no functions for displaying multimedia data stored in multimedia database. The sources of multimedia data may be various, like a Web site, a local site or a network place. The object-oriented types defined in Oracle interMedia system for multimedia data description are: • ORDAudio for audio data, • ORDImage for static images, • ORDVideo for video sequences, and • ORDDoc for heterogeneous data. The metadata automatically extracted by interMedia system, and used inside the application are: • Information about data storage, like location, data source name and the multimedia data location, i.e. inside the database or as external files, • Multimedia data format, • Coding format, the number of audio channels, sampling rate, compression type and the audio sequence duration – for audio data, • Image format, image size, compression format - for static images, • Frame size, frame resolution, frame rate, sequence duration, compression format, and frame number – for video data.
6. Conclusions The development of an open architecture to build the template based multimedia presentations, automatically
adapted to the characteristics of various kinds of peripheral devices and to different speeds of communication networks is a very complex, useful and strong solution for multimedia web presentations. Because of the wide range of technical solutions used to build web applications, the standardization process in the development of web applications is a very important one. The multimedia databases assure a powerful tool to manage and to build the visualization solution and the integration support for both multimedia data used inside the multimedia application and metadata needed to generate dynamically the adapted multimedia web presentation. The open, template based and standardized solutions used to develop the architecture proposed in this paper assure the possibility to generalize the multimedia web applications development.
7. References [1]
[2]
[3]
[4]
[5]
[6] [7]
[8]
H. Kosch, Distributed Multimedia Database Technologies Supported by MPEG-7 and MPEG-21, Auerbach Publications, 2004. ***, MPEG-7 Overview (version 8), ISO/IEC JTC1/SC29/WG11, http://mpeg.telecomitalialab.com/standards/mpeg7/mpeg-7.htm, Klagenfurt, 2002. A. Silberschatz, H. Korth., S. Sudarshan, A Database System Concepts 3rd Edition, WCB/McGraw-Hill, New York, 1999. ***, ISO/IEC JTC1/SC29/WG11 N 4333, ISO/IEC TR 21000-1:2001(E), Information technology — Multimedia framework (MPEG-21) — Part 1: Vision, Technologies and Strategy, Sydney, Australia, 2001. ***, MPEG-21 Goals and Achievements, 2003, http://www.chiariglione.org/mpeg/tutorials/papers/MMp aper.pdf. Guojun L., Multimedia database management systems, Artech House computing library, 1999. Reveiu, A., Dardala, M., Furtuna, F., Using Contentbased Multimedia Data Retrieval for Multimedia Content Adaptation, Proceedings of the 12th Int. Conf. on Human-Computer Interaction HCII'2007 (Beijing, 22-27 July 2007), Vol. 3, Jacko, J. (Ed.), LNCS 4552: Lecture Notes in Computer Science, Springer-Verlag 2007, ISSN: 0302-9743, pag. 486-492; Halsall, F. Multimedia communications – applications, networks, protocols and standards, Pearson Education Limited, 2001;
