IndexTV: A MPEG-7 Based Personalized Recommendation System for Digital TV Marc Rovira, Jordi González, Alejandro López, Jordi Mas, Albert Puig, Jordi Fabregat and Gabriel Fernàndez Digital Television Center, La Salle School of Engineering, Ramon Llull University, Barcelona, Spain
[email protected] Abstract Digital television is a reality providing new facilities for multimedia content delivery. Due to the large amount of available content from multiple broadcasters, the use of personalized recommendation techniques can help the user in the process of consuming A/V content. The viewer, who is actually a passive subject, will be changed into an active one, who will be able to organize the TV Program Guide of its interest whenever and however he wants, achieving a real “à la carte TV”. Within the scope of this paper we present an end-to-end system able to offer A/V contents according to the TV viewer preferences. The system adapts its suggestions to the changes in viewer’s behaviour across the time.
1. Introduction The introduction of digital Television is already a reality. With the digitalization of television broadcasting through different environments, i.e. satellite, terrestrial and cable, new challenges and innovation opportunities appear. We are now in a phase where the basic digital TV technology is already well-established and users can enjoy a great quantity of contents and interactive services. Digital TV receivers have increased their capacity for receiving more contents, with the obvious requirement to process and organize the received information. In parallel, TV receivers have also increased their storage and computational capacity thanks to the reduction costs of hard disks, memories, etc. Digital TV systems must provide new services according to user preferences and also to TV consumption habits. In this way it is required to develop learning tools that automatically select, filter, memorize and look for the A/V contents allowing the user to enjoy an authentic "television à la carte" [1].
This paper describes the IndexTV project. The aim of this project is to develop the content description tools and the necessary mechanisms to turn the digital TV receivers into systems that allow content personalization and a more efficient interaction between the users and the contents to visualize. The system implements an end-to-end digital TV personalization system based on the application of MPEG-7 standard for content description, synchronized MPEG-7 metadata broadcasting and the application of artificial intelligence mechanisms to support a faithful user profile according to his television preferences. The system is compliant to well established standards as MPEG-2 and DVB. The paper is organized as follows. Section 2 will describe the overall IndexTV framework, including all the developed tools as the MPEG-7 cataloguer, metadata manager, play-out, and receiver. Section 3 will explain the functional process. Finally, section 4 will draw some conclusions.
2. IndexTV framework 2.1 Overall description The IndexTV system framework consists of a DVB Play-out System called FlowServer (FS) previously developed and detailed in [2] and two new elements, the Cataloguing Tool and the Metadata Manager. The Cataloguing Tool generates MPEG-7 descriptions of A/V content in a semi-automated manner. MPEG-7 descriptions are the input of the Metadata Manager. The Metadata Manager will encapsulate, signal and broadcast the metadata synchronously with A/V content by means of the FS System. The metadata generated will be delivered in form of dynamic metadata service explained below in this paper compliant with the recent Amendment of MPEG-2 Part 1 [3] for carriage of metadata. Finally, the multiplexed
MPEG-2 TS is modulated and sent to the receivers where, by means of a TV Program Assistant (TVPA), the system will recommend A/V content to the end user. Figure 1 illustrates the block diagram of the IndexTV framework.
1.
Signalling generation. The system automatically generates the signalling needed for the metadata service.
2.
Metadata Service Generation. The system divides the descriptions of the A/V contents into the DecoderInit and smaller pieces named Access Units (AUs) as defined and MPEG-7 [6]. These two parts are transported using different methods, DSM-CC OC for the DecoderInit part and Metadata Sections for the AUs.
3.
NPT Generation. Time references in the metadata are based on Normal Play Time (NPT) time base. The metadata manager generates the NPT allowing the receiver to recover the time references inserted in the metadata.
2.2 Cataloguing Tool The developed Cataloguing Tool allows the user to describe a video content with general, audiovisual, temporal and shot descriptions. The result of these descriptions is an MPEG-7 file containing the description of the video. The application provides a friendly and easy environment to the user to describe the content. Different techniques have been developed to automate the edition process such as automated shot detection. Colour histogram and dominant colour techniques have also been developed to generate advanced descriptions of the image allowing matching in search applications.
2.3 Metadata Manager The Metadata Manager is responsible for the encapsulation, carriage and synchronisation of the Metadata Services in the context of the IndexTV project. These Metadata Services are generated according to the idea of Dynamic Metadata Services explained in [4], where two transport methods are combined for the transmission of the MPEG-7 metadata: DSM-CC (Digital Storage MediumCommand & Control) Object Carousel [4] for static program descriptions and Metadata Sections [2] for dynamic shot descriptions. In order to correctly encapsulate, transmit and synchronize the MPEG-7 metadata, the Metadata Manager performs the following tasks:
2.4 FlowServer FlowServer is a Play-out System consisting of several components interconnected via TCP/UDP [2]. It becomes a complete system for playing-out and managing interactive data contents for digital television. The key point of the system relies on its architecture that allows full scalability and distributed managing as it can be seen in Figure 1. This architecture is especially useful in the broadcasting of MHP interactive applications or any kind of data provided by multiple agents. In the IndexTV project, the FlowServer is used for encapsulating the MPEG-7 metadata in DSM-CC OC and Metadata Sections and multiplexing them in real time synchronously with A/V content providing a compliant MPEG-2 Transport Stream.
TCP
TV Screen
FS Manager A/V Coder
TCP Server Manager
DSM-CC MHP Coder
TCP Metadata Manager
MPEG-7 Files
TCP
DSM-CC Metadata Coder
Metadata Section Coder UDP
NPT Coder
SI/PSI Coder
UDP Set-Top Box PVR TV Program Assistant
UDP
UDP
UDP
UDP UDP
Cataloguing Tool FlowServer
Figure 1. IndexTV framework
FS MUX
MPEG-2 TS DVB-ASI
DVB Stream Modulator DVB-S/-T/-C
2.6 The TV Program Assistant (TVPA)
At the receiving side, a Set-Top Box (STB) with the typical functionalities of a commercial PVR (Personal Video Recorder) has been implemented. The STB is endowed with some intelligent mechanisms, referred as the TV Program Assistant (TVPA). TVPA is responsible for content recommendation and user preferences update.
The TVPA module consists of generating the content suggestions of possible interest for the viewer from the result of matching the TV program description and user preferences. This recommendation mechanism is based on a Knowledge Based System (KBS) [9]. All KBSs are constructed on the basis of three principal pillars: Knowledge Database (KDB), Inference Engine (IE), and Input/Output Interface (I/OI) as illustrated in Figure 3.
To achieve the functionalities of a commercial PVR (e.g. to be able to switch to any channel, do instant recording, time shifting or simply playback the recorded content, etc.) we have used the open source project called VDR (Video Disk Recorder) [7] under Linux OS. VDR uses the drivers for commercial digital TV tuner boards from the open source project called LinuxTV [8]. In fact, the extra functionalities of IndexTV project have been developed as two VDR plug-ins. Figure 2 illustrates the block diagram of the module.
User Responses Event Filtering DVB stream
VDR
SVDRP
Video Disk Recorder
Channel Monitoring
Suggestion Messages
Presentation Plug-in Internal Events
Metadata Plug-in File Saving
User actions
Knowledge Database
Knowledge Database Updater
Inference Engine
I/O Interface
Suggestions User responses
Figure 3. The TV Program Assistant architecture
TV Program Assistant
A/V content
Program Description
2.5 The Set-Top Box with PVR capabilities
MPEG-7 Files
Synchronous Notification
Usage of MPEG-7
Figure 2. The Set-Top Box architecture
As it can be seen in Figure 2, the system consists of handling the user actions at the same time as it is decoding and storing the MPEG-7 descriptions of all the broadcasted A/V services by means of the Metadata plug-in. The MPEG-7 descriptions are synchronously delivered to the TVPA. The TVPA manages the recommendation messages that will be shown on the TV screen and processes the user responses by means of the Presentation plug-in. Communication between modules is done by using the SVDRP (Simple VDR Protocol). Thanks to this feedback, the TVPA will be updating its Knowledge Database (KDB) that will be reflecting the dynamic user behaviour.
The KDB is the user preferences database. It is initialized with the explicit declaration of user preferences and it is updated as the system learns from the user behaviour. The IE searches coincidences between the program descriptions of the content that the viewer is not watching at that moment and his preferences stored in the KDB obtaining a preference value. If this preference value is higher or equal to the preference value of the current channel being watched by the user, or a given threshold, the system performs the suggestion of the new channel, either for viewing or recording through the I/OI. The I/OI supports the communication with the Presentation module to show the suggestions received from the IE and sends the user responses (from the recommendation) to the Knowledge Database Updater (KDBU). Finally, the KDBU updates the KDB according to the user responses. The KDBU mechanism is described in Section 3.
3. Functional Process Description To better understand the system functionality, we will divide the process in four steps: 1. Creation of user profiles and system login: before the user logs in the system for the first time, it is necessary to fill a TV preference form. The user may add his favourite personages with an associated role in addition to the genres and channels of his interest. The user is also asked to introduce a user id with an associated PIN (Personal Identifier Number). When
the user starts the system he is requested to be authenticated by means of an OSD (On Screen Display) login screen. At this point the system will retrieve the user profile from the database and will start learning the user actions. 2. MPEG-7 Metadata reception: in a background process, the system starts scanning all channels decoding the incoming broadcasted metadata. For each metadata description, the TVPA will be notified. 3. System suggestions: in the next step the Presentation plug-in will remain expecting to receive recommendation messages from the TVPA. Once a message is received, the suggestion message is shown to the user (see Figure 4) as an OSD pop-up window and immediately waits for few seconds to get the user response. At this point, the possible user actions are: • • •
Switch to the suggested channel. Instant recording of the suggested content Refuse the suggestion.
The TVPA updates the channel preference if the user has accepted the suggestion and it is in the KDB, or the user has refused the suggestion but it is in the KDB. The TVPA creates a new preference if the user has accepted the suggestion, but it is not in the KDB. Finally, nothing is done when a suggestion is refused and it is not in the KDB.
4. Conclusions In this paper we have presented an end-to-end system that provides personalized A/V content in digital TV systems. The system is compliant to current standards, i.e. MPEG-7/-2 and DVB. It allows immediate deploying of personalized TV services in current digital TV networks. The system broadcasts synchronously MPEG-7 metadata previously extracted during the cataloguing process. On the receiving side, the STB has been also implemented with PVR, user preferences learning and personalized recommendation functionalities. In the future, we will improve the recommendation engine as well as provide further personalized services compliant to the emerging TV Anytime standard.
5. References
Figure 4. A recommendation message
In any case, the action chosen by the user becomes effective and the TVPA is notified. If no answer is given, the system starts recording automatically the suggested content and the user is notified by means of a new message later on. As a response to this message the user can decide to continue recording the recommended content or to stop recording and eliminate what has been recorded up to now. 4. KDB Update: Depending on the user response to the suggestion, the KDB is updated. The update criterion mechanism is illustrated in Figure 5. User response Suggestion Suggestion accepted? accepted?
yes Is Is the the suggestion suggestion in thethe KDB? inside KDB? yes
Is Is the the suggestion suggestion in the KDB? inside the KDB? No
Update Update the the preference preference
No
Make Make aa new new preference preference
yes Update Update the the preference preference
Figure 5. KDBU dynamic mechanism
No Do Do nothing nothing
[1] M.Ceccarelli, G. Mekenkamp, E.Persoon: "AVIR Home Multimedia Systems: On Personal Video Libraries", in Proc. of IEEE Multimedia (ICMCS’99), Florence, Italy, June 1999 [2] D. Gonzàlez, X. Vila, P. Gambín, A. López, M. J. Tomé, C. Vicente, and G. Fernàndez, “Management and broadcasting of MHP applications from distributed providers”. Proceedings of 2001 International Broadcasting Convention, Amsterdam, September 2001. pp. 272-278. [3] Generic coding of moving pictures and audio: Systems. Amendment 1: Carriage of metadata over ITU-T Rec H.222.0 | ISO/IEC 13818-1 streams. [4] A. López, D. Gonzàlez, J. Fabregat, A. Puig, J. Mas, M. Noé, E. Villalón, F. Enrich, V. Domingo and G. Fernàndez. “Synchronized MPEG-7 Metadata Broadcasting over DVB networks in an MHP Application Framework”. Proceedings of 2003 International Broadcasting Convention, Amsterdam, September 2003. pp. 256-263. [5] ISO/IEC 13818-6: Generic coding of moving pictures and associated audio information. Part 6: Extensions for DSM-CC. [6] ISO/IEC 15938: Information Technology – Multimedia Content Description Interface. [7] Klaus Schmidinger’s VDR website. www.cadsoft.de/vdr [8] LinuxTV website. www.linuxtv.org [9] Gonzalez, A., and Dankel, D.D., The Engineering of Knowledge-based Systems Theory and Practice, Prentice Hall, Englewood Cliffs, NJ, 1997.
