Second International Workshop on Semantic Media Adaptation and Personalization
Semantic Content Filtering using Self-Organizing Neural Networks Damon Daylamani Zad Brunel University, Uxbridge, London, UK
[email protected] multimedia search results is not satisfactory as the details of contents and user’s information requirements are ignored [4]. User behavior should inform the ranking of results produced from searching and filtering the modelled multimedia content. The objective is to provide users with the most relevant results that satisfy their requirements [5]. To effectively utilize the user behavior as the basis of any ranking system the MPEG-7 model should be designed in a way that the user preferences match the content model and therefore mapping the semantic data in the content to the users’ preferences. This can be achieved using the hanging basket model [6] in order to close the gap between user model and the content model [6]. In this paper we will present a new approach to filtering ranked content which uses SONNS on the hanging basket model. The rest of this paper is sectioned as follows: Firstly, we review related work in the area of MPEG-7 content and user modelling for content filtering. Then, we present our new approach to filtering content using SONNs on the hanging basket model. Finally, the paper is concluded in the fourth section.
Abstract COSMOS-7 is an application that can create and filter MPEG-7 semantic content models with regards to objects and events, both spatially and temporally. The results are presented as numerous video segments that are all relevant to the user’s consumption criteria, yet these results are not ranked according to the user’s preferences. Using self organizing networks (SONNs) we rank the segments to the user’s preferences by applying the knowledge gained from similar users’ experience and use content similarity for new segments to derive a relative ranking. To bridge the gap between the user preferences and the content model, an MPEG7 model is proposed that uses the hanging basket model to better relate the users’ preferences and usage history to the content.
1. Introduction Filtering multimedia content is a complex task as the medium is transient both temporally and spatially. This means that the content has different semantic meaning spatially and temporally. To filter multimedia content, a content model that describes the semantic relationship of the objects and events both temporally and spatially is required. When such a model is developed, a filter is required that will sort through the model based on its users’ information requirements. COSMOS-7 [1] is an MPEG-7 compliant application that reduces the complexity of creating such a content model and filter. It exclusively uses part 5 of the MPEG-7 [2] standard (Multimedia Description Schemes) that semantically describes objects and events and their relationships both temporally and spatially. Unlike other multimedia content modeling systems [3] it does not use low level (syntactic) features, only high level (semantic features) that are meaningful to the user. Using the COSMOS-7 filtering manager a filter is created that can exploit the rich detail captured in the content model by allowing a user to filter out undesirable content. The experiments show there are many relevant results in the returned list of the COSMOS-7 filter manager that fitted the filter criteria. Yet these results are not ranked by the relevancy to the context of user’s information requirements. Therefore, the relevance ranking of the
0-7695-3040-0/07 $25.00 © 2007 IEEE DOI 10.1109/SMAP.2007.22
2. Related research review: The Hanging Basket Model Many researchers have explored the application of MPEG-7 user modeling tools and have either tries to make better use of the available design and confines or have endeavored to expand these confines and come up with new designs for the user model. For TV and broadcasting, approaches have ranged widely. In the IndexTV system [8], MPEG-7-based personalization is applied to TV programs. When the user logs in to the system for the first time, he fills out a TV preferences form, which allows him to specify a small number of preferences: genre, personages and channels of interest. Considering personalized Web access, Karpouzis et al. [9] also use categories, such as sports, arrivals-departures, politics, and education, to help predict video shots that will interest a given user. Martinez et al. [13] propose a design where they use Variation sets to group the Creation and Usage information together and use a Profile Manager to
253
create and update predefined profiles stored in the Profiles Database. Tsinaraki et al. [14] present a user model in which a user preference description is comprised of an optional user identifier, filtering and search preferences and browsing preferences. The user preference descriptions may be allowed to be automatically updated, according to the allow automatic updates Boolean attribute. The MPEG-7 user interaction tools consist of user preferences, embodied in the UserPreferences DS (Description Scheme), and the history of the user, embodied in the UsageHistory DS. The UserPreferences DS enables users to specify their likes and dislikes for types of content, ways of browsing content and ways of recording content [2, 7]. The UsageHistory DS groups together a set of UserActionHistory DSs, each with its own observation period. It consists of action type-specific lists that include identifiers for each application associated with each action. The time of user actions can be indicated as well as the time extent of the multimedia content that was consumed. These DSs although very useful they do not make very descriptive data about the users’ preferences in regard to the content. There is no effective and fully descriptive mapping from the objects and events in the semantic content to the users’ preferences. The outcome is a very incomplete mapping of user models to content models which is termed the MPEG-7 content-user gap [6]. The hanging basket model [6] tries to make the user model and the content model isomorphs and therefore they may both be realized in MPEG-7 using MPEG-7 content description tools. Using SemanticDescriptionType and ContentEntityType to describe the semantic content of the videos, enables the model to present the events and objects of the content in the Semntics DS. The EventDS and Object DS describe the events and objects respectively. Within these descriptors, the MediaOccurence DS is used to refer to the appropriate video segment where TemporalMaskTypes define appropriate masks on the segment. Events are related to objects through the use of the Relation DS and the agent relation of the SemanticRelation CS. Events are related to objects when the objects participate in that event. The SemanticRelation CS is also used to define the spatial relationships between the Objects and their states. The Relation DS, which was used extensively throughout the hanging basket content model, employs a strength attribute which is of type zeroToOne. Consequently, all preferences must be specified as a float number in the range [0, 1]. Hence, within the [0, 1] range, 0.5 is taken as the neutral (default) value and values below this specify negative preferences
(dislikes), while values above it specify positive preferences (likes). In the user model a single top-level type is used, the SemanticDescription-Type, to gather together an entire user model for a single user. The preferences within the model are then grouped together using several Semantic DSs. The user is identified in the top level Semantic DS. The events are realized within a SemanticState DS within the second half of a Semantic DS that also groups together preferred spatial relationships from the objects. The SemanticState DS is used to express the user’s preference for an event as an AttributeValuePair. The state relation from the SemanticRelation CS is used to tie the preference with the relevant Event DS from the hanging basket content model. Preferred temporal relationships are specified using the Graph DS, in the same way as in the MPEG7 content model, with the addition of the strength attribute within each Relation element. The objects are realized within three distinct areas of the MPEG-7 user model. First, the first half of the Semantic DS discussed above groups together preferred spatial relationships between objects. Second, the first half of the next Semantic DS is used to group preferred objects within an event together. The Object DS encapsulates dynamic preferences for a single object within an event via one or more Relation elements of type agentOf appended with the strength attribute. Third, the preferred object hierarchy as is grouped together within a graph structure by the Semantic DS, where all object relationships are expressed using the SemanticRelation CS. This mapping of the user model to the content model enables users to store all kinds of preferences and therefore get better and more effective results from filtering as the systems now can filter the results and rank them based on anything from genre to the objects and their relations. As these rankings are stored in the Relation DS, the filtering system will have more information to use when ranking the results.
3. Formulating new research: Filtered content ranking using SONNs The proposed model will be based on the hanging basket model. In this model, users are grouped together and the user behavior, preferences and history are stored in groups. A Semantic DS is created for each user group, forming a template of user preferences and history to use for all the users of the same group. This ensures the new users to have preferences and history and therefore they can use the filtering system better without needing to start anew with no history. Using
254
this model, every action a user takes, and all their ratings will be stored in MPEG-7. This might cause a problem when users have conflicting interests. To solve this, the model is designed so that in case of a conflicting interest or ranking, all sides will be stored and tagged by subgroups. This means the there will be subgroups under each group of users yet still these subgroups have more similar interests than conflicts to unite them under a super group. Proposed here is a COSMOS-7 ranking module that is employed after the filtered results have been sorted in an MPEG-7 model that is based on the hanging basket model. For doing so, it uses two SONNs to order the segments in terms of the user’s requirements. In related works, neural networks have been applied to collaborative filtering to cluster users into groups based on similar tastes [10]. Their neural network uses the user demographic and preference attributes for similarity clustering. The basic idea of the ranking module based on the collaborative filtering and recommendation principles is that a user should be recommended items that are preferable by other people with similar tastes and preferences. Thus, by investigating the items previously rated by other similar users, a utility value can be given to these items. The rating value of an item Ik and user Uj is computed as an aggregate of the ratings of some other users Un belonging to the same user group G for the same item Ik. Group G is the set of users as derived by the first neural network, which clusters the users based on their similarities. The ranking of the users and user groups can be easily retrieved from the MPEG-7 model based on objects or events as the hanging basket model has provided the user preferences to be stored and ranked in the MPEG7 according to the semantic content. Naturally, SONNs are able to determine as many optimal classes as their internal neurons. Neural networks are used for the user clustering since many studies have showed that SONNs provide different result compared to statistical clustering. In [11] the authors demonstrate that SONNs were able to recognize clusters in a dataset where other statistical algorithms failed to produce meaningful clusters. Furthermore, neural networks clustering results indicate clustering stability [12], which is an important factor for high quality clusters. While the above framework is ideal for formulating the ranking of COSMOS-7 filter results, it is also being borne with some of collaborative filtering drawbacks. New items, not yet rated by anyone, cannot be evaluated. Therefore until the new item is rated by a substantial number of users, the system would not be able to use its rating value appropriately. COSMOS-7 content ranking algorithm is responsible for evaluating new items; the basic idea is that a user will be
recommended items similar to the ones the user preferred in the past. The algorithm uses the same principles as the content recommendation and filtering principles. A SONN clusters the video segments into a number of groups based on their similarities. The content ranking algorithm is utilized when the collaborative ranking cannot be used though. The item Ik and user Uj is computed as an aggregate of the ratings of some other items In belonging to the same item group G for the same user Ui. The second neural network uses the average ratings of items for a user. This neural network clusters the items into classes according to their similarities. The base vector includes attributes from COSMOS-7 model such as objects, events etc. Therefore each item (visual segment) is a neural network input vector with attribute values taken from its content model.
Figure 1. COSMOS-7 ranking module The advantage of the system is that the SONN adapts the user’s peer group as their preferences change over time. As the user interacts with the content the usage history log is updated. The usage history is reduced dimensionally into attribute pairs that are used to update the user profile. The changed user profile is then used to recalculate the user’s peer group. This allows the user’s changing requirements to be tracked automatically and reflected instantly in the ranking of
255
the content. This process is iterative and makes peer matching more accurate over time. A key feature is as the MPEG-7 is using hanging basket model for the user model, there are valid values to rank each and every Object, Event, relation and attribute of the videos as the strength value stored in the Relation DS. As stated before the user model groups the users together and stores the data for the group. The ranking will be based on the groups’ preferences and behavior and so there will be some filtering material even for the new users. The proposed system builds a new path for multimedia databases. For example, online radios such as Yahoo! LAUNCH and video communities such as YouTube could use the system for searching and recommending multimedia content to users and user groups.
on-demand systems", ACM SIGOPS Operating Systems Review 40(4), Proceedings of the 2006 EuroSys conference, Leuven, Belgium, April 2006, pp. 333–344 [6] Agius, H.W. and Angelides, M.C., “Closing the ContentUser Gap in MPEG-7: The Hanging Basket Model”, ACM/Springer Multimedia Systems, 2007, In Press [7] ISO/IEC: Information Technology—Multimedia ContentDescription Interface—Part 5: Multimedia Description Schemes: Amendment 2: Multimedia Description Schemes User Preference Extensions. International Standard 15938-5/Amd.2, Geneva, 2005 [8] Rovira, M., González, J., López, A., Mas, J., Puig, A., Fabregat, J., Fernàndez, G., “IndexTV: a MPEG-7 based personalized recommendation system for digital TV.” 2004 IEEE International Conference on Multimedia and Expo, vol. 2, Taipei, 27–30 June 2004, pp. 823–826. [9] Karpouzis, K., Moschovitis, G., Ntalianis, K., Ioannou, S., Kollias, S. “Web access to large audiovisual assets based on user preferences” Multimedia Tools and Applications 22(3), 2004, pp. 215–234
4. Conclusion In this paper, a ranking module based on neural networks is proposed in order to rank the results of multimedia content-based filtering systems. The ranking module works with COSMOS-7 [1] and the hanging basket user model [6].
[10] Lee, M., Choi, P. and Woo, Y.-E. “A Hybrid Recommender System Combining Collaborative Filtering with Neural Network.” Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Malaga, Spain, 2002, pp 531-534.
5. Acknowledgements
[11] Ultsch A. “Self Organizing Neural Networks perform different from statistical k-means clustering.” Proceedings of GfKl, Basel, Switzerland, 1995, pp. 1-13
I would like to thank my supervisor, Professor Marios C. Angelides, for his encouragement, advice and support during the preparation of this paper.
[12] Santosh K. Rangarajan, Vir V. Phoha, Kiran S. Balagani, Rastko R.Selmic, S.S. Iyengar, “Adaptive Neural Network Clustering of Web Users”, Computer 37, 2004, pp. 34-40
6. References [1] Angelides, M.C. and Agius, H.W. “An MPEG-7 Scheme for Semantic Content Modeling and Filtering of Digital Video”, ACM/Springer Multimedia Systems 11, 2006, pp. 320-339
[13] J. M. Martínez, C. González, O. Fernández, C. G. J. de Ramón, "Towards universal access to content using MPEG7", Proceedings of the 10th ACM International Conference on Multimedia, Juan-les-Pins, France, December 2002, pp. 199-202
[2] ISO/IEC: Information Technology −Multimedia Content Description Interface – Part 5: Multimedia Description Schemes. Geneva, Switzerland, International Organisation for Standardisation, 2002
[14] Tsinaraki, C. Christodoulakis, S. “A multimedia user preference model that supports semantics and its application to MPEG 7/21”, Proceeding of 12th International MultiMedia Modelling Conference, 2006, pp. 35-43
[3] Koprinska, I. and Carrato, S., “Temporal video segmentation: A survey”. Signal Processing: Image Communication 16, 2001, pp. 477-500 [4] Lai, W., Hua, X.-S. and Ma, W.-Y., "Towards contentbased relevance ranking for video search", Proceedings of the 14th Annual ACM International Conference on Multimedia, Santa Barbara, CA, USA, October 2006, pp. 627-630 [5] Hongliang Yu, Dongdong Zheng, Ben Y. Zhao, Weimin Zheng, "Understanding user behavior in large-scale video-
256
