A framework for human-centered provisioning of ambient ... - CiteSeerX

Multimed Tools Appl DOI 10.1007/s11042-009-0285-9

A framework for human-centered provisioning of ambient media services M. Anwar Hossain · Jorge Parra · Pradeep K. Atrey · Abdulmotaleb El Saddik

© Springer Science + Business Media, LLC 2009

Abstract The provisioning of ambient media in the user’s environment requires that a system handles the different aspects related to the media selection process. For example, ambient media is delivered to the user depending on their context and hence, the system needs to dynamically determine the context and provide media that are relevant therein. To set the premise of ambient media, a system may also need to customize the physical environment, for example, by dimming the lighting level or by lowering the volume. Besides, users’ need for media services changes over time and space that requires mechanisms to continually update their preferences based on their mobility in the environment. In this paper, we propose an ambient media service provisioning framework that incorporates the above requirements while keeping the user at the center of the media selection loop. To demonstrate the usefulness of this framework, we show experimental results by considering real-life scenario in a smart home environment.

M. A. Hossain (B) · A. El Saddik Multimedia Communications Research Laboratory (MCRLab), School of Information Technology and Engineering (SITE), University of Ottawa, Ottawa, Ontario, Canada e-mail: [email protected] J. Parra Software Technologies, Ikerlan-IK4 Technological Research Centre, Arrasate-Mondragón, Spain e-mail: [email protected] P. K. Atrey Department of Applied Computer Science, The University of Winnipeg, Winnipeg, Manitoba, Canada e-mail: P. K. [email protected] A. El Saddik e-mail: [email protected]

Multimed Tools Appl

Keywords Ambient media · Context awareness · Personalization · Smart environment · Service provision · Smart phone

1 Introduction The vision of ambient media [5, 12] is rapidly gaining popularity. Supporting users with the media services based on their context and need has been an ever expected demand. The interconnection of heterogeneous devices, sensors, media and information services in a seamless fashion has provided the ground to meet this demand. However, selecting or recommending media that are of interest to the users in their ambient environment is a challenging task. This is due to the fact that 1) the number of available media services increases over time, making it difficult for people to manage and access them properly and 2) people’s preference changes over time, which puts burden on a system to adapt the media selection based on their changing need. Therefore, we need mechanisms to estimate the utility of the services in the identified context including the mobility of the users. This can be done by estimating the gain of the available media services in different contexts [6], which can help users to reduce their cognitive load in selecting media from a potentially huge list of services. Ambient media is also viewed as spreading into the natural environment of humans [5, 8, 9, 12]. We need mechanisms to interconnect heterogeneous devices in the environment and control them for customizing user’s surroundings when selecting media. For example, when a user requests for a music, a responsive system may select the best music of his choice (preference) and play it via the nearest audio device, however lowering the volume (i.e. device control) when it identifies somebody is sleeping in a close by room (i.e. context). The context-awareness, the management of users changing preferences over time and space, the dynamic estimation of gain, and the control of related devices and artifacts in the environment thus constitute the requirements of a framework for providing ambient media. It is however challenging to develop such a framework that can incorporate the above requirements in a coherent manner. In this paper, we propose FAMe as a Framework for Ambient Media provision to address this challenge. In addition to supporting the above requirements, the proposed framework promotes the view of human-centered computing approach by keeping the user at the center of the media selection process. Research on human-centered computing (HCC) [10, 11] poses new agenda in this direction such as considering the subjectivity associated with individual’s choices, incorporating social contexts and personal human factors, such as mood and attention. In FAMe, we support some of these views such as to manage user’s contextual preferences and to provide them simple interface to control and experience ambient media whenever they need. There are existing works that provide mechanisms to personalize media and information for the user. Research on user modeling [3, 27], context-aware information systems [13, 15, 23] and recommendation systems [1, 14, 25] address the problem of personalization, contextualization and recommendation issues. For example, the user modeling research usually analyzes a user’s knowledge, goals and interaction to model a user’s preference for personalization and adaptation of information and services. Likewise, research on context-awareness provides mechanisms to deliver information based on user’s context, while the works in recommendation system

Multimed Tools Appl

suggest items and/services to the user based on the feedback and ratings provided by other users. We use several concepts laid out in these works and develop an integrated framework for ambient media service provision to enhance user’s experience. Our contribution in this paper is the following. First, we provide the design and development of an ambient media provisioning framework. The proposed framework incorporates user’s context, changing media preference, control of environment artifacts and user’s mobility using a service oriented design. Second, we provide mechanism to dynamically update user’s media preference based on interaction data and show how the changing preference along with the reputation of a service is later used to compute gain of the media service. Third, we show how to support user’s mobility in terms of media provision by preserving user’s changing preference on the smart phone and facilitating access of ambient media via smart phone interface. Note, earlier in [6] we described the dynamic preference update and gain estimation with preliminary results. Later, in [7], we used the dynamically computed gain value and considered service cost to show a trade-off between the gain and cost value while selecting services for user. The approach presented in [7] is applicable in cases when the cost of the services is available. However, there are cases when the cost is not available or cannot be easily realized, especially when more and more media services are becoming available for free. Therefore, we ignore cost issues in this paper and focus more on the design and development of the proposed framework in order to provide media based on the highest gain. In the remainder of this paper, we briefly comment on some related work (Section 2), followed by problem description in Section 3. The proposed framework is described in Section 4. We then present the implementation details and results in Section 5. Finally, Section 6 concludes the paper and also mentions our future work. 2 Related work Our work in this paper is related to the works in the domain of user modeling, context-aware information systems, and recommendation systems with respect to providing personalized and context-aware media service to the user. Therefore, we briefly comment on some works in these domains. Research on user modeling (UM) focuses on analyzing a user’s knowledge, goals and interactions to model the user’s preferences for personalizing information and services for the target user. A user modeling process uses the observable information about a user, such as his interaction history, and generates unobservable information about him, such as preference scores [27]. UM is generally used in many personalization applications, which are mostly web information centric. Although, the proposed work utilizes some UM principles, such as the inclusion of interaction history information, it aims to cover the broader aspects of service provisioning in ambient environment. The proposed work is also motivated by the works in context-aware information access and presentation such as [2, 15, 23, 24, 26]. The work in [26] provides a mechanism to show information of various types in the large display based on the user’s preferences and display templates. However, it is not clear if and how the user’s preference attributes change over time and impact the subsequent information delivery. Authors in [24] present a relevance-feedback based preference learning

Multimed Tools Appl

mechanism, where the high-capability master device learns the user’s preference based on the feedback from the low-capability slave devices by observing the user’s behavior. The learned preference is used for multimedia personalization in a pervasive environment. In [23], the authors proposed a content filtering and presentation approach based on co-occurrence analysis from historical interaction data. Based on the co-occurrence, they used a content utility score for different content types and utilized this score as a basis for selecting subsequent content for the user. This work however, does not provide any detail as to how the user’s preference for an individual media item would change over time and how all the different aspects of the overall system would work. The authors in [15] proposes a framework for combining the user’s profile and situational context in order to provide services based on the capability of a mobile device. Our work, in contrast to [15], focuses on recommending ambient media in smart environments which also includes the controlling of smart artifacts to enhance media experience. In a recent work [2], the authors present a semantic-based ambient media selection framework for intelligent home media entertainment. Like ours, this work uses the different dimensions such as the user’s profile, interaction history, and context to personalize media, however, it does not provide the details of how the preference scores are evolved and also does not address the mobility issue of the user. Finally, we see a resemblance of our work to the rating estimation problem in typical recommendation systems. For example, in [25] the authors presented the CoMeR platform that supports media recommendations for smart phones using a hybrid mechanism, which consists of a content-based scheme for determining similarity of a media item to the user’s preference context, a Naive Bayes classifier for determining the relevance of an item to the situation context, and a rulebased scheme for checking the presentation suitability of a media against devicecapability context. Unlike [25], which recommends media to render in smart phone, we recommend and provide provide media in the user’s ambient environment and use smart phone as an interface to access ambient media. The work in [1] considers multiple dimensions to recommend items to users based on the time, the user and the media item. Unlike the recommendation system referenced here, which usually recommends a particular item, our focus in this paper is different in that we aim to recommend media by considering surrounding context of the user and describe the detail development of the proposed framework.

3 Problem description In the context of ambient intelligence [4], user’s environment is technologically augmented with myriad sensors and networked devices in order to provide support to the users. In such an environment, providing ambient media to the users based on their context is a challenging issue. This requires addressing several key issues, which are context determination, user’s preference management, control of smart artifacts, and managing users mobility. Besides, the development of a framework that can incorporate these aspects in a coherent manner is not a trivial issue. In the following, we highlight the difficulties that appear while addressing the above issues. –

User’s context plays an important role while delivering ambient media in the environment. Several sensory data need to be fused to determine the context.

Multimed Tools Appl

–

–

–

–

However, context determination is not always straight forward due to the imprecision of sensing and media processing tasks. The dynamic derivation of context and the granularity of context parameters also brings challenge in the overall task. We describe the context model in Section 4.3. User’s media preference changes over time, which need to be managed in order to recommend users with relevant and appropriate media at a particular context. The dynamic update of preference need to consider user’s context and interaction history to reflect user’s contextual preference, which is later used to estimate the gain of available media services. However, how to incorporate interaction history to update the changing preference and how to estimate gain remains to be an important issue. This is elaborated in Section 4.4. Providing ambient media may also require to customize the environment in order to provide better quality of experience. Environment customization relates to the control of several household artifacts, such as changing the lighting intensity or lowering the volume of an audio device. The heterogeneity of devices and artifacts as well as their interconnections need to be considered in this respect. This is explained in Section 4.5. It is usual that the mobility of the users change. However, they should be provided with ambient media wherever they are as per their need. Therefore, a mechanism is needed not only to obtain their updated preferences but also to make it available in mobile locations to personalize their media selection. We address this issue in Section 4.6. The development of the framework that incorporates the above issues also poses challenge. This is mostly related to practical issues such as the use of particular technology, the communication among several components, the availability and access of the media, the interconnection of the devices to name a few. We address this issue in Sections 4.7 and 5.

Our objective in this paper is to design and develop an ambient media provisioning framework that addresses the above issues in a coherent manner. We also demonstrate how the proposed framework is implemented.

4 Proposed framework 4.1 Overview In this section, we provide an overview of the ambient media provisioning loop to clarify the design of the proposed framework, FAMe. Figure 1 shows this loop [6]. As per this figure, it is clear that ambient media is centered around the users. The users interact in the environment while context is identified by the context identification process using several context sensors. Based on the interaction history in a particular context, user’s ambient media preference gets updated that is later used along with the reputation of media services to estimate the gain of the available media services. The service selection process afterwords uses this gain to select and provide media to the user. The realization of the whole media provisioning process requires the development of several components, which should work together in a logical sequence. This is explained in the next section.

Multimed Tools Appl

Fig. 1 Schematic view of the ambient media provisioning loop

4.2 High-level architecture We now draw the high-level architecture of the proposed framework in Fig. 2, which highlights its core components. In this architecture we show how it incorporates the several aspects earlier mentioned in Section 3. For example, a) the users context is handled by the Context Identifier module, b) ambient media preference update and gain estimation is instrumented by the Ambient Media Selector module along with the Interaction Manager module, the Media Source Manager and the Gain Manager module, c) the control of environment artifacts is taken care by the Ambient Controller module in cooperation with other control module such as smart space automation and media renderer, and d) user’s mobility management is addressed by using the users mobile device as the interface to access media and to store ambient media preference. In the following, we describe these aspects in turn in Section 4.3 to Section 4.6. Finally, we elaborate in Section 4.7 on the integration of several components of the framework.

Fig. 2 High-level architecture of the proposed ambient media provisioning framework

Multimed Tools Appl

4.3 Context model Context influences the selection and delivery of ambient media. For example, if a user is with his/her children, the system should select media (on request) that are properly censored. In general, context can be characterized using several attributes such as where (user’s location, Cloc ), when (time of presence, Ctime ), with whom (the companion of the user, Ccompanion ), what (the activities, Cactivity ), and even what mood (physiological states, Cmood ). This characterization is derived from existing work in context awareness such as [17, 18]. Several sensory inputs are usually processed to obtain these context attributes. By considering the above context attributes, a context model may support numerous context situations that can be represented as, Cx = Cloc × Ctime × Ccompanion × Cactivity × Cmood

(1)

Where, Cx is the context identifier of a particular situations, x is the total number of combinations. Note, however, the context attributes may be considered as optional, i.e. at a particular situation we may not have the values for all the context attributes. In that case also, the system should work with partial context information. In FAMe, we propose to use a discrete context model to identify each of the context situations. This helps to avoid handling unlimited context situations having minor differences among them. However, how many values each of the context attributes will have will depend on the level of granularity user needs and the particular context identification framework in use. 4.4 Ambient media preference and gain management The proposed framework dynamically maintains a set of media preferences for the user based on different contexts. We call this as Ambient Media Preference or AMP, which contains the scores of metadata-related attributes of different types of media services [6]. The scores reflect a degree of preference of the user on the different attribute values of media at a certain context. Figure 3 shows the structure of AMP with some sample preference scores of a user in a particular context. Note, we have considered lighting and volume in the same level as media type. This is done due to the fact that users usually have a preference of lighting, volume or similar aspects while they experience traditional media. For example, a user may prefer to watch movie under low light, while he/she may wish to listen to music with high volume. Note, for each media type, the AMP contains the scores for individual attributes of different media types. Each attribute may contain several dataitems and the sum of the scores of all the dataitems in a particular attribute is normalized to 1. The scores in the AMP for a user change over time, which reflect their changing need for media services in different contexts. During initialization, the AMP scores may be set to some given value or the user can explicitly input the scores. However, how to maintain the changing preferences of a user over the period of time is an issue to address. In this respect, we consider user’s interaction history and show how to update the AMP scores. In Section 4.4.1, we show the AMP score update process. In our approach, the contextual preference value is used later to estimate the gain of a service, which is to determine the extent a media service is preferable to a user. Our

Multimed Tools Appl Fig. 3 The AMP structure showing a set of sample preference attributes of a user in a particular context. Note, the dataitems corresponding to all the attributes are not shown in the figure for clarity

system uses this gain to determine the best services to be selected or recommended to the user. We explain the gain computation process in Section 4.4.2. 4.4.1 AMP score update Let us consider Fig. 3, which shows a portion of a sample AMP for a user, and Fig. 4, which shows a sample service usage history obtained based on user’s interaction in the environment. Note that a user may select any type of media at a particular instance, such as a movie and a news feed. However, in Fig. 4 we only show the case

Fig. 4 Example scenario of movie service usage over time in a particular context. Detail interaction data is suppressed for clarity of the figure

Multimed Tools Appl

of movie service to demonstrate the update of AMP. The computation of other type of services follows the same procedure. Note, Fig. 4 shows that a user has watched a movie alone while at home during weekend evening on a TV in low light and medium volume. Therefore, after the t1 instance the AMP for this user is recomputed to update the preference for each of this selection. This is done in two steps: (a) In the first step, the AMP score of the dataitem of a attribute (e.g. genre = action) that is selected is updated using (2).  wk,U (t) = i ,C x

1 + wk,Ui ,Cx (t − 1) × noInteractions(t − 1) 1 + noInteractions(t − 1)

(2)

 (t) is the updated score of the selected dataitem for a particular Where, wk,U i ,C x user (U i ) at a particular context (Cx ). The term wk,Ui ,Cx (t − 1) is its past score. Also, the term noInteractions(t − 1) is the number of interactions corresponding to the attribute (e.g. genre) being considered, which also represents that from the user interaction the scores of the dataitems of this particular attribute has been updated noInteractions(t − 1) times. (b) In the second step, the scores of all the remaining dataitems for the same attribute are updated using (3).  wk,U (t) = i ,C x

wk,Ui ,Cx (t − 1) × noInteractions(t − 1) 1 + noInteractions(t − 1)

(3)

We use the above equations to gradually update the scores of the dataitems of all the attributes in the AMP. For example, by considering the initial scores as of Fig. 3 and assuming noInteractions(t − 1) = 10, the updated dataitem scores of the genre attribute after instance t1 become: w Action,Ui ,Cx (t) = (1 + 0.30 ∗ 10)/11 = 0.36, using (2)  wComedy,U (t) = (0.30 ∗ 10)/11 = 0.27, using (3) i ,C x w Romantic,Ui ,Cx (t) = (0.20 ∗ 10)/11 = 0.18, using (3) w Horror,Ui ,Cx (t) = (0.20 ∗ 10)/11 = 0.18, using (3) Please note, in [6] we used a slightly different approach to update AMP and also defined a window of interval to determine the frequency of AMP update. However, in this paper we update the AMP scores after every instance of service usage with an aim to immediately reflect the preference of the users. 4.4.2 Dynamic gain estimation Based on the updated AMP scores, we compute the gain of the media services by following a three-step process. These are a) gain based on AMP; b) gain based on AMP and reputation; and c) overall gain computation. We describe these steps in the following. (a) Gain based on AMP: This step computes gain based on the AMP scores. The gain is computed for media services that are available in the media repository. We give an illustrative example of gain computation in Fig. 5. In this figure we consider a set of sample AMP scores as updated after time instance t1 (see Fig. 4) and based on these scores we estimate the gain of the media

Multimed Tools Appl

Fig. 5 Gain estimation of a media service based on AMP, given a set of sample AMP scores

services in the media repository. The process of gain estimation can be precisely expressed as,  gi,amp

=

j 

Scoredataitema ∗ βa

(4)

a=1  is the gain of the media service Si based on AMP scores of a user, Where, gi,amp which is considered as a temporary gain at this step. Scoredataitema ∗ βa refers to the score in AMP of an attribute’s dataitem of a media service Si , as computed using Eqs. (2) and (3). The variable j refers to all the attributes related to a  media service type (e.g. music) whose score is in AMP. gi,amp is computed as the weighted average of the AMP score of an attribute and the weight assigned to the attribute itself. The term βa in Eq. (4) refers to the relative importance of the attributes of a media service to a user. For example, a user might be more interested in the genre attribute than the other attributes and hence the genre attribute should have a higher weight when computing the gain. In the particular example shown in Fig. 5, we consider the weights 0.5, 0.3 and 0.2 corresponding to genre, actor and actress, respectively. In a similar fashion, the gain of all the available media services can be computed dynamically. (b) Gain based on AMP and media reputation: In our approach, we also consider service reputation as a factor that influences the computation of gain. Reputation usually refers to how good or trustworthy a service or vendor is in terms of content, delivery and other aspects. A reputation system collects feedback about the participants and products and helps to determine trustworthiness among the participants [16]. With respect to the ambient media services, reputation can play a role when determining a particular service gain based on the ratings provided by different users in a collaborative manner. For example, a particular movie may have been rated positively higher and such a rating score

Multimed Tools Appl

may be considered along with the AMP scores, when determining the gain of a service to a user. We should, however, note that in cases when reputation is not available, the proposed gain estimation method will ignore this value and estimate gain only based on the AMP scores. To compute gain based on AMP and reputation, we aggregate the temporary gain as computed using (4) with the reputation of the media services. This is done using (5) in the following.   gi,amp,R = α ∗ gi,amp + (1 − α) ∗ Ri i

(5)

 Where, gi,amp,R is the gain of the media service Si based on AMP and the i reputation Ri of that service. α and (1 − α) are the weights assigned to the AMP-based gain and the reputation, respectively. Therefore, in Fig. 5, if we assume the reputation of Movie1 as 0.74 and the value of α as 0.70, the updated gain of Movie1 becomes 0.42 according to (5). The value of α should be set as per the importance of AMP scores and reputation to a user. It should be noted that the gain estimation method uses AMP and reputation to compute gain in media services, where the reputation is subject to the availability. However, we may also ignore reputation if it is not available. This is clear from (5), where we can set α = 1 to ignore the reputation and give full importance to the AMP-based gain estimation. We also like to mention that even if the reputation of a service from an external source is not available, we can obtain it gradually by receiving direct feedback from the user. (c) Overall gain computation: In this step, we reevaluate the gain that is computed using (4) or (5). The reevaluation is based on the perspective that a particular media service (e.g. a movie) will have less gain to a user if it has been selected for his/her in the past. In particular, we need a mechanism that would restrict the selection of some media services multiple times in a row. Nevertheless, in the ambient context, this is not true for all services. There are cases when the media services would have no decay in gain (e.g. RSS feed, music). We define some rules in order to handle these cases that reflects the user’s requirements.

4.5 Environment control As mentioned earlier, our vision of ambient media also expands to controlling some appliances that would provide enhanced experience to the user. Therefore, the proposed framework integrates related environment control aspect while providing ambient media service to the user. For example, the setting of lighting level and the adjustment of volume based on the context. However, what would be the preferred lighting or volume level to a user is determined based on the past interaction history as described in Section 4.4. For actual control of artifacts, we follow the smart environment automation process. Accordingly, we adopt a web service based mechanism to issue command to the respective devices. The devices are controlled using standard automation mechanism, such as X.10. We further describe the automation issues in Section 5.

Multimed Tools Appl

4.6 User’s mobility One of the neat features of FAMe is that it handles user mobility using a simple yet intuitive approach. Because the user changes his/her location at different instance, we propose to keep the AMP scores in user’s mobile device. Therefore, we can support mobility at two different levels, a) intra-mobility, that enables mobility within an environment and b) inter-mobility, that facilitates mobility among different environments. We describe this two schemes in Section 4.6.1 and Section 4.6.2, respectively. 4.6.1 Intra-mobility Due to the use of mobile device as a placeholder for the updated AMP scores and also as an interface to access ambient media, a user is able to retrieve ambient media from any location within an environment. For example, from different room in a smart home. We propose to leverage the indoor location system in cooperation with the mobile device to facilitate this option. 4.6.2 Inter-mobility In this scheme, whenever a user moves to a different location outside an environment, the mobile device carry the updated AMP in the new location and negotiates with the smart system of that location to advertise its preference and accordingly obtain ambient media. 4.7 Framework integration In this section we describe how the different aspects described above are integrated in FAMe. Figure 6 shows the overall component model of FAMe. Note, this is a more detail version of the high-level architecture presented in Fig. 2. In Fig. 6, We have explicitly shown the corresponding stereotype with the links that follow between any two components. Note that in this figure, the Ambient Media Selector component acts as a coordinating component to establish connections among other components to perform ambient media selection tasks. The Media Source Manager connects to different external database for extracting the media descriptor. The Context Identifier is connected to several sensing devices based on which it identifies user’s current context. The Ambient Controller component uses the support of Home Automation component to control the environment artifacts. It also uses the Media Renderer component to select the appropriate rendering device and to control the volume of the audio output. The Gain Manager component collaborates with the Interaction Manager, Media Source Manager and AMP Manager to dynamically update the AMP scores based on the interaction history. The AMP Manager handles the access of the AMP scores to/from the AMP score repository. In our framework we propose to keep the AMP Manager and AMP score repository in the user’s portable device. In FAMe, a user gets support from the proposed framework to select the appropriate and relevant ambient media based on his/her current context. The media selection can occur in two different modes, which are automatic support mode and

Multimed Tools Appl

Fig. 6 Component model of the proposed ambient media provisioning framework

semi-automatic support mode. We describe these two process in Sections 4.7.1 and Section 4.7.2, respectively. 4.7.1 Automatic support In this mode, a user merely indicates the system that he/she is willing to experience ambient media. The system automatically takes care of the other issues, which includes the determination of context, suggesting the media that has the maximum gain, and finally rendering the media in the appropriate rendering device. In light of the overall component architecture (Fig. 6), we provide a sequence diagram in Fig. 7 to show the sequence of interactions that take place among the components in the automatic support mode.

Multimed Tools Appl

Fig. 7 Sequence of interactions that take place among the different components in automatic support mode for media provision

4.7.2 Semi-automatic support A user in this mode may choose to change the automatic media selection by changing the recommended media and the different other settings such as a rendering device. Basically, in this mode also the system recommends appropriate media and ambiance to the user as soon as the user asks for it. However, this option gives the user facility to override the media selection that is made by the system. Figure 8 shows the sequence of steps that occur in an example case of the change of renderer by the user. Similarly, the change of other settings takes place.

Multimed Tools Appl

Fig. 8 Sequence of interactions that take place in semi-automatic support mode for media provision. Note, in this figure we only show a case when the user changes the renderer selection that has been proposed by the system

5 Implementation and results This section describes the implementation details of a prototype system, which is based on the design of the proposed framework. In the following, Section 5.1 describes the experimental environment and the different devices connected on the environment. The development of different services are elaborated in Section 5.2. Finally, Section 5.3 shows some test results based on the developed framework. 5.1 Environment setup This section shows the layout of the experimental smart environment and the different devices that are in that environment. Figure 9 shows such a layout of the environment where ambient media will be provided. In this environment, all the rooms are equipped with X10 [22] lamp allowing user or system control of the light level. The environment is also equipped with Ubisense indoor location system [19], which provides continuous 3D positioning of any object or user equipped with a tag. Table 1 lists the different devices that are in the environment, which we use to conduct our experiment.

Multimed Tools Appl

Fig. 9 The layout of the experimental environment

Besides the devices listed in Table 1, all the networked devices are connected to a router using Ethernet (Media Center + Kitchen PC) and alternatively using WiFi connection (Laptop, SmartPhone and Ultra Mobile PC). 5.2 Development of different services In our implementation we adopted a web service based design approach so that different technology platform can be made interoperable. We leveraged OSGi and .NET development framework. We now briefly illustrate the design of different services and components, which are context identification service, AMP update and gain management service, Media server, Media renderer, Home automation service and Media provisioning service. Table 1 The devices/resources in the experimental smart environment Location

Devices

Description

Living room

Media Center

Bedroom Kitchen

Speakers 40” TV screen Dell Latitude D520 Laptop 19” TFT Monitor Speakers PC

Fujitsu-Siemens Scaleo E, running Windows XP Media Center Edition 2005 Ambient sound Media renderer

Portable location

Samsung Q1 Ultra Mobile PC HTC Touch Cruise Smart Phone

Media renderer Ambient sound X10 home automation interface is connected to this PC Used as a mobile renderer User interface, also contains AMP

Multimed Tools Appl

5.2.1 Context identification Our context identification service primarily depends on Ubisense [19] real-time location system. Using ultra-wideband (UWB) radio frequency technology, Ubisense location system proportionates 3D location of active tags (connected to people and/or device) in indoor environments. The Ubisense platform offers a .NET based SDK, allowing us to develop a ASP.NET Web Service wrapper to provide realtime location information of users and mobile objects at home. Therefore, using this technology we obtain the values of different context attributes such as who is around, their location and time. 5.2.2 AMP update and gain management service As mentioned in Section 4, that we maintain user’s interaction history to dynamically update user’s ambient media preference. We use (2) and (3) to update such preference. We use C# code to write a method for this purpose and later wrap it using in a ASP.NET web service so that this can be utilized easily whenever AMP update is needed. We develop another web service that collects rating data from the web (e.g. IMDB movie database) and uses this rating (if available) with with updated AMP scores to compute gain in a media service. This is based on the approach described in Section 4.4.2 and more specifically using (4) and (5). Note that the estimated gain value is used by the Ambient Media Selector to recommend the user a media that has the highest gain. 5.2.3 Media server We adopted the technique described in [20] to develop our media server. It is an OSGi based DVB-S and DVB-C Tuner for TV content streaming using RTP-RTSP protocol. We used the MediaCenter in our environment to store movies and songs. For content metadata acquisition, we consulted IMDB and TVDB through a web service that is built on top of the API provided by the respective parties. 5.2.4 Media renderer The Media Renderer Service in our design has been implemented using VLC [21]), which is a highly portable multimedia player supporting various streaming protocols and audio and video formats. VLC integrates a little HTTP server that can be used for a HTTP remote control interface. We developed a ASP.NET Web Service wrapper that controls the VLC player using VLC’s HTTP interface. Figure 10 shows the code snippet that is used by the media render web service to control the renderer. 5.2.5 Home automation service We use X10 [22] lamp modules and the ActiveHome CM11A Computer Interface to develop a ASP.NET Web Service that facilitates the lighting control in each room. Thus, the lighting level of each room can be set individually using standard Web Service protocols. Figure 11 shows the code snippet of the web methods in the web service for home automation.

Multimed Tools Appl

[WebService(Namespace = "http://www.mcrlab.uottawa.ca/")] [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)] public class VLCWebService : System.Web.Services.WebService { [WebMethod] public void Play() { HttpWebRequest myReq = HttpWebRequest.Create (@"http://localhost:8080/?control=play"); HttpWebResponse myResp = (HttpWebResponse)myReq.GetResponse(); myResp.Close(); } [WebMethod] public void SetMediaContent(string ContentUri) { HttpWebRequest myReq = HttpWebRequest.Create (@"http://localhost:8080/?control=add&mrl=" + ContentUri); HttpWebResponse myResp = (HttpWebResponse)myReq.GetResponse(); myResp.Close(); } }

Fig. 10 Code snippet of the two web methods in the media renderer web service

5.2.6 Media provisioning service We developed a ASP.NET Web Service for the Media Provisioning Service, running on the Kitchen PC, and acting as the entry point of ambient media selection at home. A Web Service client application with GUI in the user’s SmartPhone has

[WebService(Namespace = "http://www.mcrlab.uottawa.ca/")] [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)] public class X10LampWebService : System.Web.Services.WebService { [WebMethod] public void SetLightLevel(int newLightLevel) { return dllX10.SetLightLevel(this.HouseCode, this.DeviceCode, newLightLevel); } [WebMethod] public bool SetDim(int dimValue) { return dllX10.SetDim(this.HouseCode, this.DeviceCode,dimValue); } [WebMethod] public bool SetBright(int brightValue) { return dllX10.SetBright(this.HouseCode, this.DeviceCode, this.HouseCode, brightValue); } [WebMethod] public int getLightLevel() { return this.CurrentLevel; } }

Fig. 11 Code snippet of the four web methods in the lighting control web service

Multimed Tools Appl

been developed that access the proposed framework and uses the media provisioning service to ask for ambient media. In Fig. 12, we show the interconnections among the components of the developed framework. Figure 12 also shows how the connections among the components have been implemented using Web Service paradigm. The service components offer a Web Service port as exposed interface of internal business logic, and the client components use Web Service clients as proxies of the desired service. For the sake of simplicity, we have only shown the business logic part in some of the components in this figure. Note, in our design, the AMP Manager and Ambient Media Selector GUI resides in the smart phone. 5.3 Results In this section, we first show the user interface of the developed framework in Fig. 13. Here, the user initially sees the interface in Fig. 13a. In this screen the user may choose the type of media he/she wants. Once the user selects any of the options, the system automatically obtains the current context information and recommends the media of the selected type using the interface in Fig. 13b. There is a timing control at this stage such that if the user does not like to override the automatic selection, the system starts to deliver the media in the appropriate rendering device and sets the volume and lighting condition accordingly. Otherwise, the user may choose to override the automatic selection by changing the individual parameters. Figure 13c shows the case when the user wants to change the lighting level.

Fig. 12 The interconnection among the different components

Multimed Tools Appl

Fig. 13 The user interface of the prototype system on a real HTC phone (a–c)

We now show how the user’s contextual media preferences change and accordingly how the AMP scores are updated. When the user selects ambient media over a period of time using the smart phone user interface, their interaction history is kept in the interaction history repository, which are then used by the Gain Management service to update AMP scores and compute gain dynamically. Based on the interaction data, the scores are updated using (2) and (3). In Fig. 14, we only show few instances of AMP updates for clarity. It is clear from Fig. 14 that the scores in AMP are updated after every interaction. These scores reflect the changing preference of a user in the different metadata attributes of the media services. For example, at the second interaction, the user selected an action movie with actress2 and actor1. Therefore, the AMP scores corresponding to these dataitems are incremented at this interaction while the scores of the other dataitems of other attributes are decremented. Note, although in this figure we only show the scores updates of movie attributes, we applied our proposed mechanism to update AMP scores that correspond to the attributes of other media services, such as music and news. We conducted usability experiment with limited number of users to check whether the system provides appropriate media service based on their preferences. The initial response from them has been very positive. We plan to conduct extensive test with the system and evaluate several aspects such as ease of use, the appropriateness of the selected media, and the response time of the system in future. Note that we conducted experiment in a smart home environment and could provide users with ambient media depending on their location within the home. As we keep the AMP scores in the smart phone, the home system could use the AMP scores from the phone and accordingly compute gain to recommend relevant media

Multimed Tools Appl

Fig. 14 Changes in AMP scores after movie service usage by a user in a particular context. Here, a refers to the changes in genre b changes in actress and c changes in actor attributes

service. We plan to deploy this framework in other environments such as in office and let that environment use the AMP scores stored in the personal hand held device. 6 Conclusion This paper presents our proposed framework for providing ambient media services based on user’s context. The proposed framework incorporates several key aspects related to ambient media provision, which are context identification, dynamic preference update and contextual gain estimation of media services, the control of household appliances that improve user’s media perception, and the mobility management of the user such that user’s media preference are available at different locations. We also adopt a human-centered design approach in order to keep the user at the center of focus, that is, to provide media support to user when he/she needs it and to provide a very simple but intuitive interface in order to reduce the cognitive load in selecting the media. In this paper, we elaborate the design and development of the overall framework and demonstrate in a smart home setting that the proposed framework provide intuitive mechanisms to experience ambient media. Our future work includes the adoption of this framework at different environments and conduct extensive user studies.

Multimed Tools Appl

References 1. Adomavicius G, Sankaranarayanan R, Sen S, Tuzhilin A (2005) Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans Inf Sys 23(1):103–145 2. Aroyo L, Bellekens P, Björkman M, jan Houben G (2008) Semantic-based framework for personalised ambient media. Multimedia Tools and Applications 36(1):71–87 3. Billsus D, Pazzani MJ (2000) User modeling for adaptive news access. User Model User-Adapt Interact 10(2–3):147–180 4. Ducatel K, Bogdanowicz M, Scapolo F, Leijten J, Burgelman JC (2001) Scenarios for ambient intelligence in 2010. IST advisory group final report. ftp://ftp.cordis.lu/pub/ist/docs/istagscenarios 2010.pdf 5. Gellersen H-W, Schmidt A, Beigl M (1999) Ambient media for peripheral information display. Pers Ubiquitous Comput 3(4):199–208 6. Hossain MA, Atrey PK, El Saddik A (2008) Dynamic gain estimation in ambient media services. In: Proceeding of the 1st ACM international workshop on semantic ambient media experiences (SAME ’08). ACM, pp 49–56 7. Hossain MA, Atrey PK, El Saddik A (2008) Gain-based selection of ambient media services in pervasive environments. Mob Netw Appl 13(6):599–613 8. Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’97), pp 234–241 9. Ishii H, Wisneski C, Brave S, Dahley A, Gorbet M, Ullmer B, Yarin P (1998) ambientROOM: integrating ambient media with architectural space. In: CHI 98 conference summary on human factors in computing systems, Los Angeles, pp 173–174 10. Jaimes A (2006) Human-centered multimedia: culture, deployment, and access. IEEE MultiMed 13(1):12–19 11. Jaimes A, Gatica-Perez D, Sebe N, Huang TS (2007) Guest editors’ introduction: humancentered computing—toward a human revolution. IEEE Comput 40(5):30–34 12. Lugmayr A (2007) Ambient, ambient, ambient—what are ambient media? In: TICSP adjunct proceedings of EuroITV 2007, vol 35, Amsterdam, pp 89–93 13. Millard I, Roure DD, Shadbolt N (2005) Contextually aware information delivery in pervasive computing environments. In: Proceedings of the international conference on location and context-awareness, Germany, pp 189–197 14. Park MH, Hong JH, Cho SB (2007) Location-based recommendation system using bayesian user’s preference model in mobile devices. In: 4th international conference on ubiquitous intelligence and computing, Santa Barbara, pp 549–558 15. Qin W, Zhang D, Shi Y, Du K (2008) Combining user profiles and situation contexts for spontaneous service provision in smart assistive environments. In: 5th international conference on ubiquitous intelligence and computing, Oslo, pp 187–200 16. Resnick P, Kuwabara K, Zeckhauser R, Friedman E (2000) Reputation systems. Commun ACM 43(12):45–48 17. Schilit B, Adams N, Want R (1994) Context-aware computing applications. In: Mobile computing systems and applications workshop, Santa Cruz, pp 85–90 18. Schmidt A, Beigl M, Gellersen HW (1999) There is more to context than location. Comput Graph 23(6):893–901 19. Ubisense (2008) Ubisense homepage. http://www.ubisense.net/. Accessed 17 December 2008 20. Ugalde JB, Huici IA (2007) Convergence in digital home communications to redistribute iptv and high definition contents. In: 4th IEEE consumer communications and networking conference (CCNC’07), pp 885–889 21. VLC media player (2009) VLC media player homepage. http://www.videolan.org/vlc/. Accessed 25 January 2009 22. Wikipedia (2009) X10 (industry standard). http://en.wikipedia.org/wiki/X10_(industry_standard). Accessed 10 January 2009 23. Xu K, Zhu M, Zhang D, Gu T (2008) Context-aware content filtering & presentation for pervasive & mobile information systems. In: Proceedings of the 1st international conference on ambient media and systems, Quebec, pp 1–8 24. Yu Z, Zhang D, Zhou X, Li C (2005) User preference learning for multimedia personalization in pervasive computing environment. In: Proc of 9th international conference on knowledge-based intelligent information & engineering systems, Melbourne, pp 236–242

Multimed Tools Appl 25. Yu Z, Zhou X, Zhang D, Chin C, Wang X, Men J (2006) Supporting context-aware media recommendations for smart phones. IEEE Perv Comput 5(3):68–75 26. Zhu M, Zhang D, Zhang J, Lim BY (2007) Context-aware informative display. In: IEEE international conference on multimedia and expo, China, pp 324–327 27. Zukerman I, Albrecht DW (2001) Predictive statistical models for user modeling. User Model User-Adapt Interact 11(1–2):5–18

M. Anwar Hossain received the B.Eng. degree in Computer Science and Engineering from Khulna University, Bangladesh, and the M.C.S. degree in Computer Science from the University of Ottawa, Ottawa, ON, Canada, in 2005. He is currently working toward the Ph.D. degree with the Multimedia Communications Research Laboratory (MCRLab), School of Information Technology and Engineering, University of Ottawa. He worked for a few years in industry with Chrysler Canada Inc. (former DaimlerChrysler Canada) and NTG Clarity Networks Inc. before he completed the M.C.S. degree. His research interests include human-computer and human-environment interaction, information visualization, multi-sensor systems, context-awareness and ambient intelligence. He has authored and co-authored more than 25 publications including refereed journals, conference papers, and book chapters.

Jorge Parra IEEE member, received his MS in Industrial Engineering in 1999 from the University of the Basque Country (UPV/EHU), Spain. He is currently pursuing his PhD in the Department of Electronics and Telecommunications at the Faculty of Engineering of Bilbao. He has been working in Ikerlan-IK4 since 1999 as a researcher in the Software Technologies Area, where he has taken part in a number of projects related to home systems. His research interests include distributed systems, home automation, service oriented architectures and Ambient Intelligence.

Multimed Tools Appl

Pradeep K. Atrey is an Assistant Professor in the Department of Applied Computer Science at The University of Winnipeg, Canada. He was with the Multimedia Communications Research Laboratory, University of Ottawa, Canada as a Postdoctoral Research Fellow from 2006 to 2008. He obtained Ph.D. in Computer Science from National University of Singapore in 2006. He was a Lecturer at the Delhi College of Engineering, University of Delhi and at the D.B.C.R. University of Science and Technology, Murthal in India from 1992 to 2002. His research expertise is in the area of Multimedia Computing with a focus on Surveillance Security, Privacy, Smart Environment, and Web. He has authored and co-authored more than 35 refereed journals, conference papers, and book chapters.

Abdulmotaleb El Saddik (F’IEEE-09) is University Research Chair and Professor, SITE, University of Ottawa and recipient of the Professional of the Year Award (2008), the Friedrich WilhelmBessel Research Award from Germany’s Alexander von Humboldt Foundation (2007) the Premier’s Research Excellence Award (PREA 2004), and the National Capital Institute of Telecommunications (NCIT) New Professorship Incentive Award (2004). He is the director of the Multimedia Communications Research Laboratory (MCRLab). He is a Theme co-Leader in the LORNET NSERC Research Network. He is Associate Editor of the ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMCCAP) and IEEE Transactions on Computational Intelligence and AI in Games (IEEE TCIAIG) and Guest Editor for several IEEE Transactions and Journals. Dr. El Saddik has been serving on several technical program committees of numerous IEEE and ACM events. He has been the General Chair and/or Technical Program Chair of more than 20 international conferences, symposia and workshops on collaborative haptoaudio-visual environments, multimedia communications and instrumentation and measurement. He

Multimed Tools Appl was the general co-chair of ACM MM 2008. He is leading researcher in haptics, service-oriented architectures, collaborative environments and ambient interactive media and communications. He has authored and co-authored two books and more than 200 publications. He has received research grants and contracts totaling more than $10 million and has supervised more than 90 researchers. His research has been selected for the BEST Paper Award three times. Dr. El Saddik is a Senior Member of ACM, and is an IEEE Distinguished Lecturer.