A middleware that provides application session and device management would greatly simplify the application development for smart homes. Increasing storage ...
InterPlay: A Middleware for Integration of Devices, Services and Contents in the Home Networking Environment Alan Messer, Henry Song, Praveen Kumar, Phuong Nguyen, Anugeetha Kunjithapatham, Mithun Sheshagiri Samsung Information Systems America (SISA) San Jose, CA, USA {amesser, hsong, pkumar, pnguyen, anugeethak, msheshagiri}@sisa.samsung.com
Abstract - Current home networking technologies such as UPNP, Jini and HAVi, only focus on connecting devices electronically in a variety of different forms that allow simple device interactions. The result is that devices act in incompatible islands, making their use confusing to users and a nightmare for application developers. To realize a smart home environment, we need to seamlessly integrate devices and services into our every day lives. In this paper, we describe a middleware platform to provide the integration of devices, services and contents in a home. Our middleware facilitates integration of both data (content, preferences, etc.) and applications across home networking technologies to improve consumer and developer usability. Keywords-middleware; home networking; integration; single view; session
I
INTRODUCTION
Future smart homes will have a variety of intelligent devices. These devices are able to be networked and can store massive amounts of content. In order to take full advantage of device capabilities, they need to perform three tasks – (1) they need to be controllable in an ad-hoc environment independent of devices’ underlying networking technologies; (2) they need to collaborate with each other to enable sophisticated services for a user; and (3) they need to provide easy, seamless contents access. We argue that smart homes must provide more middleware support for devices collaboration, services integration and contents aggregation than those provided by current networking technologies, such as UPnP [1], HAVi [2], and Jini [3]. We believe that the middleware must not only support devices communication, but also needs to understand devices’ functionalities regardless of their differing underlying technologies. This requires a middleware to abstract devices with technology-independent description of devices and capabilities. Users in a smart home benefit from sophisticated applications that involve collaboration of multiple devices and complex interactions. Current home networking
technologies only provide simple control interfaces for devices. As a result, an application that requires complex interactions among devices must manage each device separately. This slows down market adoption of devices, and increases application development cost. We argue that application session 1 and device management must take a systematic approach. A middleware that provides application session and device management would greatly simplify the application development for smart homes. Increasing storage capacity means a home can hold huge amount of contents. Finding the right content becomes increasingly difficult as contents are scattered among devices and in different directories. To find a particular content, a user must first know the exact location of the content including the device and directory. We believe that a middleware must support aggregation of contents from devices in a home; resolving duplication and versioning such that they can be easily located regardless of the data locations on devices. There can be multiple users in a home. Each user may have personal preference on devices and contents settings. For example, a user may prefer to turn the subtitle on while watching DVD movies. In addition, access control on contents and devices are also important in a home for users. For example, children in a home should not be allowed to view some contents that are intended for adults only. We believe that a middleware must support personalized entertainment experience for users in a home with preferences and access control policies. In this paper, we describe InterPlay – a middleware for networked home environment. Building on top of the existing home networking technologies, InterPlay provides devices, services, and contents integration that meets the above requirements.
1
We define application session as application states in an application lifecycle
1-4244-0086-4/06/$20.00 (C) 2006 IEEE This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2006 proceedings.
The rest of the paper is organized as follows. Section II provides discussion of the InterPlay architecture in details. Section III provides brief discussion our prototype implementation of the InterPlay on a consumer electronic (CE) device with limited resources. Section IV describes related works, and Section V concludes the paper with future directions. INTERPLAY ARCHITECTURE
II
InterPlay provides support for device integration, contents aggregation and session management in a home networking environment with heterogeneous devices using different networking technologies. InterPlay is designed as three-layer architecture. Figure 1 shows the major building blocks in each layer.
InterPlay Applications Service Abstraction Layer AV Service
Printing Service
Content Manager
Sync Service Session Manager
Single View Layer Content Device Directory Directory
User Directory
Common Framework Layer Eventing
Device Communication
Device Discovery
Devices Figure 1. InterPlay Architecture
We divide InterPlay major functions into three layers. Each layer focuses a different concern in the system. In a heterogeneous environment, we must provide interoperability for devices. This is the focus of the common framework layer. The layer is a shared infrastructure that provides lower-level functionalities that hide device heterogeneity from upper layer components. It also provides system-wide functionalities, such as an event system, for other components in the system. Enabling device interoperability allows us to focus device abstraction and data aggregation in the single view layer. The single view layer aggregates heterogeneous devices, contents and user data into a consistent, single view point. This layer provides three basic functions. The first is device abstraction. It abstracts heterogeneous devices obtained from the common framework layer and describes them in a platform independent format with high level device capabilities. The second function is the content aggregation. It fetches the content metadata from devices, resolves duplication and
versioning. It provides content information that is independent of their locations. The third function is the user data aggregation including user preference and user access control policies for devices and contents in a home. This layer simplifies accessing the devices and their contents for users. We resolve application session management in the service abstraction layer. The service abstraction layer provides applications with session management that relieves the applications from managing devices individually. To further simplify the development of applications that requires multiple devices collaboration, the layer also provides enhanced services for common applications, such as A/V applications, data synchronization applications, and printing applications. Utilizing the InterPlay middleware, InterPlay applications provides user interface to control devices in a home. A. Common Framework Layer The common framework layer hides the device heterogeneity by providing a set of device-independent interfaces to the upper layers and informs the upper layer components when devices’ states change. Device Discovery - Device discovery provides functions to discover devices in a home. When a device becomes online in a home, its device type, services and how its services can be accessed are discovered. This information is passed to the device communication and components in the single view layer via the eventing component. To accommodate different types of networking technologies, the device discovery contains multiple plugins, each of which is used for a specific networking technology. Device Communication - Device communication provides interfaces to access devices in a protocolindependent way. Components in the single view layer use the device communication for service invocation and content access. Like the device discovery, the device communication contains multiple plugins. Each plugin is designed to communicate with devices with a particular networking technology. When accessing shared resources, the device communication is able to (un)lock them on a device if the device supports the locking mechanism. Eventing – Eventing is responsible for dispatching events inside InterPlay, and between the InterPlay middleware and InterPlay applications. The eventing employs a WhiteBoard pattern-based [4] model, which decouples event sources from event listeners. This makes the system more reliable by allowing event sources and event listeners to fail without affecting one another. B. Single View Layer The single view layer abstracts devices, aggregates contents and user data into a central place where upper layer components have a single view on them. Device Directory – The device directory contains metadata of devices that have been discovered in the home. It provides an interface to upper layer components for monitoring devices’ states. In addition, the device directory classifies devices into categories based on services they
This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2006 proceedings.
provide. For example, an A/V device is a device that is capable of playing A/V contents. The categorization of devices enables InterPlay to provide device independent interfaces for device control, device monitoring and to dispatch device independent events regarding to device states changes. Content Directory – The content directory aggregates metadata of contents. It resolves the content duplication and versioning such that only the most up-to-date copy is visible. The content directory organizes contents with a 2-level structure such that locating content does not require traversing the original content hierarchies. The 1st level contains albums and those contents that do not belong to albums. Within each album, the content directory contains the contents that belong to the album. Contents in the content directory are updated when devices become online/offline User Directory – The user directory contains user preferences, user access control policies on devices and contents, and other data in a home. Firstly, it is a central repository where components can obtain personal reference for devices. For example, it can contain preferences of the home theater system for a user such that whenever the user wants to play music on it, the application can automatically tune the volume to the users’ liking. Secondly, the user directory also contains access control policies on devices and contents for users. For example, a home owner can specify that certain contents are not visible and accessible to children. If devices in a home contain access control mechanism (e.g., UPnP security), policies from these devices are aggregated in the user directory. Thirdly, the user directory can contain other data. For example, a user may select a list of favorite songs from different CD albums, groups them into a playlist, and saves them in the user directory. When the user wants to play these songs, he does not need to search them one by one. Instead, he only needs to play the saved the playlist. C. Service Abstraction Layer The components of the service abstraction layer are designed for interactions with InterPlay applications. The design goals of this layer are to provide (1) query interfaces for contents, (2) identify and authenticate users, (3) control interfaces for sessions and devices, and (4) enhanced services involving collaboration of multiple devices and contents. Content Manager - InterPlay applications access contents in a home using interfaces from the content manager. The content manager allows applications to browse contents, query contents based on certain criteria, access control policies and group contents in playlists/albums. The content manager obtains the content information from the content directory and the user directory. Session Manager – the session manager manages session lifecycles and devices’ states for InterPlay applications. Control interfaces provided by session manager allow other components and applications to change the session from one state to another state without dealing with a device directly. After a session is created, devices can be added to and removed from the session. When a device is added to a session, the session manager starts to monitor the device’s
states using the device directory. The dynamicity of addition and removal of devices in a session enables complex device interactions without complicated logic in applications. For example, an application can add a second TV to during its DVD playback session such that the movie would be displayed on the two TVs simultaneously. The application would not need to deal with the second TV directly. In stead, the session manager manages and monitors the second TV. 1) Enhanced Services We have designed three types of enhanced services for most common applications in a home: audio/video entertainment, image printing and content synchronization. These services are built with the session manager and content manager, and are designed such that an application does not need to configure and control devices directly. AV Service provides functionalities to perform A/Voriented operations. It can utilize any media rendering and transcoding devices, and can chain them into a rendering pipeline for transcoding. Printing Service provides functionalities to perform printing operations. It contains a set of simple interfaces for printing images. Sync Service provides synchronization of contents from one device to another device. Using the synchronization interface, a user can synchronize a single content or a list of contents (e.g., an album). III IMPLEMENTATION We have implemented InterPlay on a digital TV (DTV) prototype with J2ME CDC Foundation Profile. To test our implementation, we used a variety of media players, media servers, and cell phones with Ethernet and WiFi connections, all of which use UPnP. We have also implemented UPnP plugins in the device communication and device discovery within the InterPlay for supporting UPnP devices. By adding this support for controlling existing UPNP devices we are able to control existing heterogeneous devices in an uniform and integrated fashion. We also implemented PIN-based user identification mechanism for user authentication and authorization. A PIN is a 4-digit number that can be easily entered from a remote control. Each user in a home has a unique PIN number. The InterPlay system uses PIN number to obtain preferences and security control for the user. We have measured the runtime footprint of our InterPlay implementation. The memory footprint for InterPlay middleware (not including J2ME CVM) is about 4M of memory, of which 2M is used by the UPnP stack for the control point. 1) Usage Scenario InterPlay and its enhanced services enable advanced applications with sophisticated collaboration of devices. In this section, we describe a particular application scenario where a user is able to migrate a live DVD movie session from one TV to another, as shown in Figure 2. The scenario demonstrates that advanced applications can be developed easily with InterPlay middleware. The scenario contains two DTVs and a
This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2006 proceedings.
networked DVD player. DTV A is an UPnP media renderer as well as an UPnP control point. InterPlay resides on top of this UPnP control point. DTV A also contains the InterPlay application with a UI that a user can use to locate devices and contents quickly on the TV screen. DTV B is a regular DTV with an UPnP media renderer. The DVD player is a media server that is networked with DTV A and DTV B. DTV A
InterPlay Application 1
UPnP Media Renderer
2
InterPlay Middleware UPnP Control Point 4
3 2 DVD Player UPnP Media Server
DTV B 5
UPnP Media Renderer
Figure 2. Session Migration
An application uses DTV A to locate a DVD movie in the DVD player, and wants to play the movie on DTV A. The InterPlay application first identifies the user by asking the user’s PIN number. It then creates a session using the session manager. The application then adds the DVD player to the session as the media source; and adds DTV A to the session as the media sink. The user starts to play the movie. The movie starts to stream from the DVD player to DTV A. DTV A is set according to the preference data in the user directory. The session manager transitions the session state from initialized to active. At a later time, the user pauses the play session, and decides to migrate the session from DTV A to DTV B. When the session is paused, the session manager transitions the session state to idle; and saves the session states and the movie status. The application removes DTV A from the session, and adds DTV B to the session. The application uses the session manager to resume the session. The session manager restores the session states, initializes DTV B and starts to stream the media from the DVD player to DTV B. Again, DTV B is set according to user’s preference. IV RELATED WORKS Gaia [13], Interactive Workspace [12], and Windows Media Center [11] have similar goals as InterPlay. However, Media Center is centered on the concept that a home desktop
PC is the home entertainment center. In comparison, InterPlay takes a different view. We believe that a TV should be the center of home entertainment that offers “lean-back” experience for home users. In addition, our design and implementation place much less requirements on hardware resources. Interactive Workspaces focus on collaborative environment, InterPlay is designed for a home environment where diversified activities can take place and ease of use is very important. By comparison the Gaia application framework requires the predefined composition templates, applications in the Interplay can dynamically allocate resources, and run on devices by using the session manager. NFS [6] and Samba [10] are networked file systems that allow a device to integrate a part of a file system from another device. Both are for traditional computing devices that do not aggregate files and resolve duplication and versioning. The Coda file system [7] is designed for mobile computing environments where disconnection must be accounted for. Like NFS, Coda preserves the file system structure, which makes it unsuitable for CE devices. InterPlay, however, works in the home networking environment where network disconnection occurs much less frequently. Unlike existing approaches, InterPlay aggregates contents metadata from devices into a flat structure such that a particular content can be easily located. This is especially important on a CE device where navigation to a file is inconvenient. In addition, InterPlay dynamically adds content metadata when these become available and removes these when they go offline. Session management is important in a pervasive computing environment, such as a networked home. EasyLiving’s [8] Geometric Model (EZLGM) provides a general geometric service for applications. The application can query the service for adjacent resources to migrate the application from a set of entities to another set of entities. Applications, however, must manage the session states transfer. This is in contrast to our session management model where applications explicitly request devices to join and leave a session while the session manager manages the session and device states on behalf of an application. Our session manager is similar to the task manager in Aura [9]. The task manager is responsible for allocating resources and restoring session states for an application when it migrates from one environment to another. Unlike the task manager in Aura, where it allocates devices and resources before a session migrates, our session manager is more flexible in the way that it allows devices to dynamically join and leave a session. V
CONCLUSION
In this paper, we have described the motivation, the design and the implementation of the InterPlay middleware platform. InterPlay provides seamless collaboration of heterogeneous devices with different networking technologies, aggregation of contents and other data from different devices and directories, and a systematic session management framework for complex applications. We have implemented InterPlay on a DTV prototype with a small footprint and offers real world performance.
This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2006 proceedings.
There are many future directions in which to improve InterPlay. We would like to add context information (e.g., user location, device location and etc.) in the system such that the user can have personalized entertainment experiences. We would also like to integrate security and DRM into the system for user authentication, content access control and content distribution.
REFERENCES [1]
The UPnP Forum, http://www.upnp.org
[2]
Home Audio/Video Interoperability, http://www.havi.org
[3]
The Jini Network Technology, http://www.jini.org
[4]
The OSGi Alliance, “Listeners Considered Harmful: The Whiteboard Pattern”, Technical Paper
[5]
JSR 46: J2ME Foundation Profile, http://jcp.org/aboutJava/communityprocess/final/jsr046/index. html
[6]
Sun Microsystems, “RFC 1094 – NFS: Network File System Protocol Specification”, Mar. 1999
[7]
M. Satyanarayanan, “Evolution of the Coda File System”, In ACM Trans. Computer Systems, ACM Press, 2002
[8]
B. Brumitt, B. Meyers, J. Krumm, A. Kern, and S. Shafer, “Easyliving: Technologies for Intelligent Environments”, In Proc. of the International Conference on Handheld and Ubiquitous Computing, 2000
[9]
J. P. Sousa, D. Garlan, “Aura: an Architectural Framework for User Mobility in Ubiquitous Computing Environment”, In Proc. of the 3rd Working IEEE/IFIP Conference on Software Architecture (WICSA’2002), May 2002
[10] The Samba Project, http://us2.samba.org/samba/ [11] Windows Media Center, http://www.microsoft.com/windowsxp/mediacenter [12] B. Johanson, A. Fox, and T. Winograd, “The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms”, IEEE Pervasive Computing Magazine 1(2), AprilJune 2002 [13] M. Roman, C. K. Hess, R.Cerqueira, A. Ranganathan, R.H. Campbell, and K. Nahrstedt, “Gaia: A Middleware Infrastructure to Enable Active Spaces”, IEEE Pervasive Comupting, pp 74-83, Oct-Dec 2002.
This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2006 proceedings.
