Design and Implementation of Media Content Sharing ...

Design and Implementation of Media Content Sharing Services in Home-based IoT Networks Chih-Lin Hu1 , Hung-Tsung Huang1 , Cheng-Lung Lin1 , Nguyen Huu Minh Anh1 , Yi-Yu Su2 and Pin-Chuan Liu2 1

Department of Communication Engineering National Central University Taoyuan, Taiwan, R.O.C. 2 Information & Communications Research Laboratory (ICL) Industrial Technology Research Institute of Taiwan Hsinchu, Taiwan, R.O.C. Abstract—The penetration ratio of broadband networks into residential areas increases rapidly by the wide distribution of Internet service providers and networks. People are able to distribute and play various media content with many types of networked multimedia devices for home multimedia entertainment in residential environments. This paper addresses a new idea of home-based IoT networks where home-networked devices are able to communicate with others in a friendly, networked manner instead of traditional manual configurations and wired cabling operations. Accordingly, this paper proposes a novel intelligent media distribution system based on a home-based IoT network. The design of this system integrates UPnP, face recognition, intelligent human-machine interface, and family database technologies. UPnP-compatible HNDs With UPnP, networked devices can discover neighboring devices in a network. Face recognition is incorporated and so provides the UPnP networked devices with the capability of identifying the operating user in front of them. When a user moves in a home-based network, the intelligent human-machine interface allows a user to enforce any media content to be distributed to or displayed onto the UPnPbased device nearby the user. Furthermore, this paper presents a prototypical development, as well as a real demonstration with experimental UPnP-based network devices in home networks. Therefore, the study in this paper enables a ubiquitous media distribution service in home-based IoT network environments. Keywords—Media content sharing, smart home, digital home, home networks; IoT.

I.

I NTRODUCTION

Not only traditional personal computers and laptops, many modern devices of consumer electronics (CE) and mobile handheld devices have high-end computation, extensible storage, networking, and image capturing and processing capabilities. When people have broadband Internet access services at home, they like to experience digital home a new digital lifestyle using various home-networked devices (HNDs) such as PC, CE, mobile device categories in residential environments. In the recent decade, the trend of digital home stimulates the introduction of many new data and information applications dedicated to household services. Among home automation, multimedia entertainment, and security and surveillance is the media content sharing service that is much close to customers from the viewpoint of CE markets [1]. It is desirable that family members could share audio and visual content with each other when they stay in different zones inside the house.

For example, Bob in a study wants to share some video stored in his PC with Alice in the living room. Then, Alice turns on a TV and play the video in a networked manner when both the PC and TV are connected onto the home network. The above use case may be generalized for various HNDs that exist in different zones inside a house. As long as HNDs appear in a home network, they can find neighbors automatically and according to user operations. Note that home-networked multimedia entertainment indeed offers an easy way to operate media content services, as compared with the ordinary, manual operations that are complicated or difficult to elder or junior users. Consider that HNDs in a home network introduce a new playground of home-based Internet of Things (IoT) where HNDs are able to perform IP networking, data transfer functionality in a restricted network domain. Based on a homebased IoT network, this paper designs a novel intelligent media distribution system for media content sharing among HNDs in residential environment. The design of this system comprises several functional components, including a UPnP protocol stack, face recognition, intelligent human-machine interface, and family database. This design is applicationlevel and platform-independent, so resulting in a software application that can be imported and activated on various HNDs. Specifically, HNDs behave the device and service discovery services according to the interoperability specifications announced by Digital Living Network Alliance (DLNA) and Universal Plug and Play (UPnP) Forum [2]. Any UPnPcompatible HNDs in a network can know the existence of neighboring UPnP devices. UPnP AV technologies [3] also allow devices to access and control media playing services between media server and media player devices. In addition, the system integrates face detection to provide a human awareness service. This result by face recognition is used by the intelligent human-machine interface to identify the operating user as well as provide personal services in reference to user profiles from the family database. Functionally, this paper develops an interesting media content sharing scenario when the proposed intelligent media distribution system is employed in a home-based IoT network., When a user is free to move and close to a device in a house, this device can detect the appearance and figure out

the identity of this user. More importantly, this device can reflect the location change of this user and notify the device in another zone that a user previously visited. So, the media playing service that a user operated in another zone can be transferred to the device in the current zone. In this way, whenever the user moves inside a house, the media content is always distributed to the HND nearby the user, thereby resulting in a kind of ubiquitous multimedia service in homebased IoT network environments. Hence, the technical content of this paper specifies an intelligent media distribution system for media content sharing services in home-based IoT networks, as well as a novel scenario that people can play personal media content services, anywhere with any UPnP-compatible HNDs, in residential areas. Prototypical implementation is deployed on several off-the-self HNDs, including personal computer, laptop, and mobile phones. Real demonstration manifests the effects of the proposed system and services. The remaining of this paper is organized as follows. Section II mentions the related technology. Section III describes the system architecture and the function of each component. Prototypical development and implementation are given in Section IV. The conclusion sums up in Section V. II.

R ELATED W ORK

In recent several years, the upgraded hardware and software capabilities of HNDs, particularly TCP/IP networking and image capturing and processing, support network and multimedia services in home networks. This section briefs the background knowledge of the UPnP technology and the image recognition technology, the respective uses in the digital home domain, and the likelihood of combining both technologies for new home network services A. UPnP Applications In regard to DLNA interoperability specifications, UPnP is recognized as a protocol framework for developing network applications in home networks, particularly home multimedia services. In addition to stationary media players and servers, Hu, et al. [4] firstly introduced a role of mobile media server which can transfer media content to be played on media renderers, showing mobile video media sharing scenarios. In [5], Hu, et al. designed an application gateway that integrated peer-to-peer (P2P) file distribution and UPnP technologies. This design allows UPnP devices to access P2P media downloading and streaming services, so introducing a hung volume of media content on the Internet into home networks. Hu, et al. [6] further incorporated a Near Field Communication (NFC) facility with the UPnP functionality and proposed a novel gesture-assisted remote control design to instruct media playing services between any two HNDs in home networks. After our companioned works in [4][5][6], this paper employed an intelligent control method using face recognition to conduct personal media services between UPnP devices in home-based IoT networks.

such as recognizing vehicle license plate number, hand-gesture, face recognition, etc. Regarding the demand of face recognition in this paper, [7] surveyed some face recognition technologies and applications for smart interactions. Particularly, there are two main kinds of smart interaction applications: face recognition on smart devices, and face recognition in smart environments. Both of them compare partial characteristics of captured face images with stored ones to discern the identity of a user. Compared with conventional usage on security and access control systems, face recognition is not often employed on CE devices in the past because it induces high computation cost that conventional CE devices may not support. In the literature survey, few works integrates both UPnP and image recognition on HNDs. In [8], the work considered that many UPnP devices have ambiguous name presentations in a network, so a user is not easy to properly discern a target device. This work designed a mixed interaction method by which a target UPnP device can be selected with not only its naming identity but also its image captured by mobile phones. Notice that modern HNDs in recent years have upgraded hardware and software specifications. The design of CE devices with face recognition to operate smart interactions will be of future prospect in residential environment. Therefore, although neither UPnP nor image recognition technology is not new to some extent, their combined effects have not yet inspected well in the literature. The work in this paper attempts to exploit the potential of combining both technologies for media distribution services in home-based IoT networks. III.

S YSTEM OVERVIEW

This section mentions the system design in home-based IoT networks. Section III.A gives an overall system workplace in a network. Section III.B describes the software architecture of the proposed media distribution system. Section III.C specifies the operational procedures among system components. A. System Workplace Fig. 1 depicts a conceptual system workplace where a home network includes different types of UPnP-compatible HNDs like personal computers, laptop, mobile phone networkattached storage, networked TV, etc. s. These devices construct a LAN/WLAN in the household surrounding. Each UPnP device is firstly assigned an IP address by a DHCP server when it is turned on. Then a UPnP device can advertise its appearance, discover others, and control any neighboring devices in a network according to the UPnP AV specification [3] for home multimedia entertainment. In addition, UPnP devices can be equipped with camera modules to capture images in front of them. Any captured images will be transferred to family database and then compared with user profiles there. Thus, any registered user in front of a UPnP device can be identified. With the information of a user, UPnP devices can interact with the user and provide personal media services in a smart fashion. B. System Architecture

B. Image Recognition Image detection technologies are matured when they are successfully applied in many intelligent control applications,

The system architecture comprises four major software components, including image detection, media playing engine, control point, and family database.

UPnP devices

Home-based IoT network (UPnP network) Family database

Face Recognition

Media Distribution System

Fig. 1.

Fig. 2.

An overview of the system workplace.

Image Detection

Sensing the context like brightness, and recognizing facial features

Media Playing Engine

Operating media playing devices like PC and networked TV

Control Point

Connecting both image detection module and media player devices, and triggering media content delivery

Family Database

Storing user profiles, e.g., name, facial features, favorite media renderer devices

Functional components in the system architecture.

1) Image Detection: The system applies image detection module to detect the appearance of any users face in the screen, while the brightness in the surrounding is treated as well. This module adopts a fast face detection algorithm [9] with three steps, as described below. The fist step is integral image creation that uses a median to represent an image characteristic so that the image matrix can be obtained fast by nine 9 entities within an integral image. The use of loose classifiers and Adaboost algorithm in the second step can make up a strong classifier from meaningful some from many loose classifiers. The final step distills key characteristics of a face image from a cascade of classifiers obtained in the second step. In addition to face detection, face recognition is required to figure out whom is the user in front of the screen. Although there are some classical face recognition techniques [10][11], they may be unsuitable to be used in embedded devices due to high computation cost. Thus, since the number of members in a family is small, it is possible to match a captured image with registered face images by a statistical approach, such as Bhattacharyya distance. For resource saving, a statistical approach with family database is able to fulfill the face recognition in this study. 2) Media Playing Engine: This module is to wrap the internal media playing engine that is hardware-platform-dependent in usual cases, and provides flexible functions and APIs for

upper applications and GUI customization. By this module, people can operate media playing devices, like PC and networked TV, in a networked way same as the ordinary way they are used to as operating standalone media playing devices. On the other hand, this module can gather the profiles of the media device and the information of local media content. Other functional components can communicate with a media device indirectly through this module. 3) UPnP Control Point: In compliance with UPnP device architecture [2], basically, there are three types of logical UPnP devices in this system: media server, media renderer and control point devices, where the last two are often combined on the same device, so-called media player. Every UPnP device performs the UPnP in six functional layers: IP addressing, discovery, description, control, eventing, and presentation, as listed in bottom-up order. Associated with the function of a UPnP control point (CP), a media device can find the existence of any neighboring UPnP devices in a network by the simple service discovery protocol (SSDP) that runs in a multicasting manner in a LAN/WLAN. A UPnP device can learn the services that other devices offer according to their device and service description files. Then, a UPnP device can access a service on the other device by simple object access protocol. When a UPnP device is operated and then changes its status, it can notify other devices of events upon the general event notification architecture (GENA). Finally, a presentation file of a device describes an HTML-based interface provided by devices for users to control or monitor them directly. For home multimedia services, the UPnP AV profile [3] specifies the interaction behavior and functional among any media server and renderer devices. A media server provides the content directory service, connection manager service, and AV transport service. The content directory service specifies a set of actions that control points invoke to enumerate media items. For example, the browse action allows control points to obtain file attributes, such as data formats and transfer protocols, of media items available on the media server. The connection manager service manages the connections associated with a device. A control point invokes a preparing-for-connection action to notify a media server to prepare for upcoming media transfer. This action returns an instance ID of AV transport service that the control points can use to control the transfer flow and connection. The AV transport service controls the player operations; for example, stop and pause. By contrast, a media renderer is instructed by a control point to control what and how media items are rendered, providing the rendering control service, connection manager service, and AV transport service. The rendering control service provides actions for a control point to adjust the rendering of media contents; for example, brightness and volume. The connection manager service and AV transport service are similar to those of a media server 4) Family Database: A family database manages the profiles of registered users in a home network. Because a media device in a system must resolve the identity of a user for access control, a user must register at the family database before the user begins to use the proposed system. For every registrant, the registered information contains a name, a photo for face detection and recognition, a preferred media device, a set of favorite media objects, and etc. Note that a preferred media

device implies a particular zone in a residential area that a user frequently visits or stays for a long time in daily life. Accordingly, a media device can ask the family database when it captures an image of a user in front of its screen. In addition, with user profiles, the system is able to provide personal media services among media devices in a network. C. Functional Procedures The system processes the intelligent media distribution services in four phases: user registration, image detection mode, face recognition, and personal media services, which are based on a sequence of functional procedures among three system components, i.e., image detection module, control point module and family database.

Face Recognition

UPnP Control Point

User Registration

User Profile

take a photo return a photo image choose a preferred device ok

image detection detect images

face recognition

discover neighboring devices

capture an image fetch image objects

return image objects and associated profiles

To ease comprehension, Fig. 3 shows the action flows in the proposed system, as described below. The system initially enters the image detection phase that executes continually to sense image motion on the screen of this media device. With every captured image frame, this image detection module uses the fast Adaboost algorithm to calculate the probability of any human face symbol statistics within an image frame. Once the calculation result shows that the face is found in the image, this module will compare the image data with the photo data stored in the family database. If the face is recognized as a registered user, the name of this user will be prompted on the screen for the users confirmation. Elsewise, if the face is new to the system, the system can launch a registration phase that instructs the user to finish a registration process and customize personal media services such as a preferred media device and media objects by default. The control point continues to check multicasting packets to learn the existence of any neighboring UPnP devices, and thus keeps a list of active UPnP devices in a home network. According to UPnP AV profiles, active UPnP devices are divided into two groups of media server and media renderer. When a user is recognized by the face recognition phase, the system is able to launch the media personal services corresponding to the profile of this user. Particularly, a preferred media device, i.e., media renderer or player device, is looked up from the list of active UPnP devices. Then, the system gets the state of this device and prepares to invoke the AV transport service on this device. Herein, regarding the media playing service, two usage cases are considered. •

•

First, as this user just comes in the residential area or begins to operate the system, the system will ask this device to play the users preferred media object, if it is in an idle mode. Second, as this device is playing some media object, the system will ask it to transfer the playing job to the other media device that is nearby the users current location, that is, the one capturing the appearance of this user. Thus, the control point will negotiate with the job transfer of continuing media playing with the target device. The target device will proceed to play the media object via a transport URI indicated by the control point.

Note that the media playing service is seamless despite the location change of this user in a residential area. This seamless

similarity calculation successful match personal service offer this user's service

access the personal setting of this user return a preferred device

obtain this user's preferred media object, and then show the media content on the preferred device or transfer the media content onto the device close to the user

Fig. 3.

Functional procedures.

media streaming effect is based on special customizations on the media playing engine of this system. IV.

D EVELOPMENT & D EMONSTRATION

This section describes the prototypical development of the proposed intelligent media distribution system. The development results in a software implementation that can be deployed on several HNDs in an experimental home-based IoT environment. The demonstration exhibits the intelligent interaction between a user and the media device to perform ubiquitous and personal media content services in a home computing environment. A. Prototypical Development The prototypical work develops several software modules, including UPnP protocol stacks, face recognition and algorithms, and family database on Android-based embedded systems. Thus, target runtime executions could be the sorts of mobile phones and IP set-top-boxs, capable of networking, multimedia processing, extensible storage, etc. This development adopts two open-source resources, Cling Java/Android UPnP micro-stacks [12] and OpenCV (open source computer vision library) [13]. The former is a set of Java-based program packets that offers a mini-core UPnP software architecture with six functional layers as mentioned in Section III.B.3. Based on this architecture, some media servers, media renderers, and a control point are implemented and deployed on the network in compliance with UPnP AV specifications. These devices provide content directory service, connection manager

bedroom

study room bedroom

Living room

(a)

dining room

Fig. 4.

(b)

Fig. 5. Registration: (a) creating a photo ID, and (b) selecting a preferred media device.

A household environment with several zones.

1 2

service, and AV transport service for medial content sharing in a home network. The latter is a famous computer vision library that is mainly implemented in C language with some C++ extensions. The recent progress also supports Java and Android platforms. In this development, OpenCV is used to compute image detection and face recognition, together with additional algorithms to fasten the computation B. Demonstration of Results This subsection first presents an ideal playground in a household environment, and then mentions a scenario for the demonstration of intelligent media distribution in home-based IoT networks. 1) Playground & Scenario: The demonstration is realized in an experimental household environment, comprising several zones such as living room, dinning room, study room, and bedrooms, as the drawing in Fig. 4. For a scenario exhibition, let a family member, Bob, first watch a video file on his laptop in a bedroom. Bob likes to see the video content in a bigger screen on TV in the living room. He walks to the living room and turns on a networked TV. As the associated camera module on this TV detects and recognizes his appearance in front of the TV, the playing video on the laptop is immediately streamed to the TV and displayed on the TV screen. Bob then sits on the sofa and continues to enjoy video in a comfortable manner. 2) Snapshots of Real Demonstration: Fig. 5 illustrates an initial phase of user registration before a user can operate the intelligent system in a residential area. When the system fails to discern a users identity, it will prompt a friendly GUI for a new member to do registration. The basic registration information contains a users photo, name, preferred media device, and favorite media object by default. Fig. 5(a) shows the front-end interaction interface between a user and the backend system. The system activates the digital camera associated with the media device in front of the user. The image detection module continuously scans the scene in front of the device, and analyzes the image symbols of captured image frame. If the symbol characteristic indicates the appearance of a human face, this face is marked by a green square on the screen. This square also helps the user know what picture will be actually

Fig. 6.

Media distribution and playing in a network.

stored and considered as a particular photo ID. When the user decides to create a photo ID as shown, the only thing to do is to touch the square on the screen of the device like a mobile phone, or input the confirmation signal of the device like a networked TV. A user name can be given to link with a photo ID. Finally, as Fig. 5(b) shows, a list of media renderer devices in a home network is prompted for a newly registered user to select its preferred media device. Later, as soon as this user is captured by any device in a house, the system will fetch its profile from the family database. The users favorite media object will be played by the preferred media device. Further, when the user moves around different zones in a house, the media object will be distributed to and shown onto another media renderer close to the user. For better presentation, Fig. 6 shows a real case where the user, Bob, originally watches a video file using his laptop, and then moves to another zone. As soon as Bob enters the zone space, the camera on top of the networked TV captures his image. Then, his face image is recognized by the system. With Bobs profile from the family database, the system learns that a video is playing on his laptop. So, the system commands the visited media renderer, i.e., the networked TV to play the remaining part of the video file. Accordingly, the UPnP control point is activated to communicate with the original device and obtain an URI of the playing media object. Then, it gives the URI to the visited device. These two devices thus negotiate the media transfer and playing service to fulfill the seamless media streaming over the home network. V.

C ONCLUSION

The study in this paper achieves an intelligent media distribution system in a home-based IoT network. According to the standard specifications of UPnP device architecture and UPnP AV profiles, UPnP-compatible HNDs in a network are able to

discover each other in proximity and then control the other parties in a distributed manner. In addition, UPnP-compatible HNDs associated with face recognition are able to sense the appearance of operating users and correspondingly activate personal media services in a smart network environment. Therefore, the deployment of the proposed system in a home-based IoT network enables a family member to distribute and share media content across UPnP-compatible HNDs that are placed in different zones inside a household environment. The prototypical development results in a proof-of-concept software implementation that can run on several types of experimental HNDs in a network. Real demonstration exhibits the effects of the proposed design, providing novel and interesting media playing services for home multimedia entertainment. ACKNOWLEDGMENT This work was supported in partial under the project contact no. B5-10223-2H-1 by Industrial Technology Research Institute of Taiwan. R EFERENCES [1] [2] [3] [4]

[5]

[6]

[7]

[8]

[9] [10]

[11]

[12] [13]

L. Harte, Home Media Networks: Systems, Technologies, and Operation, 1st ed. Althos Publisher, USA, 2011. UPnP Forum, “UPnP device architecture 1.1 version,” October 2008. ——, “UPnP av architecture:2 version: 1.0,” December 2010. C.-L. Hu, W.-S. Liao, and Y.-J. Huang, “Mobile media content sharing in UPnP-based home network environment,” Journal of Information Science and Engineering, vol. 24, no. 6, pp. 1753–1769, November 2008. C.-L. Hu, B.-J. Hsieh, H.-C. Lin, and Y.-F. Hsu, “Integrated home gateway design for enabling P2P content sharing in UPnP-based home networks,” in Proceedings of the 29th IEEE International Conference on Consumer Electronics (IEEE ICCE’11), January 2011, pp. 229–230. C.-L. Hu, C.-L. Lin, P.-C. Lin, and Y.-Y. Su, “Gesture-assisted remote control for media content sharing and distribution in home networks,” in Proceedings of the 17th IEEE International Symposium on Consumer Electronics (IEEE ISCE’13), June 2013, pp. 87–88. H. K. Ekenel, H. Gao, J. Stallkamp, M. Fischer, and R. Stiefelhagen, “Face recognition for smart interactions,” in Proceedings of IEEE ICME’07), July 2007, pp. 1007–1010. P. Belimpasakis and R. Walsh, “A combined mixed reality and networked home approach to improving user interaction with consumer electronics,” IEEE Transactions on Consumer Electronics, vol. 57, no. 1, pp. 139–144, February 2011. P. Viola and M. Jones, “Robust real-time face detection,” International Journal on Computer Vision, vol. 57, no. 2, pp. 137–154, May 2004. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigenfaces vs. fisherfaces: recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711–720, July 1997. M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’91), June 1991, pp. 586–591. 4th Line GmbH, “Cling java/android upnp library and tools,” Open source resource, online available via http://4thline.org/. OpenCV, “Open source computer vision library,” Open source resource, online available via http://opencv.org.