International Journal of Advanced Culture Technology Vol.5 No.1 51-57 (2017) https://doi.org /10.17703/IJACT.2017.5.1.51
IJACT 17-1-7
Design of Low Cost Real-Time Audience Adaptive Digital Signage using Haar Cascade Facial Measures Dongwoo Lee1, Daehyun Kim1, Junghoon Lee2, Seungyoun Lee2, Hyunsuk Hwang3, Vinayagam Mariappan4, Minwoo Lee4†, Jaesang Cha4 1
NAMUGA Co., Ltd, Seongnam, Korea {dwlee, ginkokim}@namuga.co.kr 2 Dept. of Electrical Information Control, Dongseoul College, Seongnam, Korea {jhlee, alyssa}@dsc.ac.kr 3 Dept. of Electrical Engineering, Seoil University, Seoul, Korea
[email protected] 4 Graduate School of Nano IT Design Fusion, Seoul National Univ. of Science & Tech., Seoul, Korea
[email protected], †
[email protected],
[email protected]
Abstract Digital signage is becoming part of daily life across a wide range of visual advertisements segments market used in stations, hotels, retail stores, hotels, etc. The current digital signage system used in market is generally works on limited user interactivity with static contents. In this paper, a new approach is proposed using computer vision based dynamic audience adaptive cost-effective digital signage system. The proposed design uses the Camera attached Raspberry Pi Open source platform to employ the real-time audience interaction using computer vision algorithms to extract facial features of the audience. The real-time facial features are extracted using Haar Cascade algorithm which are used for audience gender specific rendering of dynamic digital signage content. The audience facial characterization using Haar Cascade is evaluated on the FERET database with 95% accuracy for gender classification. The proposed system, developed and evaluated with male and female audiences in real-life environments camera embedded raspberry pi with good level of accuracy. Keywords: Digital Signage, RPi-CAM, Face Detection, Haar Cascade, HTML5, Audience Adaptive, Interactive Device, Media Content Server.
1. Introduction Digital signage is an integrated form of computing device and electronic display that shows multimedia content in public places for informational or advertising purposes. Recent days such advertisement services in public spaces has been changing from fixed or mechanical signage to digital signage. Indeed, public, semi-public and private spaces such as shops, hotels, streets, airports, offices, homes are increasingly relying on digital signs for disseminating information in day to day activities. Manuscript received: Jan. 16, 2017 / Revised: Feb. 5, 2017 / Accepted: Feb. 17, 2017 Corresponding Author:
[email protected] Tel:+82-02-970-6431, Fax: +82-02-974-6123 Graduate School of Nano IT Design Fusion, Seoul National Univ. of Science & Tech., Seoul, Korea
52
International Journal of Advanced Culture Technology Vol.5 No.1 51-57 (2017)
The flat-panel displays are emerging as a new, efficient way for providing optimized information-and-appearance attractive multimedia content on digital signage system [1, 2]. However, the usage of digital signage media content has been gradually changing from static content to dynamically adaptive content by audience interaction in the global digital signage industry as a bi-directional service. The user interaction can be in the form of buttons, remote control, sensors, touchscreens, smartphone based control, camera etc. This paper propose a novel interactive digital signage system by oversee and interact with the presence, activity and characteristics of the audience using state-of-the-art computer vision. The proposed approach, the digital images captured by the camera and processed with the digital signage software in real-time, extracting temporal, spatial and demographic features of the audience. By comparing the determined features with the predefined content descriptors, the display software automatically selects and broadcasts content relevant to the specific detected audience, for example, this could be information that is targeted to adult males or female. The proposed system is implemented using OpenCV library [3] on Raspberry Pi open source hardware platform built-in with camera connected with 24ʹ computer display.
2. Related Work The digital signage displays are advantageous compared to earlier static signs based signage due to the nature of display varying multimedia content such as images, video, audio and animations as shown in Figure 1. Recent days, a large majority of the digital signage applications are interfaces to advertising, internal or public or information, brand building the customer’s behaviours by enhancing the customer’s real-time experience [4].
Figure 1. Digital Signage Displays Pervading in Public Spaces The digital signage system media display content can be changed in real-time, which, in principle, allows for full context and audience adaptation [5]. The modern research of digital signage are actively exploring mechanisms for engaging audience interactive content display on signage system display [6, 7]. There are different audience interaction modalities have been proposed in digital signage system, including buttons, IR sensors, speech, facial expression, touch screen, smartphones, and hand gestures [8]. However, the high potential of digital signage displays has not yet been fully exploited, as the displayed content is most often generic and uninteresting for audience, causing the display blindness [9] effect. To make digital signage a more effective information interface, the displayed content should be informative, dynamic and attractive [10].
3. Interactive Digital Signage Most of the digital signage systems are designed to only broadcast advertising the media content according to used ventures business domain. However, audience interaction makes a digital signage far more attractive and
Design of Low Cost Real-Time RPi-CAM Based Audience Adaptive Digital Signage using HAAR Cascade Facial Measures
53
influencing content to the audience. So audience interactivity become one aspect of digital signage fundamental design characteristics. Now a days, Interactive Digital Signage (IDS) have been widely used in digital advertisements or general communications like in shopping mall, hospitals, conference halls, train stations, etc. The digital signage user interaction techniques include buttons, keypads, touchscreens, cameras, smartphones, and gesture interaction with depth cameras, etc. The touch screen based interactive signage system shown in Figure 2. The very earliest form of interactive signage systems employed a console of buttons which is labeled and dedicated to particular functions.
Figure 2. Interactive Signage Using Touch Some interactive signage displays model allows user to browse through the media content by pressing buttons located on signage or by driven by PC keyboard nearby the signage to provide a full-featured and flexible interface for media content access. The touchscreen use has experienced a resurgence in popularity as a result of the success of mobile devices that use this interactive interface. As an interesting consequence of touchscreen popularity is that most of the people who see non-interactive digital signage are touching the screens and expecting a reaction in the content access. Even for signage system that have a traditional touch capability, users are likely to try use the multi-touch gestures to zoom in and out on the display. Another signage system interaction technique to use of a smartphone to control the media content. The smartphone based media content management used the Bluetooth or other short-range wireless technologies to interact with signage system and the smartphone become a remote control for the display, presenting the control interface on signage screen. The smartphone based signage control by user is an effective method of low-latency interaction due to the short-range wireless communication minimizes delays between user and signage interaction. The system designers of interactive signage have also trying to use conventional digital cameras to enable automatic adaptation of digital content to the audience by audience interaction based on the audience’s average height, , the length of a person’s hair and facial features, etc. to display the audience specific digital content.
4. Audience Adaptive Digital Signage Design The audience adaptive digital signage system is characterized by its provision of user specific advertisement awareness, large and clearly visible displays based on user interaction with signage system. Therefore, the design of audience adaptive digital signage system involves the hardware cost as well as the cost of software solutions, especially copyright limitation to use algorithms. These considerations were, the main motivation to use Raspberry Pi based open source hardware platform with OpenCV algorithm for propose the system design. This propose a low cost real-time RPi-CAM based audience adaptive digital signage system using facial features measures of the audience. In this approach, the digital images captured by the camera then extract the spatial, temporal, and demographic facial features of the audience with the digital signage software in real-time.
54
International Journal of Advanced Culture Technology Vol.5 No.1 51-57 (2017)
The extracted facial features are compared with the predefined content descriptors features, and then the signage display software automatically selects and broadcasts specific content relevant to the specific detected audience type. For example, in this paper targeted to male or female audience specific content delivery according to the male or female audience observation. This camera built-in on the designed system for the interaction with the observers the user activity frequently as it is already built into the frame of digital signage displays as shown in Figure 3. Along the defined hardware frame work, applied reliable state-of-the-art computer vision solutions on the camera observed audience image, optimizing with advanced computer vision functions, real-time processing speed and ease of integration, as well as their license free implementation.
Figure 3. Audience Adaptive Digital Signage System In this paper proposed approach, there are several computer vision methods are combined to achieve the optimal determination of the features of the audience to display audience adaptive content on the signage display. The algorithm flow model is shown in Figure 4.
Figure 4. Audience Adaptive Digital Signage System Algorithm Flow In this paper, background segmentation designed with Mixture-of-Gaussians based background modelling [11] to extract interested facial foreground regions and define the possible presence of audience. The Haar Cascade face detection algorithm [12] is applied on the background segmented image, to distinguish true audience who facing the display. The Haar Cascade frontal face detector determine the location of the detected faces of the
Design of Low Cost Real-Time RPi-CAM Based Audience Adaptive Digital Signage using HAAR Cascade Facial Measures
55
audience down to the size of 20x20 pixels per face in real-time, regardless of their actual position and physical scale in captured image. The positively identified audience are tracked using a template algorithm provided in OpenCV library for real-time video processing. The detected facial images are registered and extracted the 66 facial feature points to obtain demographic features like gender, age group etc., using the OpenCV Library. The proposed design specifically, the face registration extracts and determines the position of 66 facial feature points in face, where for an example, each audience eye is described with 6 feature points that form a convex polygon around the eye orbit of the audience. The centroid of this extracted feature polygon is calculated in order to determine the centre of the audience eye. The gender of the audience classified using SVM classifier with use of 66 facial feature points and classified with SVM classifier. The audience specific adaptive content displayed on signage display based on the classified gender using audience face value measures. To evaluate the face detection and recognition efficiency FERET data base is used in this paper.
5. Real-Time Emulation and Results Analysis To evaluate the proposed low-cost audience adaptive digital signage, the signage system designed with Raspberry Pi 2 built-in with Pi CAM and connected to monitors with HDMI port. The Raspberry Pi open source based signage system design main components are shown Figure 5. The Raspberry Pi 2 will be packaged as a digital signage box that will act as audience adaptive digital signage system.
Figure 5. Raspberry Pi 2 Model B with Pi Camera Module The proposed digital signage software designed with OpenCV computer vision library using C++ and content display on signage display implemented using HTML5. The emulated signage system demonstrated model is shown in Figure 6. The audience adaptive content display on signage system is evaluated by signage content display based on male or female audience gender classification using demographic features of the audience.
Figure 6. Audience Adaptive Digital Signage Emulated Result
56
International Journal of Advanced Culture Technology Vol.5 No.1 51-57 (2017)
The demographic features of the audience from the aligned facial images, was evaluated on the FERET database and emulated on Raspberry Pi based signage system in real-time. The FERET database includes annotated facial images with corresponding gender of 856 individuals. The extracted aligned frontal facial feature vectors were used as a gender classification input for SVM based machine learning algorithms. The highest gender classification accuracy was achieved using SVM which resulting in a classification accuracy of 95.2%. The real-time audience adaptive content broadcasting was tested in a laboratory environment with a limited no of selected audience in a more controlled environment on this paper result analysis. The main contribution of this proposed paper is a real-time, low cost interactive and audience adaptive signage system where the viewing statistics of the audience interaction using efficient and real-time computer vision algorithm on cost effective open source hardware platform where the system developer do not have any complexity to build the system. This paper hopes that the proposed design contribute to the future development and design of intelligent audience adaptive digital signage systems. The use of more complex algorithms could improve audience tracking and classification accuracy but would at the same time require to select the hardware platform with more processing time or more processing power. The priceperformance trade-off will probably be determined only after more experience is gained in design of future interactive and audience adaptive signage systems.
6. Conclusion This paper described the design and implementation of an audience adaptive digital signage system using facial features of audience determined by computer vision methods provided by OpenCV library. The cost-effective signage system emulation model is designed using Raspberry Pi based open source with computer vision algorithm and performance are evaluated using FERET data set as well real time audience based adaptive content display on signage display. The performance of proposed system is assessed in a real world environment in the context of a marketing statistic of the digital signage system. The license-free computer vision and machine learning algorithms on low-price hardware that operate in real-time were selected in this proposed emulation of signage system.
Acknowledgement This research was supported by a grant from the Fundamental R&D program for Technology of Material & Components funded by the Ministry of Trade, Industry and Energy, Republic of Korea.
References [1] Krumm J, “Ubiquitous advertising: The killer application for the 21st century”, IEEE Pervasive Computing 10:6673, 2011. [2] Davies N, Langheinrich M, Jose R, Schmidt A, “Open Display Networks: A Communications Medium for the 21st Century”, IEEE Computer 45(5): 58-64, 2012. [3] Viola PA, Jones MJ, “Robust real-time face detection”, International Journal of Computer Vision 57(2):137-154, 2004. [4] Lundstrom LI, “Digital Signage Broadcasting: Content Management and Distribution Techniques”, Focal Press, 2008. [5] Bauer C, Spiekermann S, “Conceptualizing Context for Pervasive Advertising”, Pervasive Advertising, Springer. pp.159-183, 2011.
Design of Low Cost Real-Time RPi-CAM Based Audience Adaptive Digital Signage using HAAR Cascade Facial Measures
57
[6] Michelis D, Müller J, “The Audience Funnel: Observations of Gesture Based Interaction with Multiple Large Displays in a City Center”, Computer Interaction 27(6): 562-579, 2011. [7] Müller J, Walter R, Bailly G, Nischt M, Alt F, “Looking glass: a field study on noticing interactivity of a shop window”, Conference on Human Factors in Computing Systems. ACM. pp. 297-306, 2012. [8] Müller J, Alt F, Michelis D, Schmidt A, “Requirements and design space for interactive public displays”, ACM Multimedia, ACM, pp. 1285-1294. [9] Müller J, Wilmsmann D, Exeler J, Buzeck M, Schmidt A, Jay T, “Display Blindness: The Effect of Expectations on Attention towards Digital Signage”, Pervasive, LNCS 5538, Springer. pp. 1-8, 2009. [10] Huang EM, Koster A, Borchers J, “Overcoming assumptions and uncovering practices: When does the public really look at public displays”, Pervasive, LNCS 5013, Springer. pp.228-243, 2008. [11] Dodgson NA, “Variation and extrema of human interpupillary distance”, Proceedings of SPIE: Stereoscopic Displays and Virtual Reality Systems XI. Vol. 5291. San Jose. pp. 36-46, 2004. [12] Bouwmans T, Baf FE, Vachon B, “Background Modeling using Mixture of Gaussians for Foreground Detection”, A Survey. Recent Patents on Computer Science 1 (3):219-237, 2008.
