QoS-oriented Service Management in Clouds for Large Scale Industrial Activity Recognition Athanasios S. Voulodimos1, Dimosthenis P. Kyriazis1, Spyridon V. Gogouvitis1, Anastasios D. Doulamis2, Dimitrios I. Kosmopoulos3, Theodora A. Varvarigou1 1
National Technical University of Athens
[email protected];
[email protected];
[email protected];
[email protected] 2
Technical University of Crete
[email protected]
3
University of Texas at Arlington
[email protected]
Abstract— Motivated by the need of industrial enterprises for supervision services for quality, security and safety guarantee, we have developed an Activity Recognition Framework based on computer vision and machine learning tools, attaining good recognition rates. However, the deployment of multiple cameras to exploit redundancies, the large training set requirements of our time series classification models, as well as general resource limitations together with the emphasis on real-time performance, pose significant challenges and lead us to consider a decentralized approach. We thus adapt our application to a new and innovative real-time enabled framework for service-based infrastructures, which has developed QoS-oriented Service Management mechanisms in order to allow cloud environments to facilitate real-time and interactivity. Deploying the Activity Recognition Framework in a cloud infrastructure can therefore enable it for large scale industrial environments. Keywords- activity recognition, industrial workflows, service management, cloud infrastructure, QoS.
I. INTRODUCTION Large scale enterprises (like industrial plants or public infrastructure organizations) have a clear need for supervision services in order to guarantee: (a) quality adherence to predefined procedures for production or services, (b) security and safety - prevention of actions that may lead to hazardous situations, (c) production scheduling - allocation of a number of available production resources (raw materials, equipment, utilities, manpower) to tasks over a scheduling horizon. To this end, significant research has been carried out in the fields of computer vision and machine learning regarding camera localization, scene reconstruction, object detection, moving entities tracking and action/activity recognition. However, the vast majority of these algorithms are centralized. Recently, the development of distributed computer vision algorithms (a survey as of 2009 can be found in [1]) promises to advance the state of the art in computer vision systems by improving their efficiency and scalability, as well as their robustness to outliers and node
failures [2]. Nevertheless, the employment of state of the art centralized computer vision in distributed systems so as to exploit the advantages offered by the latter is not straightforward, because of the existence of a number of challenges. In many applications, camera sensor networks are constrained by severe network capacity and energy constraints. On the other hand, existing computer vision algorithms assume that the computation is centralized at a main processor, whereas collecting all the raw data at a single point is impractical or even impossible because of resource constraints. Besides, many computer vision tasks cannot be performed in a low power computing platform [2]. On the other hand, cloud computing offers the potential to dramatically reduce the cost of software services through the commoditization of IT assets and on-demand usage patterns. Virtualization of hardware, rapid service provisioning, scalability, elasticity, accounting granularity and cost allocation models allow clouds to promise the ability to efficiently adapt resource provisioning to the dynamic demands of Internet users. In this context, the services research community has been providing various outcomes to overcome limitations and address new challenges. Future Internet applications raise the need for environments that can facilitate real-time and interactivity and thus pose specific requirements to the underlying infrastructure, which should be able to efficiently adapt resource provisioning to the dynamic Quality of Service (QoS) demands of such applications [14], [15]. In this paper we describe an experiment of deploying a successful centralized computer vision and machine learning activity recognition tool on a federated cloud architecture, so as to scrutinize its effectiveness on an industrial large scale. To our knowledge the work regarding the use of workflow management mechanisms in industrial environments is limited: e.g. [12] proposes the use of agent-based workflow management mechanisms in industrial automation. However, the application described therein is different, as in this work equipments and smart objects are wrapped as agents and exposed as web services that contain real-time status information, which can then be used to form a workflow
that describes a manufacturing process. Besides, our experiment also aims at evaluating, validating and optimizing QoS-Oriented Service Management mechanisms in Large Scale Federated Clouds [3] implemented in the framework of the EU IRMOS project [8], which are being extended in the context of the VISION Cloud project [16]. The remainder of the paper is structured as follows: Section II briefly describes the industrial activity recognition framework and highlights the challenges raised that lead us to consider a distributed approach. In Section III the Service Management Mechanisms of the Cloud federation are presented, while highlighting the interdependencies between the components of the architecture that supports the computer vision real world application scenario under discussion. Finally, Section IV concludes the paper. II. INDUSTRIAL ACTIVITY RECOGNITION USING COMPUTER VISION AND MACHINE LEARNING Robust automated activity recognition and production line monitoring using visual sensors in industrial environments is a notoriously difficult problem in computer vision. Under challenging industrial conditions, it is very hard to acquire good quality data for achieving intelligent activity analysis tools. Nevertheless, it is important to address such problems by building generic quasi-performing automated workflow monitoring tools for industrial operation management purposes. This view will contribute in the improvement of process quality standards and health and safety in industrial manufacturing environments of the future. In the following we briefly describe our Activity Recognition Framework ([5], [6]), capable of achieving good recognition rates in real-life installations, based on a holistic representation of the raw input data, and hidden Markov model (HMM)-based statistical pattern recognition methodologies. A. Activity Recognition Framework A flow diagram of the proposed visual activity recognition framework is presented in Figure 1. The first functional procedure of our framework is environment modeling / background subtraction, i.e., the creation of a model of what belongs to the scene, as opposed to the actors or moving objects entering and leaving the scene that are identified in the next step of motion segmentation. The moving objects are then identified by calculating their distance from the model. The next processing step is the extraction of features for the effective representation of the raw input data (followed by a dimensionality reduction step for the obtained feature vectors). The resulting cameraspecific information streams are input to classifiers capable of modeling and recognizing time series, i.e., Hidden Markov Models (HMMs). Gaussian mixture models are typically used for modeling the observation emission densities of the HMM hidden states. Moreover, different fusion approaches of multiple information streams (e.g., parallel HMM, multistream fused HMM [10]) have been
examined so as to exploit the complementarity of different views offered by multiple cameras, and thus solve occlusions and improve accuracy. The activity recognition framework based on computer vision and machine learning is described in detail in [5], [6], where also a thorough experimental evaluation of the proposed framework is provided, considering real-life visual behavior recognition scenarios in the context of the assembly lines of a major automobile manufacturer. The algorithms implemented use as input two real-world datasets recorded in an automobile construction industry. An example screenshot of the ARF tool on the industrial dataset is shown in Figure 2. Both datasets [9] (available at http://www.scovis.eu) describe complex industrial processes which have as a goal the assembly of a car in the factory. The recorded frames depict metal sparks, cars’ equipments racks, and workers performing the assembly, robotic movements, humanmachinery interaction and abnormal situations and events that should trigger alarms. The datasets include videos ranging from one day (with 20 working cycles) to three days activities (with much more complex content since simultaneous processes can be executed by the workers) and can thus pose different computing requirements to ARF. Camera 1
Camera k
Background subtraction
Background subtraction
Motion Segmentation
Motion Segmentation
Holistic Feature Extraction
Holistic Feature Extraction
Fusion of information from multiple cameras
Activity Recognition
Figure 1. Flow diagram of the Industrial Activity Recognition Framework.
B. Challenges The above described activity recognition framework achieves significant success rates as shown in [5], while we are constantly researching on ways to further enhance the recognition rates of the framework. However, deploying the Activity Recognition Framework in a large scale industrial plant in order to fully cover the production area raises some considerable limitations, mainly as far as resources are concerned. To begin with, the processes of background subtraction, motion segmentation and feature extraction are performed for every frame of every camera, and this should
be done in almost real-time, so that the framework can have the biggest possible industrial impact. Furthermore, the Hidden Markov Models employed for time series classification demand significant processing and memory requirements in the training phase, especially when it comes to such large datasets of complex cluttered industrial environments. The HMMs should be trained with long time series (i.e. video sequences of long duration), the model parameters should be stored and re-used in the testing phase for the remaining video frames. Additionally, fusion of different information streams stemming from multiple cameras is an important constituent element of the ARF, therefore the increase in number of available sensor cameras could lead to higher accuracy, but also significantly raises computation cost in all phases of the framework. What’s more, the requirement for robust real-time activity recognition is important for the system to have a significant industrial impact. In case that the developed framework is to be applied for large scale industrial environments, like for example the full sectors of the factory, the aforementioned limitations may lead to important obstacles, which could be overcome by deploying the ARF service in a Cloud infrastructure, thus also enabling it for large scale industrial environments.
Figure 2. Example of the outcome of the ARF tool on our challenging industrial datasets.
III. SERVICE MANAGEMENT MECHANISMS A. Service Management Mechanisms Description Based on the Cloud Service / Platform / Infrastructure (SPI) layered model [7], focus is put upon the Platform-asa-Service (PaaS) characteristics (i.e. Service Management
mechanisms) in order to support for QoS guarantees, given that various approaches on the IaaS layer (e.g. ISONI [13]) allow for provision of QoS guarantees (using mechanisms such as fault-tolerance, resiliency, temporal isolation, etc). A leading Internet of Services (IoS) project (EU IRMOS), has developed QoS-oriented Service Management mechanisms in order to allow Cloud environments to facilitate real-time and interactivity. Here we present an approach on how to engineer the application services described in Section II to be executed in the specific cloud environment and evaluate their effectiveness when deployed for an application with strict timing requirements. We focus on Monitoring, Event Evaluation and Workflow Management using the computer vision application scenario. A brief description of the proposed Service Management mechanisms follows [4]: •
Workflow Management: Workflow management plays a significant role in the context of a cloud ecosystem with stringent time requirements [11]. Our solution consists of two components, namely Workflow Manager and Workflow Enactor. Since various control actions are needed in order to maintain the QoS level and the smooth operation of the application services, the Workflow Enactor Service is deployed within the Virtual Machines (VMs) to have direct access and control on the application service components. This service is responsible only for components of the particular VMs, while other instances are deployed in other VMs. All Workflow Enactor Services (instances) are controlled centrally by the Workflow Manager Service. The Workflow Enactor Service is responsible for configuring the application service components prior to their execution and managing the workflow during the execution of an application (i.e. start and stop services according to the workflow description document). Furthermore, the Workflow Manager receives notifications from the Event Evaluator regarding corrective actions which the Workflow Enactor Service invokes.
•
Monitoring: It consists of two components, namely Monitoring Manager and Monitoring Instance. The Monitoring Manager exposes appropriate interfaces for starting / stopping the operation and has access to the collected data from both the infrastructure and the application level. It orchestrates the Monitoring Instances of all VMs and has access to the aggregated information. Also, it can serve different and concurrent application service components that are being deployed in the same or different VMs. The Monitoring Instance is located within the VMs and is specific to every deployment. It exposes an interface towards the Monitoring Manager in order to initiate the application monitoring. Besides, the Monitoring Manager generates events based on the monitoring information which are being handled by the Event Evaluator.
•
Events Evaluation: It refers to the Event Evaluator, a component that receives events from the Monitoring Manager in order to trigger corrective actions that are being handled by the Workflow Manager (e.g. re-configuration of the application service components based on new resource allocation). It applies specific rules so as to realize the aforementioned actions.
B. Architecture Overview In the proposed experiment we have deployed up to one hundred fifty (150) VMs across different sites (the number of VMs being deployed varies according to the requirements of the application scenario in terms of real-time processing – based on the monitoring information, events are being generated that trigger the deployment of additional VMs, which in sequel are enacted by the workflow manager). As depicted in Figure 3, and based on the short description of the Service Management framework and the corresponding application, what is of major importance lies in the hierarchical architectural approach of the Monitoring and Workflow Management components, instances of which will reside in the VM images along with the application service components. Thus, scaling up to one hundred fifty (150) VMs affects not only the application but also the Service Management framework (answering the question “how scalable a service can be”), since the images will
contain the following services and application service components to be tested: • Workflow Enactor • Monitoring Instance • Application Service: ARF C. Component Interdependencies Figure 4 provides a graphical presentation of the components showing their interdependencies. In each VM image besides the application service component (ARF), the workflow enactor and monitor instance services will also be deployed. In a different VM (acting as the PaaS provider), the Workflow Manager, the Monitoring Manager and the Events Evaluator services will be deployed. The workflow manager initiates the workflow enactor, which in turn enables the monitoring instance, configures the application service components and invokes them. Monitoring information (application) is collected from each VM from the workflow instance and propagated to the monitoring manager, which along with the infrastructure monitoring information (obtained from the IaaS gateway) generates events that are being evaluated by the events evaluator. Based on specific policies (e.g. scale when a threshold is exceeded), the events evaluator triggers corrective actions to the workflow manager (e.g. re-configure the application component services according to updated allocated resources in order to meet the QoS requirements of the application).
Figure 3. Architecture Overview
Application Service: ARF
IaaS provider
Figure 4. Components’ Interdependencies
IV. CONCLUSION We have briefly described an Activity Recognition Framework based on computer vision and machine learning algorithms aimed at industrial environments supervision and automatic monitoring. The framework is roughly based on background subtraction, motion segmentation, holistic feature extraction, fusion of information streams from multiple cameras, and finally time series classification through HMMs for activity recognition. This centralized approach poses some resource and time related limitations, when attempting to enable it for a large scale enterprise. We therefore deploy it on the Cloud infrastructure implemented within the framework of the IRMOS EU project, which has developed QoS-oriented Service Management mechanisms, such as Workflow Management, Monitoring and Events Evaluation, which are being extended in the context of the VISION Cloud project. We describe the overall architecture as well as the components interdependencies, while the outcome of the experiment, apart from endowing the computer vision application framework with the advantages of the Cloud infrastructure, also involves evaluation, validation and optimization of the implemented Service Mechanisms. ACKNOWLEDGMENT The research leading to these results has received funding from the European Commission's Seventh Framework Programme ([FP7/2007-2013]) under grant agreement n° 257019 (VISION Cloud project). REFERENCES [1] R. Radke, “A survey of distributed computer vision algorithms,” in Handbook of Ambient Intelligence and Smart Environments. New York: Springer-Verlag, 2009, pp. 35–55.
[2] R. Tron, R. Vidal, "Distributed Computer Vision Algorithms," IEEE Signal Processing Magazine, vol.28, no.3, pp.32-45, 2011. [3] Spyridon V. Gogouvitis, Kleopatra Konstanteli, Dimosthenis Kyriazis, Theodora Varvarigou, "An Architectural Approach for Event-Based Execution Management in Service Oriented Infrastructures," 12th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), 2010. [4] S.V. Gogouvitis, K. Konstanteli, G. Kousiouris, G. Katsaros, D. Kyriazis, T. Varvarigou, "A Service Oriented Architecture for achieving QoSaware Workflow Management in Virtualized Environments," 2010 International Conference on Network and Service Management (CNSM), pp.398-401, 2010. [5] D. Kosmopoulos, S.P. Chatzis, "Robust Visual Behavior Recognition," IEEE Signal Processing Magazine, vol.27, no.5, pp.34-45, Sept. 2010. [6] D.I. Kosmopoulos, A.S. Voulodimos, T.A. Varvarigou, "Robust Human Behavior Modeling from Multiple Cameras," 20th International Conference on Pattern Recognition (ICPR), pp.35753578, 23-26 Aug. 2010, doi: 10.1109/ICPR.2010.872. [7] Peter Mell and Tim Grance, The NIST Definition of Cloud Computing, Version 15, 2009. [8] IRMOS Project, IRMOS Project website, http://www.irmosproject.eu. [9] A. Voulodimos, D. Kosmopoulos, G. Vassileiou, E. Sardis, A. Doulamis, V. Anagnostopoulos, C. Lalos, T. Varvarigou, “'A dataset for workflow recognition in industrial scenes”, IEEE ICIP 2011. [10] Z. Zeng, J. Tu, B. Pianfetti, & T. Huang, “Audiovisual a ective expression recognition through multistream fused hmm”, IEEE Transactions on Multimedia, vol. 10, no. 4, pp. 570-577, 2008. [11] E. Deelman, D. Gannon, M. Shields, I. Taylor, Workflows and eScience: An overview of workflow system features and capabilities, Future Generation Computer Systems, vol. 25, pp. 528–540, 2009. [12] Y. Zhang, G. Q. Huang, T. Qu, O. Ho, “Agent-based workflow management for RFID-enabled real-time reconfigurable manufacturing”, International Journal of Computer Integrated Manufacturing, vol. 23, pp. 101–112, 2010. [13] S. Narasimhamurthy, G. Umanesan, J. Morse, M. Muggeridge, “ISONI StorageWhitepaper”, January 2011. [14] L. Zeng, B. Benatallah, A. Ngu, M. Dumas, J. Kalagnanam, H. Chang, “QoS-aware middleware for Web services composition”, IEEE Trans. on Softw. Eng., vol. 30, no. 5, pp. 311 – 327, 2004. [15] N. Doulamis, P. Kokkinos, E. Varvarigos, “Spectral Clustering Scheduling Techniques for Tasks with Strict QoS Requirements',” EuroPar 2008. [16] VISION Cloud project, URL: http://www.visioncloud.eu.
