A Survey of Context-Aware Middleware Designs for ... - IEEE Xplore

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 24

CONTEXT-AWARE NETWORKING AND COMMUNICATIONS

A Survey of Context-Aware Middleware Designs for Human Activity Recognition Ozgur Yurur, Chi Harold Liu, and Wilfrido Moreno

ABSTRACT

Ozgur Yurur and Wilfrido Moreno are with the University of South Florida. Chi Harold Liu is with the Beijing Institute of Technology, and is the corresponding author for this article. This work was supported by the National Natural Science Foundation of China (No. 61300179).

24

The ever increasing technological advances in embedded systems engineering, together with the proliferation of small-size sensor designs and deployment, have enabled smart devices to recognize daily human-based actions, activities, and interactions. Therefore, inferring a vast variety of user-device-based activities from diverse contexts obtained by a series of sensory observations has drawn much interest to the research area of ubiquitous sensing. The existence and awareness of context provides the capability of being conscious of physical environments or situations around users, and this allows network services to respond proactively and intelligently based on such awareness. Hence, with the evolution of smartphones, software developers have been empowered to create context-aware applications for recognizing human-centric or communitybased innovative social and cognitive activities in any situation and at any location. However, the middleware services provided in mobile devices have limited resources in terms of power, memory, and bandwidth compared to the capabilities of PCs and servers. Also, power is a major restriction on implementation of context-aware applications. Mobile device batteries do not last a long time while operating sensor(s) constantly. To this end, this article extensively surveys the emerging concepts of context awareness in mobile platforms by providing up-to-date literature and future research directives. We also point out the challenges faced in this regard and enlighten them by offering possible solutions.

INTRODUCTION The evolution of smart devices together with increasing computational power has empowered developers to create context-aware applications for innovative social and cognitive activities in any situation and at any location. This leads to the exciting vision of forming a society of “smart spaces” or the Internet of Things (IoT) [1]. Hence, the key idea behind context-aware sensing applications is to encourage users to collect,

0163-6804/14/$25.00 © 2014 IEEE

analyze, and share much local sensory knowledge for large-scale community use by creating a knowledge network capable of making autonomous logical decisions to actuate environmental objects and also to assist individuals. The integration of sensing and advanced computing capability in mobile devices for utilization within the concept of personal or participatory sensing will produce sensory data and exchange information among local or system-wide resources by feeding the Internet on a social scale. This situation will emerge as the IoT [2], shifting into a collection of autonomous, ambient, intelligent, and self-operated network nodes (e.g., independently acting smartphones) that are well aware of surrounding contexts, circumstances, and environments. With these capabilities, the new network architecture will enhance data credibility, quality, privacy, and shareability by encouraging participation at the personal, social, and urban levels, and lead to knowledge discovery on human lives and behaviors, and environment interactions/social connections by leveraging the deployment capacity of smart things (e.g., smartphones, tablets) in order to collect and analyze the digital traces left by users. Human behavior highly depends on the perception, context, environment, and prior knowledge of most recent event patterns. The understanding of human activities is based on the discovery of an activity pattern and accurate recognition of the activity itself. Therefore, researchers have focused on implementing computational pervasive systems to create high-level conceptual models to infer activities and lowlevel sensory models to extract contexts from unknown activity patterns. The creation of a generic model to represent the true nature of human behavior stands as a major challenge. In this aspect, the construction of a framework by distinct middleware technologies has been put forth to provide a required model for recognition of daily human activities via observations acquired by various built-in smartphone sensors. These activities are inferred as the outcome of a wide range of sensory applications utilized in diverse implementation areas ranging from envi-

IEEE Communications Magazine • June 2014

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 25

Context-aware application Requests

TM

Meta-sensory data

Acquires

discovering the best characteristics of the target complex

Context-aware middleware

GPS

The key goal lies in

spatial phenomenon being sensed,

Sensory data Physical world

meeting the demands of the application, and satisfying the

Figure 1. It is believed that introducing intelligence and situational awareness into the recognition process of human-centric event patterns could give a better understanding of human behaviors, and also offer a chance to proactively assist individuals in order to enhance the quality of lives. ronmental surveillance and assisting technologies for medical diagnosis/treatments to the creation of smart spaces for individual behavior modeling. The key challenge faced by this concept is to infer relevant activities in a system that takes raw sensor readings initially and processes them until obtaining a semantic outcome under some constrictions. These constrictions mostly stem from the difficulty of shaping exact topological structure, and also from modeling uncertainties in the observed data due to saving energy wasted during physical sensor operations and the processing of sensory data. Most sensors currently available on smart devices are designed to perform some specific applications, such as accelerometers for detecting the screen orientation, microphones for voice conversations, cameras for capturing images, and GPS for displaying locations. However, by introducing intelligence, situational awareness, and context recognition into these devices, and given the right architecture for ubiquitous sensing by enhancing and systematizing the existing methodologies, built-in sensors could be repurposed and act as sensor nodes in order to proactively assist users in their daily activities by increasing the quantity, quality, and credibility of community-gathered data. Hence, these smart devices could be used as instruments (Fig. 1) to collect data and provide meaningful observations of user behaviors and surrounding environments, such as activity measurement by an accelerometer, ambient sound environment sensing by a microphone, and estimation of the time and location a user spends indoors and outdoors by GPS. In addition, external sensors, such as biomedical sensors (e.g., ECG, BVP, GSR, and EMG), can also be deployed with a wearable strap on human bodies. Information obtained from different sensors can be crosslinked and presented as new valuable input. For instance, the GPS and accelerometer actualize geographic information systems (GISs), potentially providing insight as to how the proximity of recreational facilities affects physical activity levels, or how the relative accessibility of grocery stores and fast food restaurants influence a diet program. WiFi can be leveraged to determine relative proximity of individuals to each other or fixed locations, and might be used for a study to

IEEE Communications Magazine • June 2014

constraints on sensor usage.

examine the spread of an infectious disease. Bluetooth, as well as ZigBee, can be used for ambulatory data collection of more traditional signals, such as blood pressure, heart rate, respiration, and blood glucose level. In the exciting development of context-aware applications, middleware systems/services in smart devices, however, only have very limited resources in terms of power, memory, and bandwidth compared to the capabilities of PCs and servers. Energy efficiency in particular is a major restriction imposed on context-aware application development, since the extraction and inference of user-relevant sensory data requires continuous sensor operations. This requirement unfortunately shortens the device battery lifetime due to the high energy consumption required by both sensor and processor operations. One solution is to take precautions on sensory operations by putting them into sleeping mode to reduce power consumption. However, it turns into an accuracy problem the middleware services may produce while providing information to applications. This situation triggers the research topic of finding optimal solutions to balance the trade-off between power consumption and sensory data accuracy in the operation of applications. Hence, the key goal lies in discovering the best characteristics of the target complex spatial phenomenon being sensed, meeting the demands of the application, and satisfying the constraints on sensor usage. The rest of the article is organized as follows. First, we introduce context awareness in smartphone sensing, including the definition of context, context presentation, and the stages of context inference. We then broaden context awareness in human activity recognition by giving application examples. We describe related research activities on context-aware middleware designs and their key properties, such as transparency, adaptability, and reflectiveness. Next, we present a variety of different context inference algorithms, and their merits and disadvantages for human activity recognition. Then we discuss the challenges that are faced during the middleware design process and system integration, and evaluate possible solutions and future directions. Finally, we conclude the article.

25

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 26

With the development and deployment of new sensor technologies into mobile devices, these devices will gain environmental intelligence, thereby providing the capability to sense, reason, and actuate the physical world.

CONTEXT-AWARE SMARTPHONE SENSING Many research activities have exploited the topic of context-aware ubiquitous sensing, which intends to use the capability of embedded computational power in mobile devices to infer the context obtained through physically distributed communication devices or sensors. The contextawareness and use of this context on a(n) individual or community scale is the most significant role of the ubiquitous/mobile computing scenarios. Context-aware systems aim at using a mobile device (e.g., a handheld smartphone or attached/ wearable device) integrated with smart sensors in order to monitor and measure individual or environmental phenomena for the purpose of assisting or evaluating human lives to achieve desirable quality of living standards. The ubiquity of mobile smart devices and proliferation of wireless networks will allow everyone permanent access to the Internet at all times and all places. With the development and deployment of new sensor technologies into mobile devices, these devices will gain environmental intelligence, thereby providing the capability to sense, reason, and actuate the physical world. In the real world, being aware of context and communicating is a key part of human interaction. A context is defined as a data source that can be sensed and used to characterize the situation of an entity. In other words, the context describes a physical phenomenon in a real-world environment. Hence, context can be described in a different way according to how the equipped sensors are being used. Context can also be defined as a characterization of a specific entity situation such as user profile, user surroundings, user social interaction, and user activity. For instance, we can define the entity by user and the context by location information. In this sense, context becomes a much richer and more powerful concept, particularly for mobile users, in order to make sensor network services much more personalized and useful. Therefore, context awareness refers to the capability of an application to be aware of its physical environment or situation, and respond proactively and intelligently based on such awareness.

CONTEXT AWARENESS: HUMAN ACTIVITY RECOGNITION Recognizing human-centric activities and behaviors has been an important topic in pervasive mobile computing. Human activity recognition (HAR) intends to observe human-related actions in order to obtain an understanding of what type of activities/routines individuals perform within a time interval. Providing accurate information about HAR relevant data history could assist individuals in having better well being, fitness levels, and situational awareness [3]. For example, patients with diabetes, obesity, or heart disease are advised to follow a predefined fitness program as a part of their treatment [4]. In this case, information corresponding to human postures (e.g., lying, sitting, standing) and move-

26

ments (walking, running, etc.) can be inferred by the HAR system in order to provide useful feedback to a caregiver for a patient’s behavior analysis. In addition, by the attachment of external sensor devices, such as a heart rate (HR) monitor, patients with abnormal heartbeats can be tracked easily and reported to caregivers in case of emergency in order to prevent undesirable consequences. In practice, HAR is only interested in single-person activity detection; however, it can be extended to be multiple-person recognition, which is called activity of daily living (ADL). ADL is a way to describe a person’s functional status and interaction with others. Hence, ADL becomes an essential part of community sensing, especially for community healthcare concerns (e.g., finding stress levels in a group of people). Because of the increasing development in sensor technologies and the deployment of smallsized sensors within mobile devices (e.g., the accelerometer, proximity sensor, magnetometer, GPS), and also the fact that these devices are carried by people throughout the day, the new generation of mobile devices (e.g., smartphones) appear to be an ideal platform to be used for the purpose of human-centric sensing. Especially the accelerometer sensor, which can return a realtime measurement of acceleration through all coordinate spaces, is commonly used for HAR. It is employed as either a pedometer to measure step counts and calorie consumption or a monitor to recognize user physical activities such as postures and movements. Most of the measured events/actions/attributes are related to human posture or movement (e.g., using accelerometers or GPS/WiFi/cell towers), environmental variables (e.g., using temperature and humidity sensors, microphones, and cameras), or physiological signals (attachment of external devices such as heart rate or electrocardiogram, finger pulse, etc.).

CONTEXT-AWARE MIDDLEWARE DESIGNS The growing deployment of sensor technologies in smart devices and innumerable software applications utilizing sensors to sense the surrounding physical environment in order to offer a wide range of user-specific services have led to the creation of a layered system architecture (i.e., contextaware middleware). In this way, the desired architecture can respond effectively for optimal sensor utilizations, acquiring large amounts of sensory data, meeting ever increasing application requirements, leveraging the pervasive contextprocessing software libraries, and considering mobile device resource constraints. The ubiquity of these computing devices in a dynamic environment where the network topologies actively change yields applications that behave opportunistically and adaptively with no a priori assumptions in response to the availability of diverse resources in the physical world, and also scalability, modularity, extensibility, and interoperability among heterogeneous physical hardware [2]. As shown in Fig. 2, within the ISO/OSI reference model, conventional middleware takes the place of the session and presentation layers by

IEEE Communications Magazine • June 2014

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 27

providing a higher level of abstraction built over the network operating system (OS), offering fault-tolerant resource sharing, and masking out the problems of facilitating the heterogeneity, stability, and efficiency of distributed systems. On the other hand, context-aware middleware is defined as an abstract layer between the OS and running applications. It aims to deal with the heterogeneity of the physical world through edge technologies, by adding more specialized mechanisms and services than an OS can provide. It is capable of wrapping (i.e., controlling physical devices and interacting with them to receive data), analyzing, and delivering physical world information (through sensor networks, embedded systems, RFID or NFC tags, etc.) to the application services in a transparent way. This degree of transparency allows the application layer to separate from the internal middleware operations and from detailed implementations of lower layers. Therefore, the applications can receive the context but not know the source of it. Hence, the middleware creates a shielded interface by both enhancing the level of abstraction support needed by the application and hiding lower layer operations between the physical layer (i.e., hardware and communications) at the bottom and the application layer at the top. This allows the computational burden required for context management to shift from the applications to the middleware, by letting the developers only deal with the implementation logic; they can easily control the created entities (i.e., the characterized context) through context management. In this regard, robust optimization opportunities on many system constraints (e.g., relative computational cost associated with entity relevant operations, limited battery power, insufficient information storage) can be achieved. Most important, the middleware will take the responsibility for all context-related entity management and provide complete global access to the common resources needed by all applications residing on the same host without any conflict [5]. A context is defined as a data source that characterizes the situation of an entity. In other words, it describes a physical phenomenon in a real world environment; whereas an entity describes its projection in the cyber world. An entity could be anything such as an activity, a person, a place, an object, or time with which application users interact. Furthermore, context awareness in a system indicates the ability to understand the contextual information and use it for its own benefit. Hence, context-aware middleware must have the capability to recognize and interpret the context, characterize it with entities, allow interactions among entities, and operate the whole system under such conditions as dynamic adaptation may require. At this point, semantic metadata is defined at a high level of abstraction to represent contexts as the structure and meaning of entities, and also to present context-related adaptation strategies, allowing the middleware to behave dynamically with minimal human intervention, as shown in Fig. 3. Having semantic metadata allows unambiguous specification of context models and knowledge to share among entities without loss of meaning. Thanks to its interoperability and openness, it also allows

IEEE Communications Magazine • June 2014

OSI reference model Application

Context-aware middleware Application

Context reasoning/inference

Middleware

Context interpreting/ classification

Network OS

Context aggregation/ acquisition

Presentation Session Transport Network Data link Physical

Physical-edge technology

Figure 2. Unlike conventional middleware, context-aware middleware is defined as an abstract layer between a running operating system and applications in software systems.

some other complex knowledge to be inferred at upper layers in the presence of various semantic metadata. However, the complexity of context resources in the heterogeneous physical world as well as the interactions among diverse context resources makes it difficult to describe relevant metadata explicitly. As shown in Fig. 3, some other properties of context-aware middleware that should be supported include: •It either runs standalone for managing an entire physical environment or accept the existence of an infrastructure that can deliver required services. This differentiation is caused by the heterogeneous sensing environment. •It can have a reflective property, representing the obtained entities through context as semantic metadata. The metadata may belong to the application, the middleware itself, the context, or interconnected (i.e., composition). Note that a context can differ (e.g., asynchronously obtainable over different sensors) and interoperate with other contexts. This reflection property allows the middleware to monitor its computation and possible change in the semantic world, allowing the middleware model itself to be self-represented. For example, the manipulation of its behavior may be changed. Hence, any change occurring at the meta-level can affect the underlying base level, or vice versa. •Adaptation is an important design merit that empowers users to customize systems according to individual preferences. This adaptation is defined by an autonomous process triggered by a set of requirements to improve quality of service (QoS) at the application layer. It is intended to sense the physical world, reason with the obtained context, and react dynamically toward the changing context by reconfiguring its behavior accordingly. It also supports proactive adaptations that describe the capability to envision future application requirements caused by the context change, and adjust the functionality accordingly to prevent/minimize direct application interaction with neither interfering with nor modifying the application logic. In contrast, in reactive adaptations, application itself has an ability of altering its functionality with respect to context changes.

27

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 28

One of the most

User intervene

User intervene

important properties Application controller

of context-aware

Application controller #1

Communication Services

middleware is to efficiently infer context from sensory data. The middleware provides basic

Adaptation manager

Context manager Context source #1

Context source #2

Context source #3

Middleware platform

Context manager Context source #1

Context source #2

Adaptation manager

Communication services User intervene Application controller #1

Context source #3 Transparency

Context-aware middleware platform

functionalities such as sensory data acquisition,

Figure 3. Context-aware middleware is capable of wrapping, analyzing, and delivering physical world information to application services in a transparent way.

processing and relevant context recognition.

•It may constitute entities from physical/virtual context for all types of applications, or provide application-specific information delivery (i.e., as a service provider). •It must run smoothly with the underlying OS. Since mobile applications run on a resourcelimited device with low memory, slow CPU frequency, and low power supply, lightweight middleware systems need to be designed.

CONTEXT INFERENCE ALGORITHMS One of the most important properties of context-aware middleware is to efficiently infer context from sensory data. The middleware provides basic functionalities such as sensory data acquisition, processing, and relevant context recognition. The applied methodology may show differences in context modeling or reasoning. The relevant context is called low-level (i.e., atomic) context, since all required operations are carried out directly from the data obtained by physical sensors. On the other hand, high-level context is obtained later through the combination of low-level and/or high-level contexts, which is called composition. Figure 4 shows the stages of the context inference problem during the lowlevel process. Accordingly, sensory readings are collected by a sliding window with a specific time interval and an overlapping value. The length of the window is an important design metric. A shorter window cannot seize the context properly, whereas a wider window would create latency in detection, and adds to the workload on computation. Thus, the obtained data segments by optimal windowing would provide more relevant information for context classification. In addition, the overlapping value is important as well to detect any change in the context. Preprocessing (e.g., context filtering or fusion) could be applied if raw sensory data are too coarse-grained. It may also offer necessary modifications to correct deficiencies in the data due to possible limitations on sensory operations (e.g., power concern). It is generally very challenging to analyze, build a classification model, and infer any context from raw sensory data, since it may consist of a large number of variable attributes, irrelevant information, and additive noise distortion. There-

28

fore, feature extraction is applied to exploit hidden information in the sensory data set, and remove a direct effect of the additive noise distortion. It also enables separability in context classification algorithms, while extracting and analyzing the spatial characteristics of sensory data in each window and assisting in identification of different context classes. A feature vector, as a representation of statistical characteristics in the contextual data, is then constructed by using diverse signal processing primitives, ranging from time-spacebased features such as mean, standard deviation, and correlation to frequency-spectrum-based features such as entropy, fast Fourier transform (FFT) coefficients, power density, and wavelet transforms. Table 1 summarizes the elements of a feature vector in the time and frequency domains. The constructed feature vector and allowing to it be used by classification algorithms require much computation. There is no necessity to compute features that are not helpful to infer context. Hence, the dimension of the feature vector can be reduced by using principal component analysis (PCA) or linear/quadratic discriminant analysis (LDA/QDA) [6, 7]. They help obtain sufficient statistics to model the context and allow lower computational complexity. The diverse characteristics of feature vectors enable us to have training data classes (i.e., structural features) for classification algorithms in order to represent different context inferences. Thus, a training data class is employed by classifiers to build a classification model, which will allow an unknown feature vector to test its membership to any class dependence. The process is basically to test one of classifiers to map a feature vector into a training contextual data class, called supervised classification. In this regard, various classification algorithms can be used as a classifier to implement a context recognition system. Techniques include, but are not limited to, naive Bayesian approaches and decision trees [8–10], pattern recognition techniques such as the Gaussian mixture model, k-means, knearest neighbors (k-NN) search, support vector machines (SVMs) [11], and multi-layer neural or fuzzy logic networks [6, 12]. In addition, a statistical tool-based classification such as hidden Markov models (HMMs) or autoregressive (AR) models [12, 13] is widely applied; and pattern recognition toolkits such as WEKA [14] provide

IEEE Communications Magazine • June 2014

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 29

Raw sensory data

Sliding window

Preprocessing

Feature extraction

Classification

Smoothing

Figure 4. Stages of context inference: a key challenge faced in context awareness is to infer a context in a system that takes raw sensor readings initially and then processes them until obtaining a semantic outcome. powerful solutions to the context clustering problem. Since the supervised classifiers need extensive computations to generate models for training contextual data classes and also to test for unknown patterns with trained models, the definition of self- or co-learning-based semi- or unsupervised classifiers to actualize proactive context inferences without knowing a prior data class have recently been investigated. The output of classifiers sometimes cannot resolve consistent discrimination in a time sequence of adjacent context inferences. In this case, a basic smoothing technique takes a majority voting scheme with a sliding window of a specific history length of context inferences. Hence, any inconsistency (i.e., false truths) can be eliminated. On the other hand, the context inference process may cause many drawbacks and trade-offs, with respect to context classification algorithms. A statistical tool-based classification, mostly using HMMs or AR models, is one of the foremost methods to infer context obtained via wearable or built-in smart device sensors. However, these studies mostly allow predefined user-manipulated system parameter settings, such as arbitrary formation of a context transition matrix in HMMs or building filtering coefficients in ARs, which is not suitable for online processing due to the increasing computational workload due to enlarging the data size. Other methods rely on creating feature vectors that aim at exploiting signal characteristics of the sensory data and then cluster these vectors according to specific data classes. These methods mostly intend to implement a framework that imposes a wide range of contextaware workload by using diverse signal processing primitives. The defined frameworks generally analyze spatial characteristics of the sensory signals, based on either time, space, or frequency spectrum. High-dimensional feature vectors are constructed by including many signal processing functions. Then these large feature vectors undergo pattern recognition techniques. The major drawback of these attempts comes from processing large data clusters in resource-limited hardware. In this sense, the desired approach to implement the context recognition process should provide the inference without considering a priori information, fixed threshold, and initial training data classes, show robustness in terms of any change in orientation of the device, and employ sufficient signal processing by causing non-redundant computational workload.

CHALLENGES AND VISION Mobile smart devices supporting emerging pervasive applications will constitute a significant part of future mobile technologies by providing highly proactive services requiring continuous

IEEE Communications Magazine • June 2014

Feature space

Features

Time domain

Mean, standard deviation, variance, magnitude, derivative, min-max, amplitude, histogram, interquartile range (IQR), mean absolute deviation (MAD), correlation between axes, peak counting, rms, sign, and kurtosis, zero-crossing rate

Frequency domain

Fourier transform (FT), discrete cosine transform (DCT), entropy, centroid, maximum frequency, FFT energy, FFT mean, and FFT standard deviation

Others

Autoregressive coefficients, wavelet transforms

Table 1. Elements of a feature vector. monitoring of user related contexts. However, a major challenge in utilizing these sensor-rich smart devices is their limited computational, storage, and energy resources. Specifically, continuously capturing user context through sensors imposes a heavy workload in the physical and computational capacity aspects of device working processes and drains battery power rapidly. To understand this issue better, an application example [15] can be examined. The accelerometer sensor built-in HTC Touch Pro is employed at a fixed sampling frequency. When the phone samples the accelerometer, overall power consumption on the device increases by 370 mW, while according to the data sheet of the accelerometer, it should consume less than 1 mW when active. Another example provided in [8] reports that today’s smart devices are not durable to employ all sensors at the same time by giving an example of a Nokia N95 mobile phone with a fully charged battery. It is experimentally shown that the phone battery would be totally depleted within six hours if GPS is switched on, whereas the same battery supports a conventional telephone conversation up to 10 h. Given examples conclude that any sensor employment in mobile devices would draw more current from the device battery than during regular device use. Therefore, the mobile device battery will not last a long enough time to support service.

DYNAMIC SENSOR SELECTION AND OPERATIONS To address power efficiency in the considered context-aware human activity inference applications, efficient sensor management systems need to be created to dynamically select a sufficient number of sensors (i.e., dynamic sensor selection) when a context-aware application is running. Thus, sensors can be put in order according to their power consumption levels and application relevance depending on an interesting context. In addition, the best energy saving algorithm would be the one that is infused into

29

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 30

The slow growth in energy densities of battery technologies compared with increasing computing power requirements and hardware capabilities is now driving the need to accurately model the power consumption profiles in the considered human activity recognition applications.

the low level sensory operations by manipulating the frequency of sensor sampling intervals. In this regard, different duty cycling approaches can be proposed for tuning the waveform to power a sensor for the desired power efficiency. Besides that, adaptively changing the sensor sampling period can also be another approach to achieve a certain level of power efficiency. By adjusting the sampling period in sensory operations as needed, the total number of sampling occurrences either increases or decreases. As a result, relevant power consumption will adjust accordingly. Above all, an adaptive sensor management mechanism/system to assign a mixture pair of duty cycles and sampling periods simultaneously would be a cure, consuming less power while running context-aware applications accurately enough. However, the intervening sensory operations to achieve this power efficiency may jeopardize the accuracy of context-aware services, thereby creating a trade-off between the power consumption caused and the accuracy provided by these services.

MODELING SMART DEVICE BATTERY BEHAVIOR FOR ENERGY OPTIMIZATIONS The use of smart devices is also highly constrained by limited battery power. The slow growth in energy densities of battery technologies compared with the increasing computing power requirements and hardware capabilities is now driving the need to accurately model the power consumption profiles in the considered human activity recognition applications. To do this, we need to explicitly consider the impact of different device usage patterns and its relationship with the projected effect on power consumption. Modeling the battery’s nonlinearities, and understanding the correlation between the usage patterns and battery depletion will ultimately lead to successful discovery of suboptimal energy reduction strategies that will eventually help maximize the used energy consumption to improve the QoS of context-aware applications.

EFFICIENT CONTEXT INFERENCE ALGORITHMS Besides the power consumption and battery density considerations, the analysis and inferral process of contextual sensory data has many drawbacks. Many studies can be found [8, 9, 16, 17] in which a framework is proposed to capture and evaluate sensory data. Most studies rely on recognition of user activities and definition of common user behaviors. The applied methods in relevant studies are based on statistical models, predefined feature extraction, and classification algorithms. However, none of these studies engage themselves in modeling a common framework in order to construct a base structure for future context-aware applications. They prefer canalized solutions to solve their own unique applications instead of a generalized approach. Therefore, these studies mostly focus on a specific sensor to discover possible target applications in order to exploit the contextual data. A generic framework that fulfills requirements set by all types of context-aware applications has not yet been clearly identified. The problem often comes from the difficulty in building a reliable

30

data set to represent a specific context interest, since the obtained sensory data can vary significantly under different circumstances (e.g., human speech with varying background noise or placement of the mobile device). As a result, classifiers would not be practical toward varying sensing conditions, and eventually would perform poorly. Hence, robustness becomes an important system attribute to consider. This issue even turns into a severe problem resulting in different inference assumptions by multiple co-located mobile devices on a similar sensing event. One solution is to take advantage of cloud computing technologies, enabling information and ensemble situational resources to be shared among co-located devices. Another important system attribute to consider is to prevent the use of supervised learning strategies. Most systems take predefined models or classifiers where training data is obtained through several repetitions of a similar experiment setup. However, it yields a large amount of data in return to process, and makes the subsequent analysis be carried out offline. In particular, obtaining training data classes to feed statistical models, pattern recognition, or machine learning algorithms in a supervised learning strategy is an expensive real-time operation for smart devices and is impractical when considering computation manners; that is, acquiring and analyzing data, resource management by storing training samples, a scalability problem in labeling data, and a bandwidth problem in exchanging large amounts of information. Therefore, the utilization of sensors must be lightweight and unobtrusive, and the applied classification/clustering and machine-learning algorithms must be applied without computationally expensive, human-intervening offline methods.

CONTEXT-AWARE MIDDLEWARE SOLUTIONS Finally, there is no common middleware system that covers all types of application settings, provides adaptation toward changing context, and acquires a collection of asynchronous heterogeneous contexts to create different abstract entities. There is not even context-aware middleware that succeeds in having full transparency, which eliminates direct involvement of an application into a context modeling process by imposing less computational workload on resource-limited mobile devices. In this direction, gathering diverse and asynchronous information, and presenting it to the application would be future work in mobile computing middleware research. Most works on creating a context-aware application emphasize either setting a minimum number of sensors when they are needed or maximizing power efficiency by solely applying less complexity in computations or changing transferring methods of obtained context to the outer network services. From the standpoint of the creation of a framework design in contextaware applications, it would be notable to mention the following studies. EEMSS [8], Jigsaw [17], Sensay [18], and SeeMon [9] use hierarchical sensor management strategy by powering a minimum number of sensors including an accelerometer, a microphone, and GPS, and applying fixed sensor duty cycles in order to

IEEE Communications Magazine • June 2014

LIU_LAYOUT.qxp_Layout 5/30/14 1:04 PM Page 31

improve device battery lifetimes. Unfortunately, sensors have fixed duty cycles, and also are not adjustable to respond differently to varying user behavior. In addition, energy consumption is reduced by shutting down unnecessary sensors at any particular time. On the other hand, classification of sensory data is based on predefined test classification algorithms. In contrast, the study in [19] enlightens future research directions by presenting a generic system framework. It focuses on inhomogeneity and user profile adaptability while examining the trade-off between accuracy in contextual inference through sensory data and required power consumption. To this end, context-aware frameworks should provide a scalable and energy-efficient monitoring architecture. The designed architecture should actively manage operations between context-aware applications and computing resources, and coordinate multiple applications to effectively leverage the limited resources and provide quality service to users.

CONCLUSION The existence and awareness of user-related context provides the possibility of being conscious of a physical environment or situation around mobile smart-device users. It helps network services respond proactively and intelligently to applications. Therefore, the inference of a vast variety of human activities by a series of sensory observations in a computationally pervasive way has recently drawn much interest. However, the evolution of the ubiquitous smartphone sensing on resource-constrained mobile devices in terms of battery, memory, and bandwidth is fundamentally driving the the design and development of context-aware middleware services. In this sense, this article extensively examines context-aware middleware designs for human activity recognition applications by studying the definition of context awareness, context-aware middleware designs, and various context inference algorithms. We also point out the challenges faced during the design process and system integration. More important, we also provide an inside look by presenting future research directions and possible solutions.

REFERENCES [1] L. Atzori, A. Iera, and G. Morabito, “The Internet of Things: A Survey,” Comp. Net., vol. 54, no. 15, 2010, pp. 2787–2805. [2] D. Bandyopadhyay and J. Sen, “Internet of Things: Applications and Challenges in Technology and Standardization,” Wireless Pers. Comm., vol. 58, no. 1, 2011, pp. 49–69. [3] E. Jovanov and A. Milenkovic, “Body Area Networks for Ubiquitous Healthcare Applications: Opportunities and Challenges,” J. Medical Sys., vol. 35, no. 5, 2011, pp. 1245–54. [4] K. G. Stanley and N. D. Osgood, “The Potential of Sensor-Based Monitoring as a Tool for Health Care, Health Promotion, and Research,” Annals of Family Medicine, vol. 9, no. 4, 2011, pp. 296–98. [5] K. E. Kjær, “A Survey of Context-Aware Middleware,” Int’l Multi-Conf.: Software Eng., 2007, pp. 148–55.

IEEE Communications Magazine • June 2014

[6] Y.-P. Chen et al., “Online Classifier Construction Algorithm for Human Activity Detection Using a Tri-Axial Accelerometer,” Appl. Math. and Comp., vol. 205, no. 2, 2008, pp. 849–60. [7] P. Siirtola and J. Roning, “Recognizing Human Activities User-Independently on Smartphones Based on Accelerometer Data,” Int’l J. Interactive Multimedia and Artifical Intelligence, vol. 1, no. 5, 2012, pp. 38–45. [8] Y. Wang et al., “A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition,” ACM MobiSys ’09, 2009, pp. 179–92. [9] S. Kang et al., “Seemon: Scalable and Energy-Efficient Context Monitoring Framework for Sensor-Rich Mobile Environments,” ACM MobiSys ’08, 2008, pp. 267–80. [10] O. Yurur, C. H. Liu, and W. Moreno, “Unsupervised Posture Detection by Smartphone Accelerometer,” Electronics Letters, vol. 49, no. 8, 2013, pp. 562–64. [11] L. Sun et al., “Activity Recognition on an Accelerometer Embedded Mobile Phone with Varying Positions and Orientations,” UIC ’10, 2010, pp. 548–62. [12] C. Zhu and W. Sheng, “Human Daily Activity Recognition in Robot-Assisted Living Using Multi-Sensor Fusion,” IEEE Int’l. Conf. Robot. and Auto., May 2009, pp. 2154–59. [13] L. T. Vinh et al., “Semi-Markov Conditional Random Fields for Accelerometer-based Activity Recognition,” Appl. Intell., vol. 35, no. 2, 2011, pp. 226–41. [14] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, “Activity Recognition Using Cell Phone Accelerometers,” SIGKDD Exploration Newsletter, vol. 12, no. 2, Mar. 2011, pp. 74–82. [15] B. Priyantha, D. Lymberopoulos, and J. Liu, “Littlerock: Enabling Energy-Efficient Continuous Sensing on Mobile Phones,” IEEE Pervasive Comp., vol. 10, no. 2, Apr.–June 2011, pp. 12–15. [16] O. Yurur et al., “Adaptive Sampling and Duty Cycling for Smartphone Accelerometer,” IEEE MASS ’13, Oct. 2013, pp. 511–18. [17] H. Lu et al., “The Jigsaw Continuous Sensing Engine for Mobile Phone Applications,” ACM SenSys ’10, 2010, pp. 71–84. [18] D. Siewiorek et al., “Sensay: A Context-Aware Mobile Phone,” IEEE ISWC ’03, 2003, pp. 248–49. [19] O. Yurur, M. Labrador, and W. Moreno, “Adaptive and Energy Efficient Context Representation Framework in Mobile Sensing,” IEEE Trans. Mob. Comp., vol. PP, no. 99, 2013.

The evolution of ubiquitous smartphone sensing on resource-constrained mobile devices in terms of battery, memory, and bandwidth is fundamentally driving the design and development of context-aware middleware services.

BIOGRAPHIES OZGUR YURUR ([email protected]) received double major degrees from the Department of Electronics Engineering and Computer Engineering at Gebze Institute of Technology, Kocaeli, Turkey, in 2008, and an M.S.E.E. degree from the Department of Electrical Engineering at the University of South Florida (USF), Tampa, in 2010. He is currently pursuing his Ph.D. degree at USF. His research area covers ubiquitous sensing, mobile computing, machine learning, and energy-efficient optimal sensing policies in wireless networks. C HI H AROLD L IU [M’10] ([email protected]) is an associate professor and head of Software Service Engineering at Beijing Institute of Technology, China. He holds a Ph.D. degree from Imperial College, United Kingdom, and a B.Eng. degree from Tsinghua University, China. He has worked for IBM Research and Deutsche Telekom Laboratories. His research interests include Internet of Things, big data, ubiquitous computing, and wireless networks. He serves as the Editor for KSII Transactions on Internet and Information Systems, and has organized many leading conferences and workshops. WILFRIDO MORENO ([email protected]) received his B.S.E.E., M.S.E.E., and Ph.D. degrees from USF. He is a professor in the Department of Electrical Engineering at USF and director of the Ibero-American Science and Technology Educational Consortium at USF (ISTEC-USF). His current areas of interest are oriented toward system integration by providing “off-the-shelf” hardware/software solutions to industrial applications.

31