Integrating Sensors, Multimedia and Semantic ...

DemaWare2: Integrating Sensors, Multimedia and Semantic Analysis for the Ambient Care of Dementia Thanos G. Stavropoulos1, Georgios Meditskos1 and Ioannis Kompatsiaris1 1

{athstavr, gmeditsk, ikom}@iti.gr Information Technologies Institute, Centre for Research & Technology - Hellas, Thessaloniki, Greece

Abstract. This paper presents DemaWare2, an Ambient Assisted Living framework to support the care of people with dementia. The framework integrates various sensor modalities, such as ambient, wearable, offline and cloudbased, together with sophisticated, interdisciplinary methods including image, audio and semantic analysis. Finegrained, atomic events, such as object manipulation, are aggregated into complex activities through semantic fusion. Applications tailored to monitoring dementia symptoms support clinicians to drive effective, timely interventions and evaluate their outcomes. The framework was evaluated for its robustness, reliability and clinical value in realworld lab trials and home installations. Keywords: ambient assisted living, ambient intelligence, knowledge representation, reasoning, dementia

1. Introduction Ambient Assisted Living (AAL) is an eminent field of research towards improving Quality of Life through technology. The ever-increasing number of portable, compact, affordable and interconnected computing devices around us is bringing the visions of Pervasive and Ubiquitous Computing to life. Ambient Intelligence (AmI) (Weiser, 1991), as one of the leading technological paradigms of the future, builds upon this infrastructure, introducing intelligence through sensing, planning and acting in such pervasive environments. Its diverse application domains range from smart homes (Friedewald et al., 2005), offices (Le Gal et al., 2001), agriculture (Eisenhauer et al., 2010) and health, such as in AAL applications. AAL is of particular interest for people in need of medical attention such as the disabled or the elderly and poses several research challenges yet to be met. Several ad-hoc and limited integrated solutions have been presented so far, which manipulate certain devices and knowledge-driven analytics to particular ends. In parallel, the average lifespan increase across the world has been accompanied by an unprecedented upsurge in the occurrence of dementia bringing about high socio-economic costs. These considerable repercussions of dementia growing rates are intensifying the need to find effective means of treatment and support. Global health systems have been forced to undergo radical changes, strongly promoting more preventative care, and care at home. In the absence of a cure for dementia, it is vital that considerable research efforts are aimed at improving care and quality of life, to minimize hospitalization, meanwhile significantly reducing the health system’s expenses and relieving caregivers and the people with dementia themselves. This work presents DemaWare2, a holistic framework to integrate several heterogeneous modalities, such as raw sensor input, real-time processing, higher-level audio and image analytics and their unanimous semantic representation and interpretation. AAL environments using DemaWare2 can not only manipulate online and open sensors but also proprietary low-cost health monitoring devices which are now dominating the market, through the integration of numerous Cloud APIs. Besides the technical contribution of unifying several communication protocol and device platform heterogeneities, the framework also integrates computer vision and audio analysis algorithms, while such existing integrated platforms are limited (De Paola et al., 2012). After unifying them at a syntactic (web

2

service) level, a universal semantic representation is established to unambiguously store information from all sensors in the form of measurements and activity detection from image and audio analysis. Semantic interpretation allows the intelligent temporal fusion and aggregation of such events, and the identification of problematic situations, which are both crucial to clinical monitoring and interventions. In detail, the framework’s analysis layer adopts a multi-sensor data analytics solution combined with intelligent decision making mechanisms, facilitating an accurate representation of the individual’s condition towards providing appropriate feedback to enhance standard clinical work-flows. The deployment of the framework in a real-world environment had to address intrinsic challenges that mainly stem from the vastly heterogeneous and noisy sources that are aggregated, as well as from the different way activities are performed even by the same person. More specifically, a common characteristic of relevant activity recognition approaches (reviewed in Section 2) is that they require the definition of strict and highly structured activity patterns in the form of complex concept descriptions (TBox axioms) or fixed rule patterns. For example, certain events are selected for designating the start/end of activities or strict temporal correlations are considered among events. Thus, they fail to capture intrinsic characteristics and address practical challenges in real-world scenarios. For instance, the use of time windows, slices or background knowledge about activity duration fails to capture the fact that the duration of activities usually varies in practice or, at least, the segmentation task becomes too complex. Moreover, the use of highly structured background knowledge relevant to the order of activities or the activity boundaries in terms of start/end activities is not always a practical solution since many activities do not have a predefined order to be carried out. Last but not least, the data cannot be directly processed by activity patterns (e.g. rules) that, for example, explicitly enumerate the sub-activities and the temporal relations that are involved, due to the incomplete and noisy nature of the domain. Rather than using ontologies as strict contextual models, the framework defines abstract situations, i.e. lightweight OWL models that encapsulate loosely coupled dependencies among lower and higher level conceptualizations, such as the locations and objects that are involved in an activity. Another research challenge addressed in DemaWare2 is its successful adaptation to the peculiarities of clinical scenarios, for which its acceptance and effectiveness have been extensively evaluated through proof-of-concept installations, in two different scenarios. The first scenario utilizes the framework to intelligently sense and assess 98 individuals during their visit in the daycare center of the Greek Association of Alzheimer Disease and Relative Disorders (GAADRD), while performing a lab trial of simulated activities of daily living (ADL) and a clinical interview. First of all, the combined semantic fusion of sensor observations with activity recognition through videos and images has reached a recall and precision close to 82%. From a clinical perspective, the average duration for each lab task has managed to distinguish the AD (Alzheimer’s Disease) group from the rest, while the usage of I/O devices in the simulated environment further distinguishes healthy from MCI (Mild Cognitive Impairment) individuals, at a statistically significant level. While the lab scenario technologically serves as an incubator for short monitoring and assessment, the home scenario broadens these capabilities and clinical aims. A slightly different set of sensors is used in residential, largerscale setups, verifying the framework’s modularity and extensibility. A methodology was developed combining psychological approaches with technological aids to design and monitor precise interventions for care. In detail, sleep, physical activity and daily tasks are monitored through object movement, presence and appliance usage, fused with image and video analytics. Then, clinicians consult the DemaWare2 monitoring applications and not just the, often subjective, participant interviews, to identify problems and drive interventions. Four residential users have been receiving treatment aided by DemaWare2 for the course of a few months. As the former user has collected enough data, algorithm evaluation has shown a similar performance to the lab environment, although the unconstraint residential environment imposes new challenges that are thoroughly discussed. The combined monitoring and clinical interventions have also shown an increase in the participants’ mental state and the ease of depression for both users. Previous work in the DemaWare platform has focused on integrating a limited set of sensors, supporting offline data transfer and existing image and audio analytics using open web standards such as WSDL (Stavropoulos et al., 2014b). However, DemaWare2 extends the platform both horizontally, with numerous additional sensors, real-time data collection, proprietary cloud services, and vertically with various transport protocols and an expanded semantic interpretation module for the context-aware fusion of the added modalities and the higher-level activity deduction. As such, DemaWare2 has been enriched with advanced knowledge representation schemes, encapsulating vocabularies for modelling the application context (e.g. activities, measurements, locations and objects) and inferencing capabilities for the derivation of complex activities and problems. It is also complemented with novel

3

applications tailored to the context of assessing and monitoring end users with dementia, and presents novel findings through deployment in various settings. The rest of the paper is structured as follows: the next section presents a brief survey of related work in platforms and frameworks for AAL, sensor-based and knowledge-driven approaches in such environments, each time in comparison to the proposed framework. The third section presents the core contribution of this work, the DemaWare2 framework, its architecture, integrated sensors, cloud services, real-time capabilities, image and audio analytics, semantic unification and interpretation and user applications. The fourth and fifth sections respectively present proof-of-concept deployments in various environments and evaluation results in terms of activity detection accuracy and clinical value. The final sections present conclusions drawn from this work and possible future directions for research.

2. Related Work Related work concerns two different aspects of the proposed framework: pervasive and mobile integration frameworks and Semantic Web technologies for knowledge representation and activity recognition. Existing research and the advances that DemaWare2 contributes are presented in both cases. 2.1. Pervasive Integration Frameworks and Clinical Applications Pervasive technologies have already been employed in several ambient sensing and assisted living deployments for the past decade. Such works have usually been motivated and driven by their domain of application, which dictates sensor modalities and analytics in each existing framework. The DemaWare2 framework proposed in this work complements developments so far, by integrating a wide range of sensor modalities, technologies and highlevel analytics, in the context of AAL. It also focuses on semantic fusion of a wide range of sensed context and specifically targets the ambient care of dementia through a combined psychological and technological approach through tailored applications and clinical deployments. The related work survey in this section accordingly follows three directions. It first examines existing holistic platforms in ambient assisted living and smart spaces in general, followed by existing work in ambient clinical care. Finally, it presents existing work on semantic interpretation in similar context, in comparison to the proposed one. Much work has aimed to provide a unified framework for Ambient Assisted Living and Ambient Intelligence, focusing in higher level services or the integration of specific hardware. In detail, the work in (Wolf et al., 2010), introduces openAAL, a general-purpose open source middleware for AAL, which provides high-level software processes such as context management, service matching, composition and work-flow execution. However, openAAL does not yet provide integration with any hardware-based component i.e. sensors. FamiWare (Gámez and Fuentes, 2011) implements a Publish/Subscribe approach that enables discovery of services and fusion of data sources. However, it supports very limited hardware i.e. Android smartphones as sensors and TinyOS sensors. Similarly, the work in (De Paola et al., 2012) provides means for context sensing and user profiling, without integrating and supporting specific sensors. In contrast to those works, DemaWare2 places equal effort in integrating different sensor modalities, higher-level multimedia and semantic analysis. In detail, DemaWare2 provides various sensor modalities offered by a wide range of hardware, while in the same time, it employs existing computer vision and audio analysis as sources of context information. Finally, semantic interpretation is used to fuse the aforementioned modalities for advanced context-sensing and problem detection. Excluding the AAL context, work in Ambient Intelligence and smart spaces in general has frameworks for sensor integration. The aWESoME-S middleware (Stavropoulos et al., 2013) integrates sensors and actuators together with knowledge representation based on semantic service descriptions. Rule-based agents and defeasible logics automatically manage a smart building environment, with an aim to reduce energy consumption while ensuring comfort (Stavropoulos et al., 2014a). Other middleware, such as AIM (Capone et al., 2009) and Hydra (Eisenhauer et al., 2010), similarly integrate sensors and actuators using various formalities for semantic service descriptions. The Hydra framework in particular has been employed in various smart space scenarios, such as smart homes, eHealth and agriculture. Compared to DemaWare2, the aforementioned middleware platforms have focused on the integration of sensors and different domains, such as smart buildings and even healthcare, in a general sense. On

4

the other hand, DemaWare2 integrates a similar set of lifestyle, smart home sensor networks, with the addition of proprietary health monitoring devices, multimedia analytics and semantic analysis. It also capitalizes in a certain domain, the ambient care of dementia, through which tailored applications and clinical methods have extensively been applied in clinical contexts aiding in the assessment, monitoring and care of the disease. Focusing on clinical care through sensing, the work in (Suzuki et al., 2006) has deployed infrared motion sensors in clinics to monitor sleep disturbances. According to questionnaires, the sensors have indeed identified days of disturbed sleep. However, the work reveals many limitations of using a single, only, sensor. Similarly, the work in (Chang et al., 2012) presents a sensor network deployment in nursing homes in Taiwan to continuously monitor vital signs of patients, using web-based technologies, verifying the system’s accuracy, acceptance and usefulness. Nevertheless, it so far lacks the ability to fuse more sensor modalities such as sleep and ambient sensing, with limited interoperability. On the contrary, such concepts have been described for instance in the E-monitor framework for ambient sensing and fusion in a clinical context (Cislo et al., 2013). DemaWare2 implements and extends these concepts by offering a unified framework with knowledge representation for sensor interoperability. Through this framework, further sleep and lifestyle metrics are measured to automatically assess disturbances and their causes, aiding clinical monitoring and interventions, verified through clinical deployments. When it comes to clinical applications for the care of dementia, the integration of sensor monitoring and fusion with image and audio analysis in DemaWare2, allows for new insights in complex activity monitoring, which is characteristic of one’s cognitive state and constitutes most valuable information when designing interventions for the care of dementia. Additionally, the integration of multiple lifestyle and physical activity sensors with analysis allows to monitor multiple aspects of life, including activities, sleep, exercise and problems, as opposed to a single aspect usually met in existing systems. 2.2. Knowledge Representation and Activity Recognition Given the inherently open nature of pervasive, sensor-driven systems, where a crucial requirement is the need to aggregate low-level context information and meaningfully integrate domain knowledge, it comes as no surprise that Semantic Web technologies (Berners-Lee et al., 2001) have been acknowledged as affording a number of highly desirable features. The Semantic Web (SW) is a resource-oriented extension of the current Web that aims at a common framework for sharing and reusing data across heterogeneous applications and systems. The rationale is to convert unstructured and semi-structured information into a ‘web of data’, where the underlying semantics are expressed in a formal and machine-understandable way. Within this vision, ontologies play a key role, providing consensual and formally-defined terms for describing resources in an unambiguous manner. All in all, ontologies and in particular OWL 2 (Grau et al., 2008), aim to bring to the table the ability to formally capture intended semantics and to support automated reasoning, supporting sharing, integration and management of information from heterogeneous sources. Via explicitly rendering meaning, ontologies try to facilitate data exchange between systems and components in an open, extensible manner, maintaining semantic clarity across applications. Overall, SW technologies have been shown to successfully address several pervasive computing challenges, such as complex sensor data representation (Nevatia et al., 2004), human activity recognition (Chen et al., 2012),modelling and querying location data across heterogeneous coordinate systems (Stevenson et al., 2009b). The use of ontologies provides an elegant solution to the data interchange within an open system (Stevenson et al., 2009a), while ontological reasoning proves useful in manipulating structured conceptual spaces (Bettini et al., 2010) (Ye et al., 2007). Under this paradigm, OWL is used for describing the elements of interest (e.g. events, activities and location), their pertinent logical associations, as well as the background knowledge required to infer additional information. For instance, in a smart home (Chen and Nugent, 2009), concrete situations correspond to individuals of OWL classes. Their realization is used to determine which context concepts describe a specific situation. A similar approach is followed in (Riboni and Bettini, 2011), (Gu et al., 2004) (Riboni et al., 2016), where complex activities are recognized based on subsumption reasoning. Time windows (Okeyo et al., 2014) and slices (Okeyo et al., 2012) (Riboni et al., 2011), background knowledge about the order (Patkos et al., 2010) (Roy et al., 2011) or duration (Ye and Stevenson, 2013) of activities also constitute commonly used approaches for segmentation and activity classification. However, existing methods are based on strict contextual patterns that cannot provide enough flexibility to handle the imprecise and ambiguous nature of events in the real world. Therefore, the contribution of this work is its use

5

of ontologies to represent abstract dependencies among high-level situations (e.g. complex activities) and low-level observations (e.g. objects and locations) in a loosely-coupled manner. Motivated by the powerful representation and reasoning capabilities of OWL 2, we have integrated and adapted in DemaWare2 a context-based fusion framework for activity recognition using OWL 2 as the underlying knowledge representation and reasoning language. The framework introduces the notions of context dependencies and context links (Meditskos et al., 2014). The contextual information encapsulated inside ontologies is used for identifying connections (links) among observations that signify the presence of complex activities and to subsequently classify them as high-level activities. DemaWare2 extends this framework with a SPARQL-based assessment layer for detecting discern traits that have been identified by the clinicians as relevant for assessment and diagnosis, supporting clinicians in obtaining a comprehensive image of the person’s condition and its progression, without being physically present, aiding them to design and adjust intervention.

3. The DemaWare2 Framework DemaWare2 aims to integrate a wide variety of sensor modalities, entailing heterogeneous communication protocols and data formats. In detail, such sensors should include not only ambient and wearable devices, but also wireless sensor networks as traditionally employed in smart spaces. Among those sensors, the framework should also exploit the most recent sensor communication technologies, through cloud proprietary APIs. DemaWare2 aims to exploit computer vision and audio analysis techniques as context-sensing mechanisms, fusing their output with sensor data. Using a universal semantic representation and interpretation techniques the frameworks extracts the highest level of aggregated and accurate information. While the platform maintains its generality and applicability in any AAL context, it also capitalizes in the use case of dementia ambient care, through tailored applications. In detail, clinical methods have helped to design applications for monitoring and intervention protocols in clinical contexts. To address those requirements and implement the aforementioned capabilities, the framework embodies a modular and layered architecture, shown on Fig 1, combining numerous platforms and technologies. Each layer offers an additional level of integration. The sensor layer includes all supported hardware storing heterogeneous data on the data layer, either on the framework’s side or on proprietary sensor cloud storage. The processing layer hosts numerous software components that address each sensor modality, data format and platform, either by simply interpreting open data, performing a primary event aggregation or even applying sophisticated image, video and audio analytics. Horizontally, the framework is extensible with any number of modules, either with respect to sensors, data formats or processing components and analytics. The output of such processing is universally stored on a unanimous knowledge base on the semantic layer, which also applies further semantic analysis, event fusion and detection of problems i.e. anomalies. All DemaWare2 functions are exposed through universal web service interfaces, which are later on used by domain-specific applications offering a tailored view to different types of users. The next subsections each present a different layer of the proposed framework. 3.1. Sensors DemaWare2 implements an extensible, modular approach to integrate various sensors, in the sense of plug-in modules addressing each of them. Each device or device bundle, such as Wireless Sensor Networks (WSNs), is manipulated by the system, through a dedicated module that conforms to its communication protocol, data format and sometimes platform, introducing an initial layer of data, or else syntactic, interoperability in DemaWare2. After raw sensor data is retrieved, processing and analysis commence on the processing layer of the framework, as described on the next subsection. This modularity and extensibility of the framework is verified through diverse deployment scenarios examined in section 4. Currently, the platform offers a rich selection of sensors and WSNs for completeness in a patient/elderly monitoring AAL context. Furthermore, each measurement type is offered by more than one sensor alternative, to best fit each scenario according to e.g. comfort, battery life and other parameters, as in the deployment scenarios of section 4.

6

Fig 1. The DemaWare2 framework architecture and its decomposition in layers from supported sensors to processing components, semantic infrastructure and user applications

Table 1 presents all supported sensors with their corresponding types, data modalities, interfaces to DemaWare2 and processing components. In detail, a sensor can be either ambient, unobtrusively hidden in the environment, or wearable taking comfort into account. Alternatively, WSNs are considered a special type of interconnected ambient sensors, at network level, pervasively surrounding the user. The data types considered in DemaWare2 cover image and audio for specialized analysis and more self-contained measurements such as physical activity, sleep recordings, object motion, presence and door/window contact. Other measurements require further aggregation to derive meaningful information such as power consumption, accelerometer movement and skin conductance. 3.1.1. Ambient Sensors Depth Camera1: a 3D camera recording images and depth data online, using its open SDK and a USB interface. The data is later exploited by computer vision algorithms: CAR, based on images and depth (Romdhane et al., 2013), and HAR, based on images (Avgerinakis et al., 2013), used for localization of subjects in the scene and activity recognition. A portion of the image dataset is used for ground truth annotation for algorithm evaluation. IP Camera: IP Cameras (any model) can be used to substitute the Depth Camera in situations where only HAR is required. The camera is integrated into the framework by configuring its FTP service to store image data online, later exploited by HAR for activity recognition. Gear42: The Gear4 Renew SleepClock is an affordable iPhone bedside dock that measures sleep (light and deep) 1 2

ASUS Xtion PRO Live - https://www.asus.com/us/Multimedia/Xtion_PRO_LIVE/ Gear4 Renew SleepClock: http://www.stage.gear4.com/

7

Device

Sensor Type

Data Types

Data Transfer

Transfer Protocol

Component

Depth Camera

Ambient

Images and Depth

Online

USB

CAR, HAR

IP Camera

Ambient

Images

Online

IP/FTP

HAR

Gear4

Ambient

Sleep Recordings

Offline

USB

SCPC

Aura

Ambient

Sleep Recordings

Online

REST, OAuth v1

APC

GoPro Camera

Wearable

Video

Offline

USB

WCPU

Microphone

Ambient

Audio

Online

USB/Audio jack

OSA

DTI-2

Wearable

Offline

USB

WWPC

UP24

Wearable

Online

REST, OAuth v2

UP24PC

Plugs

WSN

Accelerometer, Skin Conductance Steps, Distance etc. Sleep Recordings Power Usage

Online

USB

CPEP

Tags

WSN

Object Motion, Presence, Door/Window

Online

TCP/IP

TPC

Table 1. Ambient, wearable and wireless sensor networks supported in DemaWare2

and awake segments i.e. sleep interruptions and durations. The device is daily switched on by clinical staff, which also exports recorded data via USB. Aura3: Aura is the second, complementary sleep sensor in the DemaWare2 framework. Both sensors are ambient, unobtrusive and measure the same metrics. However, Aura uses pressure on the mattress for detecting presence, initiating a recording and sensing sleep phases and their duration. Still, it is not yet known which device and method is most accurate so both devices are integrated in DemaWare2 for completeness. Aura stores sleep recordings to the proprietary cloud storage infrastructure of the manufacturer, directly over Wi-Fi and immediately after a sleep session is concluded. It can then be retrieved at any point by DemaWare2 via the open REST API. Security and privacy, which plays an important role in the AAL framework, are still ensured. First of all, a user (either the end-user or the assigned clinician) signs up to the proprietary cloud service to upload data from a specific device. The proprietary owner does not, therefore, obtain personal data and the identity of the end-user. The user also requests permission and is redirected to the manufacturer’s website to authorize the DemaWare2 framework to retrieve his data. After that, the staff associates the device with DemaWare2 user account, which also contains personal information (demographics and medical) that remain internal to the framework and hidden from third parties. Data retrieval can either be scheduled to be pulled daily, or frequently polled if a near real-time scenario is investigated. 3.1.2. Wearable Sensors GoPro Camera4: The wearable camera is worn by the end users, most often with the help of clinical staff. Video is recorded during daily activities of interest e.g. morning routes or memory exercises, and later on retrieved offline by the staff (using the USB interface). The videos are exploited by different computer vision techniques for localization, object and activity recognition. Microphone: Microphones are used in DemaWare2 exclusively during clinical interviews with end-users, to record and store high-quality audio. In order to obtain the best audio quality, the microphones used are mostly wearable5, without excluding the use of ambient ones. The framework’s drivers and applications are used to timely 3

Withings Aura - http://www2.withings.com/us/en/products/aura GoPro Hero II, III - http://gopro.com/ 5 Sennheiser FreePORT - http://en-de.sennheiser.com/ 4

8

orchestrate the recordings, e.g. exclude the clinical interviewer’s voice. The recordings are used for offline speech analysis for mood assessment. DTI-26: DTI-2 is a non-commercial device worn on the wrist that measures accelerometer movement in 3D space and skin conductance values. The values are stored in internal memory and then transferred to the framework via an open API. While skin conductance is of great interest towards stress assessment, the device is relatively large in order to accommodate the skin sensors and internal memory, which also decrease battery life to no more than a day. Therefore, the device is mainly used in clinical trials and home visits and handled by clinical staff. However, charging the device daily has sometimes been exploited as a daily task to remember for people with dementia in some deployments. UP247: UP24 is the second alternative wrist-worn device to measure physical activity in DemaWare2. It is a thin wristband, comfortable to wear 24/7 and with lasting battery life of up to 7 days. On the other hand, UP24 can only measure physical activity in terms of steps travelled, distance covered and calories burned, all according to the builtin accelerometer, without the ability to measure skin conductance or temperature. Therefore, it is suitable for environments where the user is autonomous and requires longer-term monitoring at the cost of lacking stress measurement capabilities. Although there have been studies to compare such proprietary sensors for their accuracy (Case et al., 2015), the measurements cannot be taken literally. As the manufacturers claim they are mostly indicative for the relative amount of exercise rather the exact number of steps, which is still clinically valuable in such a health monitoring context. UP24, like Aura, is based on a secure proprietary cloud service for data storage and retrieval. The wristband constantly transmits measurements over Bluetooth to any smartphone, which will act as a gateway to upload them to the manufacturer’s cloud service. DemaWare2 implements the same retrieval strategy for UP24 as with Aura. The user may subscribe to the proprietary service, giving away non-sensitive information, and authorize the devices data to be retrieved by DemaWare2. Henceforth, he can associate the device with an end-user on the DemaWare2 repository containing medical and personal data. Physical activity data is then retrieved securely over OAuth v2, either once in a day or constantly for real-time monitoring scenarios. 3.1.3. Wireless Sensor Networks Plugs8: Plugs are compact, affordable devices which can easily be attached to any household electrical appliance on a plug outlet (e.g. TV, coffee maker etc.) or its cable power supply (e.g. air conditioning, lighting etc.). As part of smart home equipment bundles, commercially available in the retail market, they are easy to setup and deploy covering an entire home or building. The selected Plug bundle forms a ZigBee 9 mesh networks optimized for smart homes. Many networks can be deployed in the same space, each of them accommodating up to thirty devices and interfacing with the installed DemaWare2 framework via a USB interface. The Plugs act both as sensors, measuring the power usage of attached appliances and as actuators, by switching appliances on and off. In the framework of AAL monitoring, the first capability is emphasized in this work. Tags10: The Tag system is comprised of various retail sensors in a smart home context. Object motion Tags are compact tag-shaped sensors that can be attached to daily items, such as the TV remote, a watering can, the iron, a cupboard, and monitor their movement in the three-dimensional space, using an accelerometer. Presence Tags can detect a person’s movement in the monitored space, using standard IR motion detection (even in the dark), while Door/Window tags can detect when the door or window that they are attached to opens. All Tag sensors are compact and affordable, and form a simple standard radiofrequency star network of wide range, enough to cover an entire home or building. The network coordinator is a gateway which uploads events to the cloud and can directly push them to any subscriber, in this case the DemaWare2 framework, for processing.

6

Philips DTI-2 non-commercial wristwatch kindly provided by Philips Research NL - http://www.philips.nl/ Jawbone UP24 - https://jawbone.com 8 Circle, Cirlce+ and Stealth products by Plugwise.nl - https://www.plugwise.nl/ 9 The ZigBee Alliance - http://www.zigbee.org/ 10 Tags, PIR KumoSensor, Reed KumoSensor of the Wireless Sensor Tag System - http://wirelesstag.net/ 7

9

3.2. Sensor and Multimedia Analytics This layer of the framework hosts a set of libraries for sensor data processing which range from simple, textual data, to sophisticated image and audio analytics. Most existing frameworks focus on a certain direction, which can be individual sensors for a specific purpose, wireless sensor networks in a smart space or higher-level computer vision analytics. On the contrary, DemaWare2 places equal effort on all the above directions offering a wide range of sensor modalities and higher level multimedia analytics. In detail, this layer accomplishes the following goals: i) It processes and interprets sensor data, either using proprietary libraries (e.g. for binary DTI-2 data) or according to open data formats (e.g. CSV sleep records of Gear4), directly mapping them to meaningful information e.g. sleep measurements. ii) It analyses and aggregates raw sensor data to more meaningful events, performing a primary, early event fusion and detection. For instance, raw consumption data from Plugs are aggregated into appliance usage events according to temporal criteria and consumption patterns. Likewise, Tag motion and presence binary states are translated into events for the associated entities. iii) It implements sophisticated audio and video analytics leveraging the latest developments in computer vision techniques in ambient environments. The output of image analysis regards low-level events such as location and object recognition, but most importantly higher-level events or else daily activities and tasks that the subjects perform. These are later on fused together with other knowledge during semantic interpretation, fostering the framework’s accuracy in activity detection. In accordance to the sensor layer, each device and data type format is handled by a dedicated processing component as shown on Table 1. Each component encapsulates different software dependencies and capabilities, addressing data and platform heterogeneities and utterly unifying the extracted information. 3.2.1. Image and Audio Analysis Human Activity Recognition (HAR) employs a set of existing computer vision techniques described in (Avgerinakis et al., 2013). HAR techniques are optimized for images from ambient spaces such as those provided by the IP and depth cameras in DemaWare2. The component’s analysis output is exclusively high-level activities and daily tasks such as preparing a meal, eating and washing the dishes. Still, even such high-level activities can be further fused and filtered by semantic interpretation on the next processing layer of the framework, as described in the next section. Notably, the analysis is moderately time-demanding so it is performed at a scheduled point within a day or upon invocation, according to deployment scenarios. A sample of running HAR analysis in a deployment scenario is shown in Section 4 (Fig 5). Complex Activity Recognition (CAR) is the given name of another set of computer vision techniques described in (Romdhane et al., 2013). In contrast to HAR, CAR focuses mainly on detecting atomic events, which regard the subject’s location in the three-dimensional scene. This is done by using depth data and images from an ambient depth camera, in conjunction with a predefined model of areas or “zones” e.g. tea, phone, fridge and medication zone. The ambient cameras naturally survey the entire area of interest while the zone models are annotated during the installation and configuration phase. CAR also capitalizes in posture detection such as sitting, standing, lying, walking and even falling. CAR then performs its own internal rule-based fusion combining such atomic events with temporal constraints to infer higher-level activities such as preparing tea, talking on the phone and taking medicine. CAR processing is performed in real-time and detection results are immediately pushed to the platforms knowledge base. A running example of CAR analysis from the actual deployment can be seen on Fig 4. The Wearable Camera Processing Unit (WCPU) is a dedicated computer vision module based on learning for the videos obtained through the GoPro camera, as presented in (Boujut et al., 2012). The models developed so far are aimed at recognizing objects, activities and rooms within the end-users home. Each model is therefore trained and put to use for a specific home or building, after a thorough annotation process. WCPU is computationally

10

demanding and therefore, is performed offline, using a dedicated CUDA11 parallel-processor. Offline Speech Analysis (OSA) is also a learning-based technique targeting audio data, as presented in (Satt et al., 2013). The high quality audio collected in interviews is fed to OSA together with interview metadata, such as marking the duration of each answer, annotated using the framework during the recordings. The component then uses standard models to extract indicators such as the speaker’s mental state ranging from healthy, early and advanced dementia. 3.2.2. Physical Activity, Stress, Sleep and Lifestyle Monitoring Apart from image and audio analytics, the rest of the sensor data can be characterised as more primitive. However primitive they may be, this data proves valuable for fusion in order to verify and foster high-level event detection on their own or in conjunction with other analytics. Such sensor data include physical activity monitoring, measured as moving intensity, stress level, various measurements regarding sleep and lifestyle monitoring. The latter category includes diverse sensor data from daily objects, presence detection and usage of electrical appliances. Each of the above modalities is provided by a different processing component, dedicated to sensors and data formats in DemaWare2. The Wearable Wristwatch Processing Component (WWPC) is a library provided by the manufacturer of the DTI2 wristwatch. Its purpose is to process binary DTI-2 files retrieved from the watch and extract meaningful information. In detail, sensor measurements within the files that regard accelerometer movement in the threedimensional space and skin conductance are transformed into physical activity and stress level respectively. To do so, the library employs internal signal filtering techniques and establishes per-individual statistical baselines for those measurements. Based on that, it extracts moving intensity and stress level in a range of zero to five. These values are then examined by either clinical staff empirically or even by semantic interpretation for automatic problem detection, as presented in the next subsection. The UP24 Processing Component (UP24PC) is a processing library for the UP24 wristband data. The wristband data is retrieved from the cloud via the secure proprietary API, using OAuth v2. The component associates this data with a user account on the framework’s side and maps received steps, distance, calories, active time and idle time to the universal system representation in DemaWare2 for storage. E.g. steps are mapped to moving intensity, the system’s universal representation of physical activity intensity, directly comparable with DTI-2 measurements. The Sleep Clock Processing Component (SCPC) is a software library that interprets sleep recordings from the Gear4 sleep sensor. The underlying sleep recordings in CSV format are parsed and mapped to universal sleep measurements such as light, deep and REM sleep segments, awake and away from bed segments. The segments are aggregated to calculate and store values interesting from a clinical point of view such as total sleep, total light sleep, total deep sleep and total awake time durations. It also counts sleep interruptions, sleep latency (time in bed until falling asleep), falling asleep time and time of waking up. The Aura Processing Component (APC) extracts the same universal sleep measurements and aggregated statistics for Aura, the second sleep sensor in DemaWare2. As mentioned above, the measurements are retrieved via the proprietary third-party API, in a secure manner, and appointed to the associated end-user in the DemaWare2 knowledge base. Similarly to Gear4, raw measurements and sleep phases are aggregated to find daily total amounts for deep, light and REM sleep, as well as awake time. The resulting knowledge representation is again sensoragnostic to the clinician, coming either from Gear4 or Aura, providing interoperability. The Complex Plug Event Processing (CPEP) allows the necessary aggregation of raw power usage data to more meaningful, appliance usage events in the system. In detail, the library constantly monitors the totality of Plug WSNs in a certain local deployment. A model of temporal and power threshold rules per-appliance/sensor are provided at the time of installation and configuration. Notably, the thresholds are found empirically after a few minutes of observing each electrical appliance to determine the thresholds, eliminating power fluctuations. The thresholds in conjunction with power usage monitoring are used in CPEP to extract and immediately commit appliance usage events in the knowledge base, in other words the start and end times of using the fridge, the TV, a boiler, a vacuum, lighting in the bathroom etc. The Tag Processing Component (TPC) aggregates sensor events from all Tag sensor networks in a DemaWare2 deployment. Similarly to Plugs, Tag sensor events need to be interpreted and mapped to more meaningful 11

CUDA online: http://www.nvidia.com/object/cuda_home_new.html

11

information. For this reason, each Tag sensor is associated with a higher-level object- or location-related event to be triggered after a given temporal threshold to avoid noisy detection. E.g. when an object motion sensor attached to the TV remote is activated for a long enough time interval, the TPC commits a TV remote moved event to the knowledge base, for as long as the movement has lasted. Taking into account also presence sensors and door/window sensors, similar events include the movement of a kettle, box of pills, the vacuum, a drug cabinet, presence in the bathroom, living room and kitchen, and an open door event. 3.3. Semantic Interpretation Semantic interpretation serves a fundamental, twofold role within the overall DemaWare2 framework:  It provides the RDF knowledge structures that encode in a formal way domain knowledge and heterogeneous information collected by sensors and analysis components, such as events and observations.  It affords automated reasoning mechanisms for the high-level interpretation of the behavior of users via the integration and semantic fusion of the information made available through monitoring. More specifically, semantic interpretation uses as input sensory information regarding physiological characteristics (e.g. the individual’s physical activity) and actions performed by individuals (e.g. objects they interact with). Based on this information and utilizing background knowledge and automated reasoning techniques, it performs semantic correlation and fusion of the available inputs and generates the high-level interpretation of the individuals’ behavior (e.g. nocturia problems, meals at irregular times and/or in inappropriate places, difficulties in falling asleep due to excess noise levels, etc.), or problematic situations, e.g. missed activities, activities with long duration, sleep problems, etc. In the following sections we describe the basic technologies that underpin the implementation of the intelligent decision making mechanisms in DemaWare2. 3.3.1. Knowledge structures and vocabularies The knowledge structures provide the schema for capturing information relevant to the domain, such as:  Atomic activities and measurements that are detected by means of monitoring, such as physical activity levels, location, posture, objects used, etc.  Constructs describing how atomic information can lead to the derivation of information pertinent to the high-level behavior interpretation.  Problems and situations that the various stakeholders need to be informed about, such as missed meals, excessive physical activity levels, insufficient social activity interactions, nocturia, etc. All modelling capabilities in DemaWare2 have been designed with a minimum of semantic commitment to guarantee maximal interoperability. As such, our ontologies can be easily aligned with relevant foundational RDF/OWL ontologies, such as SEM (Van Hage et al., 2011), DUL12, LODE (Shaw et al., 2009) and Ontonym (Stevenson et al., 2009a), reusing existing conceptual models and vocabularies for modelling different aspects of activities, e.g. entities, places, etc. The DemaWare2 ontologies along with LODE-aligned sample observation datasets are available online13. Events, Activities and Measurements The Event class is the top-level class of the hierarchy and is currently specialized into two subclasses, namely Activity and Measurement. Individuals belonging to the Event class (and its subclasses) need to be instantiated along with the following object properties assertions:  startTime/endTime: it indicates the start/end time of the event.  hasAgent: this property captures agentive information, i.e. the actor of the event, as well as any other evententity relationship, e.g. the relation between a temperature measurement and the room referred to.

12

DOLCE+DnS Ultralite (DUL) ontology, http://www.loa.istc.cnr.it/ontologies/DUL.owl The DemaWare2 ontologies, a sample dataset of observations (aligned with LODE) and an open implementation of semantic fusion for activity recognition can be found online at http://www.demcare.eu/results/ontologies 13

12

The Activity subclass is the root of the activity hierarchy and represents the possible activities the individual may engage into. It currently has two subclasses, namely AtomicActivity and ComplexActivity that correspond to the activities detected by monitoring and activities inferred by high-level analysis, respectively. The Measurement subclass has two subclasses, namely PhysiologicalMeasurement and AmbientMeasurement, and represents the possible physiological and ambient measurements that may be monitored. Individuals of the Measurement class are instantiated with two assertions, one stating the quality being measured ( hasQuality) and one stating the measured value (hasValue). Complex Activity Models As described in the introduction, the majority of the ontology-based (knowledge-driven) approaches for fusion define strict contextual patterns usually in the form of complex class ontology axioms or rules. For example, a tea drinking activity in the kitchen that is inferred on the basis of detecting a drinking event while the person is sitting in the table zone and uses a spoon and a tea cup could be modelled in the OWL ontology language as: DrinkTea = Activity and (actor only (Person and (uses some TeaCup) and (uses some Spoon) and (action some Drink) and (posture some Sitting) and (in some TableZone)))

In this case, however, if the sensor (e.g. the video analysis module) fails to detect the tea cup object, then the axiom will not be satisfied, and thus, the system will fail to derive the DrinkTea activity. Moreover, many activities are carried out differently even by the same person. Thus, the use of strictly structured background knowledge relevant to the presence and order of activities or their temporal boundaries is not always able to effectively capture and reason about the context. In DemaWare2, domain knowledge about complex activities is captured through context dependency models that encapsulate correlations among low-level observations and complex activities. Their role is to provide in an abstract (loosely-coupled) manner the background knowledge required to detect the complex activities of the domain. The context dependency models are defined in terms of reusable, light-weighted ontology patterns that consist of the following knowledge structures:  ContextDependency: Top-level container class for storing dependency correlations.  :dependency: Ontology property that designates the low-level observation types of the dependency.  :describes: Ontology property that designates the complex activity of the dependency. For example, the context of the DrinkTea activity, as it has been described in the previous section, can be modelled as a context dependency using the following ontology instance (in the Turtle 14 syntax): :drink_tea a :ContextDependency ; :dependency :Sitting, :TableZone, :Spoon, :TeaCup. :Drink ; :describes :DrinkTea .

Problems Fig 2 depicts an excerpt of the ontology defined for representing information about problems encountered by individuals (in home). Problems are defined as direct or indirect instances of the Problem class and can be associated with one or more events that are considered as contributing factors ( possibleContributingFactor).

14

http://www.w3.org/TeamSubmission/turtle

13

Fig 2. Excerpt of the Problem class hierarchy

3.3.2. Ontology-based Fusion and Activity Detection The context dependency models described above serve as input to the semantic interpretation procedure for the recognition and classification of complex activities by analyzing traces of low-level observations and grouping them into meaningful situations. The interpretation algorithm consists of three steps: (a) definition of partial context, (b) identification of contextual links and (c) recognition and classification of situations that are analyzed next. Each step is briefly described below15. Partial Contexts Each observation is assigned with a partial context that captures information about the neighboring observations and the most plausible complex activities that the observation can be part of. More specifically, the local context 𝑙𝑖 of an observation 𝑜𝑖 ∈ 𝑂 is defined as the tuple ⟨𝑜𝑖 , 𝑁𝑖𝑟 , 𝐶⟩ , where 𝑁𝑖𝑟 be the set of observations 𝑜𝑗 in the neighbourhood of 𝑜𝑖 that either overlap with 𝑜𝑖 (𝑜𝑖 ∘ 𝑜𝑗 ) or are the 𝑟-nearest to 𝑜𝑖 (𝑛(𝑜𝑖 , 𝑜𝑗 ) ≤ 𝑟), based on their temporal ordering. 𝐶 is the most plausible complex activity classification of 𝑙𝑖 , derived by computing the 𝜑 similarity between the set with the most specific observation classes 𝑁𝑖𝑟 and the observations types defined in the :dependency property of the context dependency 𝑑𝑘 . The 𝐶 class corresponds to the :describes property of 𝑑𝑘 . The 𝜑 similarity is computed as:

15

An implementation of the semantic fusion for activity recognition method in DemaWare2, along with ontologies and a sample dataset is provided online at http://www.demcare.eu/results/ontologies

14

|𝑈(𝑛) ∩ 𝑈(𝑐)| ∑∀𝑛∈𝑁𝑟 max [ ] |𝑈(𝑛)| ∀𝑐∈𝑑𝐶 𝜑(𝑁 , 𝑑𝐶 ) = |𝑁 𝑟 | 𝑟

where 𝑈(𝐶) is the set of superclasses of 𝐶. Intuitively, 𝜑 captures the local plausibility of an observation 𝑜𝑖 to be part of a complex activity 𝐶. If 𝜑 = 1, then all the classes in 𝑁 𝑟 appear in some 𝑑𝐶 and, therefore, it is very likely that the local context is part of the complex activity 𝐶. Context Links The next step is to define context links, that is, links among partial contexts that will form the final situations for 𝐶𝑚

activity classification. A context connection 𝑙𝑖 → 𝑙𝑗 is defined between two partial contexts 𝑙𝑖 and 𝑙𝑗 only if they share the same complex domain activity classification 𝐶𝑚 , and the observation 𝑜𝑖 of 𝑙𝑖 belongs to the neighbors of observation 𝑜𝑗 of 𝑙𝑗 . The rationale behind context links is to group neighbouring observations with respect to their classification classes, capturing contextual dependencies among their partial contexts. More specifically, if two partial contexts have the same classification class, then it is very likely that the corresponding observations belong to the same complex activity. When the identification and assignment of context links is completed, each partial context is linked to any other relevant partial context in the neighbor, enabling the segmentation of the initially provided observation traces into groups. Classification of Situations By traversing the paths defined by the context links, the initial trace of observations is segmented into situations. In this final step, we compare the observation types of each situation against the observation types of the available context dependencies (:dependency property). Similarly to 𝜑, we define the function 𝜎 that defines the similarity of a context dependency 𝑑𝑐 against the observation types 𝑂𝑏𝑠𝑇 of a situation as: |𝑈(𝑛) ∩ 𝑈(𝑐)| ∑∀𝑛∈𝑑𝐶 max [ ] |𝑈(𝑐)| ∀𝑐∈𝑂𝑏𝑠𝑇 𝜎(𝑑𝐶 , 𝑂𝑏𝑠𝑇 ) = |𝑑𝐶 | If 𝜎 = 1, then all the classes in 𝑑𝐶 appear in 𝑂𝑏𝑠𝑇 , meaning that the situation can be considered identical to the context descriptor 𝑑𝐶 , and, therefore, to the complex class 𝐶. Otherwise, the context dependency 𝑑𝐶 with the larger 𝜎 similarity is selected as the final complex activity classification of the situation. 3.3.3. Problem Detection The primary aim of DemaWare2 is to promote enablement and safety of the individual. This is accomplished via not only monitoring a person’s daily life, but also provide clinically valuable feedback to the person, the attending clinician and respective carers. To this end, DemaWare2 implements a set of rules for the detection of clinicallyrelevant problems, to subsequently generate respective alerts and notifications (e.g. missed meals, excessive napping, insufficient utterances and communication attempts). Such a problem-detection mechanism should be as interoperable as possibly, with the ability to be implemented on top of any semantic repository (triple store), based on universal standards. It would also require some sort of temporal reasoning (for filtering based on duration or Allen operators) that is beyond native OWL reasoning capabilities. It should also possibly facilitate the creation of new individuals in the triple store, to represent problem generation knowledge. The implementation of assessment rules for the derivation of clinically relevant situations that indicate possibly problematic behavior in DemaWare2 is based on the SPARQL query language. SPARQL is the W3C recommended standard for querying RDF graphs and it is supported by existing state-of-the-art RDF triple stores, such as GraphDB 16 , employed in DemaWare2, providing a scalable and optimized SPARQL query engine. SPARQL 16

http://ontotext.com/products/graphdb

15

functions are implemented in DemaWare2 in order to check the temporal extension of the detected activities. Moreover, the use of SPARQL further facilitates generation of new individuals in the KB in the context of problem detection, which is handled by the native semantics of the SPARQL CONSTRUCT graph pattern, updating the KB with the derived triples. Although it is mostly known as a query language for RDF, by using the CONSTRUCT graph pattern it is able to define SPARQL rules (SPIN 17) that can create new RDF data, combining existing RDF graphs into larger ones. Such rules are defined in the interpretation layer in terms of a CONSTRUCT and a WHERE clause: the former defines the graph patterns, i.e. the set of triple patterns that should be added to the underlying RDF graph upon the successful pattern matching of the graphs in the WHERE clause. Triple variables are marked by the use of “?”. As an example, we present the following SPARQL rule that derives instances of the NocturialProblem class by counting the number of NightBathroomVisit instances that are temporally contained in a NightSleep activity for a specific day. The rule generates an instance of the NocturiaProblem for each grouped result that contains three or more NightBathroomVisit instances. The detected problems are then highlighted to the clinicians, informing them about nocturia problems that need to be further investigated. CONSTRUCT{ ?new a :NocturiaProblem; :isProblemOf ?pwd; :date ?clinicalDay. } WHERE { { SELECT ?ns ?pwd (COUNT(?nbv) as ?counter) WHERE { ?ns a :NightSleep; :hasAgent ?p; :startTime ?s1; :endTime ?e1. ?nbv a :NightBathroomVisit ; :hasAgent ?p ; :startTime ?s2; :endTime ?e2. FILTER(:contains(?s1, ?e1, ?s2, ?e2)). } GROUP BY ?ns ?p HAVING (COUNT(?nbv) >= 3) } BIND (:new(?ns, ?p) as ?new) . FILTER NOT EXISTS {?new a [] . } . BIND (:extractClinicalDay(?ns) as ?clinicalDay) . }

3.4. Applications for Dementia Ambient Care The Service Oriented Architecture (SOA) is one of the dominant paradigms in Ambient Intelligence, since it provides the necessary abstractions from lower level functionality and the required interoperability for pervasive applications. It also provides remote and platform-independent access. DemaWare2 follows the WSDL W3C standard18 to universally describe and expose all processing components and functions (using JAX-WS19). WSDL syntactically describes each service and underlying operations, allowing virtually any client to access them over the Web. On top of that, it allows complex constructs to be defined as operation input or output rendering data more readily interpretable. While WSDL freely allows the design and development of any AAL application, this work showcases dementia ambient care as an application domain for DemaWare2. The Controller module implements a standard WSDL client which manipulates and timely orchestrates the frameworks functions, acting as a back-end, on behalf of various web applications for dementia assessment, monitoring and intervention. These applications have been designed in collaboration with professional psychologists, carers and even according to feedback from people with dementia. Each application is tailored to a specific usage scenario and purpose, which are closely linked to two deployment scenarios, the lab trials and home care. In short, assessment 17

http://spinrdf.org WSDL W3C Recommendation: http://www.w3.org/TR/wsdl 19 The Java API for XML Services (JAX-WS): https://jax-ws.java.net/ 18

16

Fig 3. The DemaWare2 Home application, showing a daily view of automatically detected problems (upper left), patterns of sensor measurements for sleep and physical activity (upper right) and a timeline of detected events (bottom), from the actual home user 1 deployment.

applications allow clinicians and technical staff to conduct sensor-aided trials and view results and statistics. The framework in this scenario functions close to real-time and provides constant feedback. Longer-term monitoring is enabled by extended and adaptable visualization of sensor-data, activity, object and location recognition from analysis and fusion. Interventions are enabled through detecting and monitoring behavioural patterns and problems. People under care are also provided with applications to monitor their daily living, but only if they are willing and capable to do so. Also, this interaction with the framework contributes only to feeling included and is not part of any intervention. Therefore, it is left outside the scope of this work. A view of the DemaWare2 Home application is shown on Fig 3, displaying a digested view of critical activities and measurements. The upper right shows the main measurements for sleep and physical activity so that clinicians may observe patterns. To view more details, one can browse through a timeline of detected events, before and after fusion, in the bottom of the page. Other views allow choosing any time range, resolution and type of activity or measurement. The system also assists the clinician by automatic problem detection as shown on the upper left of this view. The next section describes two application scenarios showcasing the framework’s applications for the care of dementia.

4. Deployment Two use case scenarios with according applications and real world deployments have been developed: the lab trials, which target short-term assessment of an individual’s mental state, and home care which aims for longer-term monitoring and technology-enabled clinical interventions. Each scenario is presented below.

17

4.1. Lab Assessment Trials The goal of the lab trials, from a clinical perspective, is to enhance dementia assessment during short lab visits through ambient intelligence technologies. Traditionally, during such visits the psychologist utilizes questionnaires to assess an individual’s mental state and potency ranging from healthy, to MCI and Alzheimer’s disease (AD). Traditionally, this method is somewhat flawed as it is based solely on the input and perspective of the individual or the one of his relatives. Some clinics can afford to conduct short trials where the individual performs a set of simulated daily tasks, while recording his performance. This process is tedious and error-prone for the clinical staff to capture in detail, but ambient intelligence technologies, using sensors and advanced analytics can enhance it by performing automatic assessment. This has motivated the use of DemaWare2 in such a clinic, namely the day center of the GAADRD, in Thessaloniki, Greece and used to effectively assess 98 participants. The lab trial protocol includes three phases: 1) combinations of walking and counting 2) oral interview and 3) a simulation of daily tasks in a semi-directed manner. From a technological perspective, the deployment in such a constrained lab environment has served as an ideal incubator to test the framework in a subset of simulated activities before scaling it up to 24/7 monitoring of a larger set of daily activities at homes. The modular architecture of DemaWare2 enables the use of the required only components, while the web service and knowledge base infrastructure enable interoperability with any set of sensors. For instance, both DTI-2 and UP24 can measure physical activity intensity and store it unanimously in the Knowledge Base. In this case, the heavier and larger DTI-2, which also measures stress, can be tolerated for the duration of the lab trial, without inconvenience and without depleting its battery. Instead, UP24 is selected for home settings but in both situations the system is agnostic of data source and can equally interpret the measurements. Fig 4 shows the actual sensor network and application deployment in the lab trial scenario. The wristwatch is worn during the entire visit, but mainly measures the active phases, 1 and 3, as does the deployed image analysis in the form of CAR and HAR. Phase 2 exclusively uses audio analysis, while phase 3 utilizes activity recognition from image, appliance usage from Plugs and object motion from Tags. It also uses applications in designated I/O devices, namely a phone simulating application on a smartphone and an ATM simulating application on a tablet. Naturally, the sleep sensor is not required. Lab results are presented in Section 5.

Fig 4. Sensor and WSN deployment in the lab trials scenario

18

4.2. Home Monitoring and Interventions The home setting extends and, in a way, complements the lab setting in terms of clinical and technical goals. On one hand, lab trials have already been performed on more than 90 users in the same installation, while home deployments are hard to reproduce/relocate. On the other hand, the home setups aim for longer-term monitoring and are, therefore, more tolerant for small timestamp offsets but require more stability and robustness. In turn, this will aid timely and accurate intervention while monitoring longer term improvement. From a technical perspective, the deployment scales up in sensor numbers and utilizes a central server to upload observations from distributed home deployments from the four homes (two at a time). The home deployments utilize different components of the framework, once again verifying its modularity. An ambient camera for HAR analysis and two wireless sensor networks are deployed in each home: Plugs to monitor appliance usage and Tags to monitor object manipulation and human presence. The UP24 wristband is worn at all times, enabling unobtrusive, comfortable physical activity measurement. Sleep is monitored by Aura, transparently installed under the participant’s mattress. 4.2.1. Methodology A clearly-defined methodology for the home setups has been established, by combining psychological methods with the framework’s capabilities and traits. The methodology does not entirely fit the technical scope of the paper, which revolves around the framework as an enabler, but is briefly presented here for the sake of completeness. The deployment and evaluation strategy involves the following steps: a)

Eligibility criteria include a mild AD or MCI diagnosis, which followed the criteria of the NINCDSADRDA, DSM-IV and criteria of Petersen respectively, conducted by a psychologist and a neurologist. Past participants in lab trials are ideal candidates as they have experience with the sensors. The candidates should also live alone, as most components, sensors and image analysis are unable to discriminate actors. b) Sensor setup and monitoring starts upon recruitment with full sets of sensors, while the nature of each device, privacy concerns and its benefits are thoroughly explained to the participant before giving consent. Using a combination of initial (approximately a week-long) sensor monitoring and psychological interviews, the initial set of interventions is designed and the sensors are adjusted accordingly. c) From then on, the chosen interventions are closely monitored by the psychologist, showing the participants progress and discussed during his weekly visits. The psychologist also assists in charging the UP24 wristwatch during those visits. If necessary, the interventions and the sensors may be re-adjusted directly, as knowledge transport and visualization dynamically adapts to content. d) Intense monitoring ceases when the patient is consistently improving for at least a month, assessed by the system, interviews and revisiting mood (NPI, GDS, Hamilton, BDI, and BAI) and cognitive state assessment (MMSE, MoCA, TEA, RAVLT, FAS). Different exit strategies are implemented e.g. leaving the sleep and physical activity sensors together with the user and holding less frequent clinical interviews. 4.2.2. Installations Following the methodology, four home setups have been so far established in Thessaloniki, Greece. All homes include a common set of sensor types as shown on Fig 5. After signing informed consent, the camera is placed at an interesting yet unobtrusive place of the home to be used only by automatic computer vision techniques, namely HAR. What is mostly different in each setting is the set of sensor networks for lifestyle monitoring, i.e. Plugs and Tags. According to clinical feedback from interviews the sensors may be adjusted or extended. For instance, our home user 1 has been diagnosed with problems regarding daily routines including cleaning. Therefore, the sensors have been placed on cleaning items such as the vacuum, the iron, the dish washer and the washing machine to monitor both their movement and usage. On the other hand, home user 2 had no such problems but had been watching TV in two rooms, leaving them on for most of the night, as observed through detailed monitoring. Evaluation results from all four installations are presented in the next section.

19

Fig 5. Deployment in each home, including tailored user interfaces

5. Evaluation Since the framework embodies an interdisciplinary approach, it was evaluated both from an AAL research and a clinical perspective, in both deployment scenarios. Firstly, we evaluate the effectiveness of activity recognition through fusion of sensor data and existing multimedia analytics. Secondly, we present discussion and clinical results regarding clinical assessment, in the lab scenario, and monitoring and interventions, in the home care scenario. 5.1. Fusion and Activity Recognition The ontology-based fusion and activity recognition capabilities of DemaWare2 have been evaluated both in lab and home environments. Regarding privacy concerns, image and audio recordings are never viewed but rather used to train models and predict outcomes. Lifestyle sensor recordings (e.g. presence and object motion) are entirely anonymized i.e. it is impossible to infer one’s identity from the database of observations. Ground truth has been obtained, by one human annotator viewing image data, in the lab and during certain periods and locations in the home, after informing participants explicitly and thoroughly about the process. For all the above terms, all lab and home participants gave written informed consent and willingness for their data to be used in this research or provided to third-parties for collaboration. The resulting datasets are available online20 for benchmarking (bearing no identity information and being compliant to the above terms). Notably, while complex activity recognition datasets for benchmarking are gaining popularity, this does not apply for datasets suitable for semantic event fusion. Most existing implementations are either not publicly available (Riboni et al., 2016) (Okeyo et al., 2014), too use case-, domain- or vocabulary-specific or contain no temporal information21 and embedded semantics (Helaoui et al., 2013), preventing adaptation for full-fledged fusion in our domain. This fact is further supported by recent state-of-the-art frameworks also refraining from experimental comparison to other approaches as a common benchmark is lacking (Riboni et al., 2016) (Okeyo et al., 2014). For this reason our method, ontology and dataset are provided online to stimulate benchmarking in the field.

20 21

Datasets provided online: http://www.demcare.eu/results/datasets http://everywarelab.di.unimi.it/palspot

20

Regarding metrics, we use the True Positive Rate (TPR) and Positive Predicted Value (PPV) measures, which denote recall and precision, respectively, to evaluate the performance. These measures are defined as: 𝑇𝑃𝑅 =

𝑇𝑃 𝑇𝑃 , 𝑃𝑃𝑉 = 𝑇𝑃 + 𝐹𝑁 𝑇𝑃 + 𝐹𝑃

where True Positives (TP) is the number of ADLs correctly recognized, False Positives (FP) is the number of IADLs incorrectly recognized as performed and False Negatives (FN) is the number of IADLs that have not been recognized. The Jaccard coefficient metric has been used for determining whether an activity detection is considered successful (TP) or not by computing the degree of the temporal overlap between a ground truth activity G and the detected activity A as: 𝐴∩𝐺 𝐽𝐶(𝐴, 𝐺) = 𝐴∪𝐺 where 𝐴 ∩ 𝐺 denotes the overlapped interval, e.g. in seconds, and 𝐴 ∪ 𝐺 the temporal union of the two activity intervals. The activity A is considered as TP with respect to the ground truth activity G only if 𝐽𝐶(𝐴, 𝐺) ≥ 𝜑, otherwise it is a FP. In our experiments, we have used 𝜑 = 0.2, according to the OV20 evaluation criterion (Gaidon et al., 2013), a threshold commonly used in the literature (Avgerinakis et al., 2015; Ma et al., 2013). FN denotes a ground truth activity for which no activity was detected following the OV20 criterion. The context description model encapsulating domain knowledge to recognize activities in each of the two environments are given in the following sections along with results. 5.1.1. Lab environment The context dependency models in the lab involve four designated activities. They were derived after several iterations with the clinicians in order to clearly define when an activity should be considered successfully completed or not in the context of the trial. For example, in order to recognize that the participant has answered the phone, clinicians suggested that the constituent activities “phone moved”, “phone zone”, “phone object” and “talking” need to be recognized by the components and fused by the high-level interpretation module. Based on the elicited knowledge, the four context dependency models in Table 2 were defined. The models involve ontology concepts relevant to: detected scene objects, e.g. PhoneObject (from wearable video analysis), the person’s location, e.g. PhoneZone (from wearable video, depth camera and presence sensors), the objects used, e.g. PhoneMoved (from Tag sensors), electric utility usage, e.g. KettleOn (from Plug sensors), and posture information, e.g. Sitting (from video analysis). This way complimentary modalities are included in each activity e.g. presence sensors, wearable and ambient cameras are fused to provide location. Notably, the framework does not try to discriminate or guess one of four activities at each given moment (as often in machine learning-based methods), but only to recognize activities when they occur based on observations (the final activity recognition is based on fusion, not on learning/prediction). This way, the number of activities to recognize (four in this case) does not really directly influence the recognition performance, at least as long as activities involve different context (locations, objects etc.) and are not frequently interleaved. Table 3 summarizes the performance of DemaWare2 on the dataset of all 98 participants. The average recall and precision for activity recognition is close to 82%, except for the quite low performance in EstablishAccountBalance. This is attributed to the low performance in the three required concepts based on image recognition (Sitting, PenObject and AccountObject) which are harder to recognize, whereas other high-performing activities include just one or two. Also, this activity lacks entirely the electric utility usage, a highly confident modality, whereas the more sensor modalities are involved, precision and recall increase. Therefore, involving more modalities can improve results until state-of-the-art in image recognition methods improves. 5.1.2. Home environment The DemaWare2 home context dependency models define activities of interest for clinical monitoring and validating interventions. While in the lab scenario, ground truth via human annotation from images can be performed for the entire dataset, in the home only a limited set is provided due to time and space restrictions, i.e. 24/7 recordings producing a large image volume. Space and imposing concerns excluded some activities e.g. the bathroom, while the participants were notified before the annotation period. From the set of monitored activities, as suggested by

21

clinical experts, only the six evaluated ones are considered here. The context models follow the same successful principles from the lab as presented on Table 4, but further utilize a sleep sensor, i.e. in the Asleep concept, and the wearable physical activity sensor, i.e. in HighActivity concept. DemaWare2’s ADL activity recognition performance in the homes has been evaluated over a 31-day annotation period. The results shown on Table 5 seem optimistic for most activities with an average 83% recall and 76% precision. Notably, some activities differentiate themselves either positively or negatively. Sleep is the most efficiently detected one (100% recall), due to the bed sensor under the bed, still endorsed by image analysis. However, the least efficiently recognized activity was cooking, a more challenging task, due to it usually being performed in an interleaved manner. Unlike in the lab, where activities are naturally performed mostly in a sequential, uninterrupted manner, the home is a much more open, realistic environment. Therefore, the assumption that only a single activity is performed at a given moment, has been shown to fall short in such environments. On the contrary, in real-world situations, activities can be performed in an interleaving manner, pausing an activity temporarily to perform another. For example, cooking or watching TV are instances of activities that are initiated and then halted many times to do something else. In these cases, the interrupted contexts are recognized by our algorithms as individual activities (segmented in a way), dropping performance. Cooking and Watching TV are the longest and most interleaved activities, yielding a precision of 68% and 39% respectively. Still, TvZone managed to keep recall high (94%), while the same was not applicable for cooking as the kitchen is a multi-tasking location. Key challenges in this context involve the recognition of the start and end timestamps of all the activities involved and the derivation of the contextual interval when each activity was active, e.g. to classify interrupted instances of the same task as a single activity. We are currently investigating non-monotonic reasoning solutions to the aforementioned problem, investigating a combination of defeasible logic to handle conflicts and classify interrupted instances of the same task as a single activity. 5.2. Clinical Assessment, Monitoring and Intervention The clinical evaluation of the framework regards its capabilities and the fulfillment of clinical requirements in both deployment scenarios. While the entire clinical perspective is outside the scope of this work, in this paper we present in short the major clinical acceptance points and added-value contributions towards the diagnosis and care of the disease. Notably, the inclusion criteria for both lab and home scenarios included AD and MCI diagnosis followed the criteria of the NINCDS-ADRDA, DSM-IV and criteria of Petersen respectively. The lab recruitment aimed at an equal distribution of healthy, MCI and AD participants, while at home, MCI and mild AD participants living alone were preferred for handling technology. Age and gender were disregarded. Regarding the lab evaluation trials, recruiting 98 participants (27 AD, 38 MCI, 33 Healthy), aged 60-90, showed optimistic results. An achievement of the platform is the ability to autonomously conduct trials in a robust and timely manner. A single psychologist is able to simultaneously handle the system and interact with the participant, effectively examining up to five individuals per day. From the participant’s point of view, the equipment is almost entirely ambient reaching high acceptance and participation levels. Meanwhile, the actual assessment has already been proven to accurately distinguish mental states, based on the statistically significant difference in the duration and the number of attempts for certain activities. MCI individuals in average complete the phone call and the account payment activities, as opposed to AD ones, allowing the system to distinguish them with a mean accuracy of 73.67%. Similarly, statistics from the use of applications for paying a bank bill and making a phone call, are able to distinguish MCI and AD individuals from healthy ones, allowing the system to reach an 84% accuracy in distinguishing between healthy, MCI and AD. Regarding the residential pilots, four users, two diagnosed with amnestic and multi-domain MCI and two with mild dementia, have fully accepted and benefited from DemaWare2-driven interventions. The first user, recruited on March of 2015, was diagnosed with mild dementia and depression via MMSE. The psychologist has used the DemaWare2 monitoring interface, which immediately confirmed problems, such as insomnia via a large number of sleep interruptions. Detailed monitoring of all events revealed the neglect of daily tasks such as cleaning and the lack of exercise. The tailored intervention introduced daily walks, monitored through the wristband, weekly cleaning monitored through activity recognition entailing the usage of the vacuum, iron and washing machine and taking pills, monitored through image analysis, Tag and Plug sensors. The clinician has verified that the intervention has been followed to a great extent, by monitoring activity recognition in the system. Most importantly, problems

22

Table 2. Context dependency models for the lab environment Activity Concept PrepareDrugBox AnswerPhone EstaclishAccountBalance PrepareHotTea

Context dependency set DrugBoxMoved, DrugZone, DrugBoxObject, PillBoxMoved, PillBoxObject PhoneMoved. PhoneZone, PhoneObject, Talking Sitting, AccountZone, AccountMoved, AccountObject, PenObject KettleOn, TeaZone, KettleObject, TeaBagMoved, TeaBagObject, CupObject, CupMoved

Table 3. Precision and recall for activity recognition in the lab environment Activity PrepareDrugBox PrepareHotTea EstablishAccountBalance AnswerPhone

Recall (TPR) 0.833 0.816 0.292 0.827

Precision (PPV) 0.813 0.796 0.393 0.814

Table 4 Context dependency models for the home environment Activity Concept MakeHotTea Cooking PrepareDrugBox WatchTV SleepOnBed ExitHouse

Context dependency set KettleOn, TeaZone, KettleObject, TeaBagMoved, TeaBagObject, CupObject, CupMoved CookerOn, CookerZone, CutleryObjects DrugBoxMoved, DrugZone, DrugBoxObject, PillBoxMoved, PillBoxObject, DrugCabinetMoved TvOn, TvRemoteMoved, TvRemoteObject, TvZone Asleep, BedZone DoorOpen, ExitZone, HighActivity Table 5 Precision and recall for activity recognition in home Activity PrepareDrugBox Cooking PrepareHotTea WatchTV SleepOnBed ExitHouse

Recall (TPR) 0.86 0.61 0.81 0.94 1.00 0.75

Precision (PPV) 0.89 0.68 0.86 0.39 0.88 0.86

detected by semantic analysis have gradually dropped in mid-April, indicating the effectiveness of the intervention through the system, as shown on Fig 3. Weekly interviews confirmed this improvement, maintained by continuing the intervention, while MMSE was used to assess cognitive state. Likewise, the second, third and fourth home scenarios have been established using the same procedure and identifying similar problems and interventions. Improvements appeared in most participants through an increase in physical activity with an accompanying decrease in sleep problems and mood improvement confirmed with interviews and tests. A tablet application for participants, showing simple overviews of total sleep and activity, was used only by the first participant who was willing and able to do so for a couple of weeks, granting a feeling of deeper inclusion and safety. Overall, both clinical staff and participants have agreed that the framework enhanced the interventions. The (MCI and mild AD) participants at home were comfortable and accepting the equipment. In fact, most of the system’s sensors (even the camera) are truly unobtrusive and ambient, as they were in a way hidden in the environment and the participants forgot their presence in a matter of hours. The only slight obstruction is the wearable sensor, an 1cm thick, very light and soft rubber wristband. However, to turn this into an advantage, the clinicians introduced the daily routine of taking it on in the morning and off at bedtime as an intervention. In some cases, they reported feeling greater inclusion as the wristband reminds them that someone is taking care of them. Except from one case where the participant was willing to interface with a tablet device to view information, most of them had no further interaction with the technology, such as charging or switching equipment (the clinicians took care of that during weekly visits). As a result, in simple questionnaires they answered positively as they rated it with 4/5 for ease of use (“Do you believe the system is easy to use?”), 4.5/5 for its simplicity (“Do you find the system simple and

23

straightforward?”), 4.5/5 for learnability (“Do you believe most people would learn quickly how to use it?”) and 4/5 for their confidence using it (“Did you feel confident while using it?”). From the clinician’s perspective, the system increased their confidence and the effectiveness of interventions. Especially the multi-aspect monitoring diversity has steered interventions towards improvement, as both homes made use of combined activity recognition, physical exercise and sleep detection.

6. Conclusion and Future Work DemaWare2 is a holistic AAL framework integrating a variety of sensors, analytics and semantic interpretation with a special focus on dementia ambient care. New, affordable sensor modalities and technologies, such as proprietary cloud APIs, have been integrated seamlessly into the framework, enabling the interconnection of ambient, wearable and lifestyle sensor networks. A set of processing components ranging from sensor analytics for event detection to sophisticated image, video and audio analytics, infusing DemaWare2 with the latest findings in computer vision for activity detection. All knowledge is unanimously stored in a knowledge base, enabling its semantic interpretation for further fusion, aggregation and detection of problematic behaviours. While maintaining its generality in AAL, the framework has been complemented with applications specializing in the dementia care. Following an interdisciplinary approach, clinical methods have been employed in two different deployment scenarios, the lab trials for assessment and the home care scenario for monitoring and interventions. Both scenarios have yielded valuable and optimistic results with respect to accurate fusion and activity detection and clinical value in care. Future directions from this work include the extension of both the framework and its clinical applications. While some applications have focused in close to real-time, such as the lab assessment, the framework can be extended to a complete solution for real-time feedback. In detail, certain modules such as image analysis can detect urgent situations in real-time and implement a pushing mechanism to carers and clinicians, in the direction of mobile health and eHealth. Another extension regards mood and stress assessment, especially in home settings. The market since recently offers many wearable devices to measure heart rate and less often skin perspiration, maintaining their durability and battery life. Such sensors and according processing techniques together with sophisticated audio analysis may prove useful to assess mood and stress in a persistent manner. In parallel to context enrichment, the fusion and interpretation layers need to be further extended to handle the incorporation of new modalities. This translates to the development of knowledge structures able to capture the new context, as well as the adaptation of the underlying reasoning and interpretation layers for advanced decision making. We also plan to incorporate methodologies for the automated extraction of high-level habitual aspects of individuals, such as duration, frequency and onset time of certain activities that reflect the actual behavior and performance in various situations. In that way, we will be able to update the initially defined generic knowledge of the system through patient-tailored views, enabling at the same time the detection of behavior variations that might indicate problematic situations. While the framework has been deployed in numerous locations, it can yet be extended for increased portability and installability. In detail, establishing an open source, IoT-enabled semantic platform, following the latest advances in board computing would allow the platform to be easily deployed in multiple locations. Combined with the infrastructure to push the events on a cloud infrastructure, the framework could constitute a powerful platform for telemedicine and mobile health, combing sensors and sophisticated ambient intelligence techniques such as computer vision.

7. Acknowledgements This work has been supported by the EU FP7 project Dem@Care: Dementia Ambient Care - Multi-Sensing Monitoring for Intelligent Remote Management and Decision Support (contract No. 288199).

24

References Avgerinakis, K., Briassouli, A., Kompatsiaris, I., 2013. Recognition of Activities of Daily Living for Smart Home Environments, in: Intelligent Environments (IE), 2013 9th International Conference on. IEEE, pp. 173– 180. Avgerinakis, K., Briassouli, A., Kompatsiaris, Y., 2015. Activity detection using Sequential Statistical Boundary Detection (SSBD). Comput. Vis. Image Underst. Berners-Lee, T., Hendler, J., Lassila, O., others, 2001. The semantic web. Sci. Am. 284, 28–37. Bettini, C., Brdiczka, O., Henricksen, K., Indulska, J., Nicklas, D., Ranganathan, A., Riboni, D., 2010. A survey of context modelling and reasoning techniques. Pervasive Mob. Comput. 6, 161–180. Boujut, H., Benois-Pineau, J., Megret, R., 2012. Fusion of multiple visual cues for visual saliency extraction from wearable camera settings with strong motion, in: Computer Vision–ECCV 2012. Workshops and Demonstrations. Springer, pp. 436–445. Capone, A., Barros, M., Hrasnica, H., Tompros, S., 2009. A new architecture for reduction of energy consumption of home appliances, in: TOWARDS eENVIRONMENT, European Conference of the Czech Presidency of the Council of the EU. pp. 1–8. Case, M.A., Burwick, H.A., Volpp, K.G., Patel, M.S., 2015. Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA 313, 625–626. Chang, Y.-J., Chen, C.-H., Lin, L.-F., Han, R.-P., Huang, W.-T., Lee, G.-C., 2012. Wireless sensor networks for vital signs monitoring: Application in a nursing home. Int. J. Distrib. Sens. Netw. 2012. Chen, L., Nugent, C., 2009. Ontology-based activity recognition in intelligent pervasive environments. Int. J. Web Inf. Syst. 5, 410–430. Chen, L., Nugent, C.D., Wang, H., 2012. A knowledge-driven approach to activity recognition in smart homes. Knowl. Data Eng. IEEE Trans. On 24, 961–974. Cislo, N., Arbaoui, S., Becis-Aubry, Y., Aubry, D., Parmentier, Y., Doré, P., Guettari, T., Ramdani, N., 2013. A System for Monitoring Elderly and Dependent People in Nursing Homes: the E-monitor’age Concept. Stud Inf. Univ 11, 30–33. De Paola, A., Gaglio, S., Lo Re, G., Ortolani, M., 2012. Sensor9k: A testbed for designing and experimenting with WSN-based ambient intelligence applications. Pervasive Mob. Comput. 8, 448–466. Eisenhauer, M., Rosengren, P., Antolin, P., 2010. Hydra: A development platform for integrating wireless devices and sensors into ambient intelligence systems, in: The Internet of Things. Springer, pp. 367–373. Friedewald, M., Costa, O.D., Punie, Y., Alahuhta, P., Heinonen, S., 2005. Perspectives of ambient intelligence in the home environment. Telemat. Inform. 22, 221–238. Gaidon, A., Harchaoui, Z., Schmid, C., 2013. Temporal localization of actions with actoms. Pattern Anal. Mach. Intell. IEEE Trans. On 35, 2782–2795. Gámez, N., Fuentes, L., 2011. FamiWare: a family of event-based middleware for ambient intelligence. Pers. Ubiquitous Comput. 15, 329–339. Grau, B.C., Horrocks, I., Motik, B., Parsia, B., Patel-Schneider, P., Sattler, U., 2008. OWL 2: The next step for OWL. Web Semant. Sci. Serv. Agents World Wide Web 6, 309–322. Gu, T., Wang, X.H., Pung, H.K., Zhang, D.Q., 2004. An ontology-based context model in intelligent environments, in: Proceedings of Communication Networks and Distributed Systems Modeling and Simulation Conference. pp. 270–275. Helaoui, R., Riboni, D., Stuckenschmidt, H., 2013. A probabilistic ontological framework for the recognition of multilevel human activities, in: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, pp. 345–354. Le Gal, C., Martin, J., Lux, A., Crowley, J.L., 2001. Smart office: Design of an intelligent environment. IEEE Intell. Syst. 16, 60–66. Ma, S., Zhang, J., Ikizler-Cinbis, N., Sclaroff, S., 2013. Action recognition and localization by hierarchical spacetime segments, in: Proceedings of the IEEE International Conference on Computer Vision. pp. 2744–2751. Meditskos, G., Kontopoulos, E., Kompatsiaris, I., 2014. Knowledge-Driven Activity Recognition and Segmentation Using Context Connections, in: The Semantic Web–ISWC 2014. Springer, pp. 260–275.

25

Nevatia, R., Hobbs, J., Bolles, B., 2004. An ontology for video event representation, in: Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04. Conference on. IEEE, pp. 119–119. Okeyo, G., Chen, L., Wang, H., Sterritt, R., 2014. Dynamic sensor data segmentation for real-time knowledgedriven activity recognition. Pervasive Mob. Comput. 10, 155–172. Okeyo, G., Chen, L., Wang, H., Sterritt, R., 2012. A hybrid ontological and temporal approach for composite activity modelling, in: Trust, Security and Privacy in Computing and Communications (TrustCom), 2012 IEEE 11th International Conference on. IEEE, pp. 1763–1770. Patkos, T., Chrysakis, I., Bikakis, A., Plexousakis, D., Antoniou, G., 2010. A reasoning framework for ambient intelligence, in: Artificial Intelligence: Theories, Models and Applications. Springer, pp. 213–222. Riboni, D., Bettini, C., 2011. COSAR: hybrid reasoning for context-aware activity recognition. Pers. Ubiquitous Comput. 15, 271–289. Riboni, D., Bettini, C., Civitarese, G., Janjua, Z.H., Helaoui, R., 2016. SmartFABER: Recognizing Fine-grained Abnormal Behaviors for Early Detection of Mild Cognitive Impairment. Artif. Intell. Med. Riboni, D., Pareschi, L., Radaelli, L., Bettini, C., 2011. Is ontology-based activity recognition really effective?, in: Pervasive Computing and Communications Workshops (PERCOM Workshops), 2011 IEEE International Conference on. IEEE, pp. 427–431. Romdhane, R., Crispim, C.F., Bremond, F., Thonnat, M., 2013. Activity recognition and uncertain knowledge in video scenes, in: Advanced Video and Signal Based Surveillance (AVSS), 2013 10th IEEE International Conference on. IEEE, pp. 377–382. Roy, P.C., Giroux, S., Bouchard, B., Bouzouane, A., Phua, C., Tolstikov, A., Biswas, J., 2011. A possibilistic approach for activity recognition in smart homes for cognitive assistance to Alzheimer’s patients, in: Activity Recognition in Pervasive Intelligent Environments. Springer, pp. 33–58. Satt, A., Sorin, A., Toledo-Ronen, O., Barkan, O., Kompatsiaris, I., Kokonozi, A., Tsolaki, M., 2013. Evaluation of speech-based protocol for detection of early-stage dementia., in: INTERSPEECH. pp. 1692–1696. Shaw, R., Troncy, R., Hardman, L., 2009. LODE: Linking Open Descriptions of Events, in: Gómez-Pérez, A., Yu, Y., Ding, Y. (Eds.), The Semantic Web, Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 153–167. Stavropoulos, T.G., Gottis, K., Vrakas, D., Vlahavas, I., 2013. aWESoME: A web service middleware for ambient intelligence. Expert Syst. Appl. 40, 4380–4392. Stavropoulos, T.G., Kontopoulos, E., Bassiliades, N., Argyriou, J., Bikakis, A., Vrakas, D., Vlahavas, I., 2014a. Rule-based approaches for energy savings in an ambient intelligence environment. Pervasive Mob. Comput. doi:10.1016/j.pmcj.2014.05.001 Stavropoulos, T.G., Meditskos, G., Kontopoulos, E., Kompatsiaris, I., 2014b. The DemaWare Service-Oriented AAL Platform for People with Dementia. Artif. Intell. Assist. Med. AI-AMNetMed 2014 11. Stevenson, G., Knox, S., Dobson, S., Nixon, P., 2009a. Ontonym: a collection of upper ontologies for developing pervasive systems, in: Proceedings of the 1st Workshop on Context, Information and Ontologies. ACM, p. 9. Stevenson, G., Ye, J., Dobson, S., Nixon, P., 2009b. Loc8: a location model and extensible framework for programming with location. IEEE Pervasive Comput. 28–37. Suzuki, R., Otake, S., Izutsu, T., Yoshida, M., Iwaya, T., 2006. Monitoring daily living activities of elderly people in a nursing home using an infrared motion-detection system. Telemed. J. E Health 12, 146–155. Van Hage, W.R., Malaisé, V., Segers, R., Hollink, L., Schreiber, G., 2011. Design and use of the Simple Event Model (SEM). Web Semant. Sci. Serv. Agents World Wide Web 9, 128–136. Weiser, M., 1991. The computer for the 21st century. Sci. Am. 265, 94–104. Wolf, P., Schmidt, A., Otte, J.P., Klein, M., Rollwage, S., König-Ries, B., Dettborn, T., Gabdulkhakova, A., 2010. openAAL-the open source middleware for ambient-assisted living (AAL), in: AALIANCE Conference, Malaga, Spain. pp. 1–5. Ye, J., Coyle, L., Dobson, S., Nixon, P., 2007. Ontology-based models in pervasive computing systems. Knowl. Eng. Rev. 22, 315–347. Ye, J., Stevenson, G., 2013. Semantics-driven multi-user concurrent activity recognition, in: Ambient Intelligence. Springer, pp. 204–219.