A Cognitive Multi-Agent System for Emotion-Aware Ambient Intelligence

A Cognitive Multi-Agent System for Emotion-Aware Ambient Intelligence Giovanni Acampora, Vincenzo Loia, Autilia Vitiello Department of Computer Science University of Salerno Fisciano, Salerno, 84084 Italy Email:{gacampora, loia, avitiello}@unisa.it

Abstract—Ambient Intelligence (AmI) is an ubiquitous computing paradigm whose main objective is to provide living environments with personalized services for improving quality of people life. In spite of their technological definition, AmI systems are more than a straightforward integration among computer equipments. Indeed, AmI frameworks design strongly depends upon psychology and social sciences whose exploitation allows artificial systems to analyze the human being status and define the most suitable services collection for achieving users’ satisfaction. Our work meets AmI objectives by using a formal method for representing human moods and introducing the so-called cognitive agents whose inference capabilities enable the emotional services distribution for enhancing users’ comfort and simplifying the human/systems interactions. The agents reasoning engine is based on a novel extension of Fuzzy Cognitive Maps (FCMs) benefiting on the theory of Timed Automata: Timed Automata based FCMs. As will be shown in experimental results, our proposal improves the system’s usability in terms of efficiency, accuracy and emotional response. Index Terms—Fuzzy Cognitive Maps, Ambient Intelligence, System Dynamics, Emotion-Aware Systems.

I. I NTRODUCTION Ambient Intelligence (AmI) is an emerging multidisciplinary research field which proposes new ways of interaction between people and technology, making it suited to the needs of individuals and the environment that surrounds those [1]. Typically, AmI systems are realized by distributing collections of advanced services that allow environmental devices (lights, HVAC (Heating, Ventilation and Air Conditioning), etc.) to work in concert to support people in carrying out their everyday life activities. This result is gained by adapting the technology to the people’s needs by means of omnipresent computing elements which communicate amongst them in a ubiquitous way [2]. Under these conditions, Multi-Agent systems [3] seem to be the most suitable computing paradigm for implementing AmI frameworks and satisfying some key features of ambient intelligence such as autonomy, distribution, adaptation, pro-activeness, interoperability and responsiveness. However, last researches [4][5] prove that, in order to design and implement efficient AmI systems, some additional scientific backgrounds are necessary to model the emotional and psychological state of a generic user that interacts with an immersive computing framework [6]. Moreover, enhanced AmI systems would have to be strongly dynamic and adaptive

978-1-61284-060-4/11/$26.00 ©2011 IEEE

in order to face environmental changes due to some factors such as seasonal changes. Starting from this analysis, our goal is to propose a novel emotion-aware AmI architecture whose main purpose is to define a kind of intelligent environment capable of evaluating human emotion experiences and providing people with timed emotional services. This new vision of emotional services [7] enhances AmI by providing the opportune system’s behaviour according to user’s emotion generation under particular temporal conditions. Our framework has been designed by exploiting a MultiAgent System (MAS) composed by the so-called cognitive agents that are responsible of managing particular environmental aspects such as lighting, temperature, entertainment, etc. by dynamically analyzing user’s moods and environmental features and proposing the appropriate services for achieving human’s satisfaction. Cognitive agents are computational entities whose capabilities are derived from a novel fuzzy cognitive engine that directly manages emotional and temporal issues strongly characterizing the immersive computing environments. In particular, the cognitive agents reasoning is composed by two interactive components: 1) a modified version of the Russell’s Circumplex model [8] and 2) Timed Automata based FCMs. Russell’s Circumplex model enables agents to model the human emotional state, whereas, Timed Automata based FCMs are exploited to analyze human emotions, environmental features and temporal concept in order to distribute the most suitable services collection. As will be shown in experiments, where a usability study has been conducted, our proposal of cognitive AmI framework improves user’s comfort and satisfaction and enhances the system’s usability in terms of efficiency, accuracy and emotional response. II. T HE E MOTION -AWARE A M I A RCHITECTURE As mentioned in introduction, the aim of this paper is to investigate a Multi-Agent system composed by cognitive agents capable of performing an adaptable AmI environment which is aware of users’ presence and above all of their emotional conditions. From an architectural point of view, the proposed approach splits the living environments in a collection of so-called Cognitive Regions where each region contains a set of cognitive

Input

Cognitive Region 1 Agents

Cognitive Region 1

Emotional Concepts Environmental Features Concepts

Cognitive Region 2

Cognitive Region 4

Services Concepts

Timed Automata based FCMs

Time

Music Agent Lighting Agent Temperature Agent Windows Agent Blind Agent ...

Output

Cognitive Agent Cognitive Region 3

Fig. 2.

Fig. 1.

A Sample of Cognitive Regions

agents devoted to distribute services in a particular place of the environment (e.g. a room, a floor, a building, etc.). Precisely, each cognitive agent (see Fig. 2) distributes timed emotional services, each one related to a particular environmental aspect (HVAC services, Lighting Services, Music Services, TV Services, etc.) . In order to achieve this result, cognitive regions (shown in Fig. 1) exploit the emotional state of the user that lives the space and the environmental features (temperature, lighting, etc.) characterizing the same space together with a proper knowledge about the time. Different from other proposed approaches [9][10][11], the agents that populates our framework offer time-depending actions and, as consequence, they may distribute distinct services under the same emotional and environmental conditions. This choice is particularly useful in ambient intelligence scenarios where the services distribution may depend upon temporal information such as the current season or time of day. For instance, under the same conditions, a lighting agent may decide to activate two different kinds of services collections in winter and summer or, at the same way, a music agent may activate distinct services that play different musical styles with different volume levels in the morning and evening. In order to pursue this aim, agents derive their inference capabilities from a novel fuzzy cognitive engine that directly manages emotional and temporal issues strongly characterizing the immersive computing environments. These innovative inference capabilities result from the joint exploitation of a modified version of the Russell’s two-dimensional emotion model and Timed Automata based Fuzzy Cognitive Maps: the former enables agents to capture and model the human emotional state by exploiting a well-defined approach; the latter are used to infer the most suitable services collection starting from the analysis of human emotions, environmental features and temporal concept. This approach allows cognitive agents to deal with the collection of so-called cognitive concepts (human emotions, environmental features and services) which, together with the relationships among them, characterize a cognitive

A Cognitive Agent

region in a given period of time. From an implementation point of view, the agents interact with cognitive regions by means of a sensors network based on the Lonworks protocol. Lonworks is Echelon’s proprietary network and encompasses a protocol for building automation. There are many commercially available complex devices, sensors and actuators for this system. The control network is interfaced with computer systems through a gateway to the IP network provided by Echelon’s iLON 100 web server. This allows the states and values of sensors and actuators to be read or altered via a standard TCP/IP communications. Hereafter, our proposal of agents’ inference engine will be introduced by starting from the definition of the single components which realize it. The case study section will be devoted to presents the design of an intelligent agent dealing with the distribution of music services and a usability test validating the quality of proposed approach. III. A M ODIFIED E MOTIONAL M ODEL BASED ON RUSSELL’ S CIRCUMPLEX APPROACH As aforementioned, one of the main goal of this research is to create a framework for emotion-aware AmI by designing a multi-agent system that distributes advanced services by detecting the user’s emotions from living environment. In order to introduce and deal with emotional concepts, the Russell two-dimensional emotion model [8] is used. Emotion modelling is a wide research area and a lot of theories concerning the structure and the classification of emotions have been developed in the years. In particular, the definition of the Russell’s model follows the so called dimensional approach which states the existence of a limited number of dimensions on which are based all affective states, in contrast with the basic emotions theories which consider the emotion space as a set of dimensions (one for each emotion) which vary independently of the other ones. In details, the Russell’s model, named the circumplex model of Affect, characterizes the affective space through two-dimensions: pleasuredispleasure (valence dimension) and arousal-sleep (arousal dimension). In this vision, each emotion is viewed as a

linear combination of valence and arousal, respectively, the horizontal and vertical axis in Fig. 3. Specifically, Russell views the affective space tracked by eight emotions: pleasure, displeasure, arousal and sleep which represent the ends of the two dimensions and excitement, distress, depression and contentment which only help to define the quadrants of space, but don’t form independent dimensions [8]. As consequence, the affective space is represented by a circle (circumplex) built by eight points in the following order: pleasure (0∘ ), excitement (45∘ ), arousal (90∘ ), distress (135∘ ), displeasure (180∘ ), depression (225∘ ), sleepness (270∘ ) and contentment (315∘ ). Russell completes its emotional model by placing on the circumplex space the 28 emotions which, according to him, compose the affective universe. By exploiting Ross’s procedure [12], Russell computes the polar coordinates for each of the 28 emotions (for instance, happy coordinates fall at the point 7.8∘ and sad ones fall at the point 207.5∘ ) obtaining the final model showed in Fig. 3. The emotions that will be simulated in our system are those identified in the Russell’s model: tense, bored, happy, pleased, tired, angry, relaxed, sad, distressed, sleepy, etc. . In particular, in our scenario, an emotion 𝑒 is individuated by means of a triple (𝑎, 𝑣, 𝑙) where 𝑎, 𝑣 ∈ [−1, 1] are, respectively, the arousal and the valence levels of 𝑒, whereas, 𝑙 ∈ [0, 1] represents the so-called activation level of the emotion 𝑒 and is calculated starting from 𝑎 and 𝑣. Therefore, in order to identify the user’s emotion 𝐸𝑢 among those of Russell’s model, only the values 𝑎 and 𝑣 are needed. The method is the following: 1) To select the nearest emotion (in terms of degrees) to the value 𝐸 which is computed starting from 𝑎 and 𝑣 though the following function: ⎧ arctan (∣𝑎∣/∣𝑣∣) if 𝑎, 𝑣 ∈ ℝ+   ⎨ ∘ 90 + arctan (∣𝑣∣/∣𝑎∣) if 𝑎 ∈ ℝ+ , 𝑣 ∈ ℝ− 𝐸= 180∘ + arctan (∣𝑎∣/∣𝑣∣) if 𝑎, 𝑣 ∈ ℝ−   ⎩ 270∘ + arctan (∣𝑣∣/∣𝑎∣) if 𝑎 ∈ ℝ− , 𝑣 ∈ ℝ+ 2) To compute the value 𝑙 through the following formula: 2 ∗ ∣𝑎∣ ∗ ∣𝑣∣ . 𝑙= ∣𝑎∣ + ∣𝑣∣ More precisely, the activation level 𝑙 represents the strength of the emotion selected on the Russell’s circumplex model and will be used as input value for the cognitive concept related to the emotion 𝑒 individuated by the 𝑎 and 𝑣 values. Some examples of emotions modeled through the Russell’s model are: 𝑒 = (−0.4, −0.8, 0.54) → 𝐸 = 180∘ + 27∘ → 𝐸𝑢 = sad 𝑒 = (0.2, 1, 0.34) → 𝐸 = 11∘ → 𝐸𝑢 = happy 𝑒 = (−0.2, −0.4, 0.25) → 𝐸 = 180∘ + 27∘ → 𝐸𝑢 = a bit sad 𝑒 = (−0.5, −1, 0.7) → 𝐸 = 180∘ + 27∘ → 𝐸𝑢 = very sad IV. T IMED AUTOMATA BASED F UZZY C OGNITIVE M APS This section is devoted to introduce the main component of proposed framework: the Timed Automata based Fuzzy

(Arousal) 90°

Afraid

Aroused

Alarmed Tense Angry

Astonished Excited

Annoyed Distressed

Delighted

Frustrated

Happy 180°

0° (Valence) Pleased Glad

Miserable

Sad Gloomy Depressed

Serene Content At Ease Satisfied Relaxed Calm

Bored Droppy Sleepy

Tired 270°

Fig. 3.

Russell’s emotion model

Cognitive Maps. TAFCMs are an inference engine which provides agents with a dynamic behaviour that is capable of adapting agents decisions to different emotional, environmental and temporal situations. In particular, TAFCMs allow an agent to live its life as a biological entity characterized by different age brackets where the agent shows the most opportune behaviour to solve the issue for which it is designed. In our ambient intelligence scenario, this approach enables agents to show a time-depending behaviour that improves the agents’ intelligence perception in terms of unpredictability of the accomplished actions. Hereafter, TAFCMs are presented together with a short description of standard Fuzzy Cognitive Maps and Timed Automata which are the basic blocks for defining this new kind of inference engine. A. TAFCMs as an Extension of Fuzzy Cognitive Maps A FCM is a fuzzy signed oriented graph with feedback that models systems by means of a collection of concepts and causal relations among concepts. In details, concepts are represented by nodes in a directed graph and the graph’s edges represent the casual influences between the concepts. The value of a node reflects the degree to which the concept is active in the system at a particular time. All the values in the graph are fuzzy, i.e, concepts take values in the range between [0, 1] and the arcs weights are in the interval [−1, 1]. Between concepts, there are three possible types of causal relationships, that express the type of influence from one concept to the others. The weights of the arcs between concept 𝐶𝑖 and concept 𝐶𝑗 could be positive 𝑊𝑖𝑗 > 0 which means that an increase in the value of concept 𝐶𝑖 leads to the increase of the value of concept 𝐶𝑗 , and a decrease in the value of concept 𝐶𝑖 leads to the decrease of the value of concept 𝐶𝑗 . Or there is negative causality 𝑊𝑖𝑗 < 0 which means that an increase in the value of concept 𝐶𝑖 leads to the decrease of

the value of concept 𝐶𝑗 and vice versa. The value of each one concept is influenced by the values of the connected concepts with the appropriate weights and by its previous value. In our proposal, in order to provide agents with a more dynamic behaviour, an extension of Fuzzy Cognitive Maps, the TAFCMs, is provided. TAFCMs enhance the agents behaviour by introducing two novel concepts: the cognitive era and the cognitive configuration. By exploiting TAFCMs, a cognitive agent split its life in a sequence of age brackets (cognitive eras) and, in each age backet, to use the most suitable FCM (cognitive configuration) defining the agents decisions in that bracket. By using the cognitive era, each agent may modify the relationships among its concepts (the cognitive configuration) in a time dependent way and, as consequence, it may distributes different services in different time under the same emotional and environmental conditions. In order to embed the cognitive era idea into the FCMs context it is necessary to introduce a temporal abstraction capable to deal with the cognitive eras in a direct and formal way. This novel concept is named T-Time. T-Time can be viewed as a mechanism capable of modifying the FCM structure (in terms of concepts and relationships among concepts) by moving the system from a cognitive era to the successive one. In order to add these new concepts to standard FCM, TAFCMs exploit a timed automaton whose states depict the agents behaviour, by means an opportune FCM, throughout a limited period of time. The exploited timed automaton defines all the potential sequences of cognitive eras (and the related cognitive configurations) that the system could cross during its life-cycle. More in detail, TAFCMs associate each state in a timed automaton with a cognitive configuration which describes the behaviour of a system in a time interval. Therefore, TAFCMs are able to model dynamic changes in cognitive representation of system and, consequently, perform a more realistic and coherent cognitive computation. A TAFCM, as will be formally defined at the next section, is a couple of two components: a timed automaton that describes the dynamic evolution of a system and a FCM modeling the cognitive behaviour of system during first phase of its existence. Before giving a formal definition of a TAFCM, a brief description of Timed Automata is needed. B. Timed Automata A timed automaton is a standard finite-state automaton extended with a finite collection of real-valued clocks providing a straightforward way to represent time related events, whereas automata-based approaches cannot offer this feature. The transitions of a timed automaton are labeled with a guard (a condition on clocks), an action or symbol on alphabet Σ, and a clock reset (a subset of clocks to be reset). Intuitively, a timed automaton starts execution with all clocks set to zero. Clocks increase uniformly with time while the automaton is within a node. A transition can be taken if the clocks fulfill the guard. By taking the transition, all clocks in the clock reset will be set to zero, while the remaining keep their values. Thus transitions occur instantaneously.

The set of behaviours expressed by an agent modeled by means of a timed automaton is defined by a timed language, i.e., a collection of timed words. Both timed concepts are defined in the follows. Definition 1: A time sequence 𝜏 = 𝜏1 𝜏2 ⋅ ⋅ ⋅ is an infinite sequence of time values 𝜏𝑖 ∈ ℝ with 𝜏𝑖 > 0, satisfying the following constraints: 1) Monotonicity: 𝜏 increases strictly monotonically; that is, 𝜏𝑖 < 𝜏𝑖+1 for all 𝑖 ≥ 𝑖 + 1. 2) Progress: For every 𝑡 ∈ ℝ, there is some 𝑖 ≥ 1 such that 𝜏𝑖 ≥ 𝑡. Then, a timed word on alphabet Σ is a pair (𝜎, 𝜏 ) where 𝜎 = 𝜎1 𝜎2 . . . is an infinite word over Σ and 𝜏 is a time sequence. A timed language over Σ is a set of timed words on Σ. Definition 2: Let 𝑋 a finite collection of real-valued variables named clocks then the set Φ(𝑋) of clock constraints 𝛿 is defined inductively by: 𝛿 := 𝑥 ≤ 𝑐∣𝑐 ≤ 𝑥∣¬𝛿∣𝛿1 ∧ 𝛿2 where 𝑥 is a clock in 𝑋 and 𝑐 is a constant in ℚ, the set of nonnegative rationals. A clock interpretation 𝜈 for the set 𝑋 of clocks assigns a real value to each clock; that is, it is a mapping from 𝑋 to ℝ. A clock interpretation 𝜈 for 𝑋 satisfies a clock constraint 𝛿 over 𝑋 if and only if 𝛿 evaluates to true using the values given by 𝜈. Now, a precise definition of timed transition table, which define the timed automaton behaviour, is given: Definition 3: A timed transition table 𝒜 is a tuple ⟨Σ, 𝑆, 𝑆0 , 𝐶, 𝐸⟩, where: ∙ ∙ ∙ ∙ ∙

Σ is a finite alphabet, 𝑆 is a finite set of states, 𝑆0 ⊆ 𝑆 is a set of start states, 𝐶 is finite set of clocks, and 𝐸 ⊆ 𝑆×𝑆×Σ×2𝐶 ×Φ(𝐶) is the collection of transitions. An edge ⟨𝑠, 𝑠′ , 𝑎, 𝜆, 𝛿⟩ represents a transition from state 𝑠 to state 𝑠′ on input symbol 𝑎. The set 𝜆 ⊆ 𝐶 represents the collection of clocks to be reset with this transition, and 𝛿 is a clock constraint over 𝐶.

If (𝜎, 𝜏 ) is a timed word viewed as an input to an automaton, it presents the symbol 𝜎𝑖 at time 𝜏𝑖 . If each symbol 𝜎𝑖 is interpreted to denote an event occurrence then the corresponding component 𝜏𝑖 is interpreted as the time of occurrence of 𝜎𝑖 . Given a timed word (𝜎, 𝜏 ), the timed transition table 𝒜 starts in one of its start states at time 0 with all clocks initialized to 0. As time advances, the values of all clocks change, reflecting the elapsed time. At time 𝜏𝑖 , 𝒜 state from 𝑠 to 𝑠′ using some transition of the form ⟨𝑠, 𝑠′ , 𝜎𝑖 , 𝜆, 𝛿⟩ reading the input 𝜎𝑖 , if the current values of clocks satisfy 𝛿. With this transition the clocks in 𝛿 are reset to 0, and thus start continuing time with respect to the time of occurrence of this transition. Formally, this timed behaviour is captured by introducing runs of timed transition tables.

Definition 4: A run 𝑟, denoted by (¯ 𝑠, 𝑣¯), of a timed transition table ⟨Σ, 𝑆, 𝑆0 , 𝐶, 𝐸⟩ over a timed word (𝜎, 𝜏 ) is an infinite sequence of the form 𝜎1

𝜎2

𝜎3

𝜏1

𝜏2

𝜏3

𝑟 : ⟨𝑠0 , 𝜈0 ⟩ −→ ⟨𝑠1 , 𝜈1 ⟩ −→ ⟨𝑠2 , 𝜈2 ⟩ −→ ⋅ ⋅ ⋅ with 𝑠𝑖 ∈ 𝑆 and 𝜈𝑖 ∈ [𝐶 → ℝ], for all 𝑖 ≥ 0, satisfying the following requirements: ∙ Initiation: 𝑠0 ∈ 𝑆0 and 𝜈0 (𝑥) = 0 for all 𝑥 ∈ 𝐶. ∙ Consecution: for all 𝑖 ≥ 1, there is an edge in E of the form ⟨𝑠𝑖−1 , 𝑠𝑖 , 𝜎𝑖 , 𝜆𝑖 , 𝛿𝑖 ⟩ such that (𝜈𝑖−1 + 𝜏𝑖 − 𝜏𝑖−1 ) satisfies 𝛿𝑖 and 𝜈𝑖 equals [𝜆𝑖 → 0](𝜈𝑖−1 + 𝜏𝑖 − 𝜏𝑖−1 ). The timed transition table together with the run concept are the main notions used in a TAFC to embed dynamism in the standard FCM definition. C. Timed Automata based Fuzzy Cognitive Maps: Formal Definition The first step towards the formal definition of TAFCMs is the redefining of the standard FCMs by means of the graph theory. From this point of view, an FCM 𝐹 can be defined as follows: 𝐹 = (𝑉, 𝐸, 𝑎, 𝑤) 𝑉 = {𝑐𝑖 ∣𝑖 > 0} 𝑎 : 𝑉 → [0, 1] (1) 𝐸 = {(𝑐𝑖 , 𝑐𝑗 )∣𝑐𝑖 , 𝑐𝑗 ∈ 𝑉 } 𝑤 : 𝐸 → [−1, 1] where 𝑉 is the collection of cognitive concepts; 𝑎 is a function associating an concept in 𝑉 with a real activation value in [0, 1]; 𝐸 is the set of causal relationships between concepts; 𝑤 is a function associating an edge in 𝐸 with a real weight in [−1, 1]. The formal graph view of a FCM only represents a static vision of our cognitive system. The successive step is to introduce a collection of operators able to transform the cognitive structure defined in (1). These operators represent the fundamental operations on which constructing the proposed cognitive/dynamic model. They will change the cognitive configuration of a given agent’s FCM, 𝐹 = (𝑉, 𝐸, 𝑎, 𝑤) , by following the rules: ∙ To add concepts (⊕) ; ∙ To add causal relationships (⊞) ; ∙ To remove concepts (⊖) ; ∙ To remove causal relationships (⊟) ; ∙ To magnify/reduce the strength of a causal relationships in additive or multiplicative way (∔, ⊡) ; ∙ To magnify/reduce the level of system concept in additive or multiplicative way(†,‡) . Since presenting all operators formal descriptions is too long and boring, only the operator of additive modification of a relation is presented. Definition 5 (Additive modification of causal relation ∔): The operator named ∔ modifies the value of a given causal relationship by adding it to a real value 𝛼. This task is accomplished by opportunely redefining the weight assignment function 𝑤. Let (𝑐𝑖 , 𝑐𝑗 ) be the causal relationship

to modify in additive way, then 𝑤′ : 𝐸 → [−1, 1] is the modified function defined as follows: ⎧  𝑤((𝑐𝑙 ,𝑐𝑚 ))     ⎨ 𝑤((𝑐 ,𝑐 ))+𝛼 𝑙 𝑚 ′ 𝑤 ((𝑐𝑙 ,𝑐𝑚 ))=  −1     ⎩ 1

if (𝑐𝑙 ,𝑐𝑚 )∕=(𝑐𝑖 ,𝑐𝑗 ) if (𝑐𝑙 ,𝑐𝑚 )=(𝑐𝑖 ,𝑐𝑗 ) and −1≤𝑤((𝑐𝑖 ,𝑐𝑗 ))+𝛼≤1 if (𝑐𝑙 ,𝑐𝑚 )=(𝑐𝑖 ,𝑐𝑗 ) and 𝑤((𝑐𝑙 ,𝑐𝑚 ))+𝛼1

(2)

Then 𝐹 ′ = (𝑉, 𝐸, 𝑎, 𝑤′ ) is the obtained cognitive map. In our approach, the operator of additive modification of causal relationship can be used to magnify or reduce the strength of a relationship between an emotional or environmental concept and an emotional service concept in order to support, for instance, environmental conditions changing in the time. Once defined the operators changing FCM configuration it is possible to define the cognitive operator set: 𝐶𝑜𝑝 = {⊕, ⊖, †, ‡, ⊞, ⊟, ∔, ⊡, △, ⋎}. The 𝐶𝑜𝑝 allows us to redefine the Timed Automata concept in order to introduce a novel kind of transition edges capable of changing the cognitive configuration of the modeled system. In particular, the standard transitions set of timed automata is replaced with the following edges set: 𝐸𝐶 ⊆ 𝑆 × 𝑆 × Σ × 2𝐶 × Φ(𝐶) × 𝐶𝑜𝑝

(3)

and, the novel definition of timed automata based on cognitive edges idea is: Definition 6: A timed cognitive transition table 𝒜𝑡 is a tuple ⟨Σ, 𝑆, 𝑆0 , 𝐶, 𝐸𝐶 ⟩, where: ∙ Σ is a finite alphabet, ∙ 𝑆 is a finite set of states, ∙ 𝑆0 ⊆ 𝑆 is a set of start states, ∙ 𝐶 is finite set of clocks 𝐶 ∙ 𝐸𝐶 ⊆ 𝑆 × 𝑆 × Σ × 2 × Φ(𝐶) × 𝐶𝑜𝑝 is the collection of cognitive transitions. An edge ⟨𝑠, 𝑠′ , 𝑎, 𝜆, 𝛿, ∘⟩, with ∘ ∈ 𝐶𝑜𝑝 produces the same effect of a standard transition ⟨𝑠, 𝑠′ , 𝑎, 𝜆, 𝛿⟩, but it individuates the task defined by the operator ∘ ∈ 𝐶𝑜𝑝 . The set 𝜆 ⊆ 𝐶 represents the collection of clocks to be reset with this transition, and 𝛿 is a clock constraint over 𝐶. At this point, it is possible to give a formal definition of a TAFCM and the properties characterizing its dynamic behaviour. Definition 7: A TAFCM 𝑇𝐴 is an ordered pair composed by an initial cognitive configuration named 𝐹 0 together with a timed cognitive transition table 𝑇𝑀 which represents the mathematical entity acting as melting point between cognitivism and dynamism in system modeling. Formally: 𝑇𝐴 = (𝐹 0 , 𝑇𝑀 ) The TAFCM properties which define the dynamic behaviour of an agent are: cognitive evolution and cognitive run. The cognitive evolution is a mapping among the states 𝑆 contained in 𝑇𝑀 and the collection of possible cognitive

configurations obtained starting from 𝐹 0 . More in detail, the cognitive evolution is a mathematical succession, generated in an inductive way, which maps each state in 𝑆 with a one or more cognitive configurations obtained by sequentially applying over 𝐹 0 the cognitive operators in 𝑆 × 𝑆 × Σ × 2𝐶 ×Φ(𝐶)×𝐶𝑜𝑝 . The cognitive evolution will not be formally defined but, intuitively, it can be represented by the expression (4) which shows the sequence of pairs composing a cognitive evolution over 𝑠0 ∈ 𝑆0 together with the fuzzy cognitive transformations obtained by exploiting the ∘𝑖 operators. Ψ(0) : 𝑠0 ∈ 𝑆0 Ψ(1) : 𝑠1 ∈ 𝑆 2

Ψ(2) : 𝑠 ∈ 𝑆 .. . Ψ(𝑗) : 𝑠𝑗 ∈ 𝑆 .. .

→ 𝐹0 ↓ ∘0 → 𝐹1 ↓ ∘1 → 𝐹2 ↓ ∘2 .. .

precise dynamic behaviour of the system, i.e., 𝑤𝑖 defines the T-Time. Definition 9 (T-Time): If 𝑇𝐴 = (𝐹 0 , 𝑇𝑀 ) is a TAFCM and 𝑇𝑀 is a timed automaton recognizing the timed language 𝐿 = {𝑤1 , 𝑤2 , 𝑤3 , . . . , 𝑤𝑖 , . . .} and 𝑤𝑖 is a timed word and 𝑟𝑐 is a cognitive run defined over 𝑤𝑖 then 𝑤𝑖 is a T-Time of the system. Starting from the T-Time definition a formal description of cognitive era and cognitive configuration is given. Definition 10 (Cognitive era and cognitive configuration): If 𝑟𝑐 is a cognitive run defined over the T-Time 𝑤𝑖 = (𝜎, 𝜏 ) ∈ 𝐿: 𝜎1 ,∘1

𝜎2 ,∘2

𝜎3 ,∘3

𝜏1

𝜏2

𝜏3

𝑟𝑐 : ⟨𝑠0 , 𝜈0 ⟩ −−−→ ⟨𝑠1 , 𝜈1 ⟩ −−−→ ⟨𝑠2 , 𝜈2 ⟩ −−−→ ⋅ ⋅ ⋅

(4)

↓ ∘𝑗−1 → 𝐹𝑗 ↓ ∘𝑗 .. .

then time interval between the instant 𝜏𝑖 and 𝜏𝑖+1 is the 𝑖𝑡ℎ cognitive era of system and the FCM 𝐹 𝑖 that depicts the system during the same interval is defined as the 𝑖𝑡ℎ cognitive configuration. D. Cognitive Region: Formal Definition

Obviously, the cognitive evolution only represents a mapping between the states of timed automaton 𝑇𝐴 and the collection of cognitive configurations computable starting from 𝐹 0 by applying different sequence of operators in Ω; no dynamic aspects are considered in the cognitive evolution definition and, therefore, we introduce the idea of cognitive run extending the initial idea of the run of standard timed transition table. Definition 8: Let Ψ be a cognitive evolution, then a cog𝑠, 𝜈¯), of a timed transition table nitive run 𝑟𝑐 , denoted by (¯ ⟨Σ, 𝑆, 𝑆0 , 𝐶, 𝐸𝐶 ⟩ over a timed word (𝜎, 𝜏 ) and a collection of cognitive operators Ω ⊆ 𝐶𝑜𝑝 , is an infinite sequence of the form 𝜎1 ,∘1

𝜎2 ,∘2

𝜎3 ,∘3

𝜏1

𝜏2

𝜏3

𝑟𝑐 : ⟨𝑠0 , 𝜈0 ⟩ −−−→ ⟨𝑠1 , 𝜈1 ⟩ −−−→ ⟨𝑠2 , 𝜈2 ⟩ −−−→ ⋅ ⋅ ⋅ with 𝑠𝑖 ∈ 𝑆, 𝜈𝑖 ∈ [𝐶 → ℝ], for all 𝑖 ≥ 0, and ∘𝑖 ∈ 𝐶𝑜𝑝 , for all 𝑖 ≥ 1, satisfying the following requirements: ∙ Initiation: 𝑠0 ∈ 𝑆0 and 𝜈0 (𝑥) = 0 for all 𝑥 ∈ 𝐶. ∙ Consecution: for all 𝑖 ≥ 1, there is an edge in E of the form ⟨𝑠𝑖−1 , 𝑠𝑖 , 𝜎𝑖 , 𝜆𝑖 , 𝛿𝑖 ⟩ such that (𝜈𝑖−1 + 𝜏𝑖 − 𝜏𝑖−1 ) satisfies 𝛿𝑖 and 𝜈𝑖 equals [𝜆𝑖 → 0](𝜈𝑖−1 + 𝜏𝑖 − 𝜏𝑖−1 ). ∙ Atomicity: The operators ∘𝑖 ∈ 𝐶𝑜𝑝 are atomic operations and their computation time is equals to 0, i.e., they do not modify the duration of permanence in the automaton state 𝑠𝑖 , (𝜏𝑖 − 𝜏𝑖−1 ). 𝑖 𝑖 ∙ Evolution: each state 𝑠 of a pair ⟨𝑠 , 𝜈𝑖 ⟩ in 𝑟𝑐 is mapped 𝑖 on a FCM 𝐹 as described by the cognitive evolution Ψ. If 𝑇𝐴 = (𝐹 0 , 𝑇𝑀 ) is a TAFCM that models a given system then the set of cognitive run 𝑟𝑐 defined over the timed language 𝐿, generated by 𝑇𝑀 , completely describes the collection dynamic behaviours of the system, whereas, the cognitive run 𝑟𝑐 defined over a single word 𝑤𝑖 ∈ 𝐿 defines a

Once that TFCMs have formally defined, a more precisely definition of cognitive agent, cognitive region and emotionaware AmI is provided. In particular, a cognitive agent 𝑎 is defined as a TAFCM 𝑇𝑎 = (𝐹 0 , 𝑇𝑀 ) where the initial cognitive maps contains concepts related to emotions, environmental features and services. A cognitive region 𝐶𝑅𝑗 , composed 𝑝 cognitive agents, is defined as: 𝐶𝑅𝑗 = {(𝐹 0 , 𝑇𝑀 )1 , (𝐹 0 , 𝑇𝑀 )2 , . . . , (𝐹 0 , 𝑇𝑀 )𝑝 } As consequence, an emotion-aware AmI system 𝐴, composed by 𝑞 cognitive spaces is depicted as: 𝐴 = {𝐶𝑅1 , 𝐶𝑅2 , . . . , 𝐶𝑅𝑞 } V. C ASE STUDY AND E XPERIMENTS This section is devoted to validate proposed approach through the design of a cognitive agent aimed to the distribution of musical services. Moreover, usability test has been defined and performed in order to quantify the benefits offered by our services in terms of users’ comfort and satisfaction. A. A Case Study: The Music Agent In order to better explicate and validate our proposal, this section shows the behaviour of an agent, the so-called music agent, which deals with services related to the activation of different musical styles or the setting of musical parameters as, for instance, the music volume, by considering some of emotional and environmental concepts. As previously described, the behaviour of each agent can be defined by means of a ordered pair 𝑇 = (𝑇𝑀 , 𝐹 0 ). In particular, the music agent exploits a FCM 𝐹 0 characterized by the following features: ∙ seven emotional concepts representing seven moods (sad, angry, tense, happy, bored, relaxed, sleepy) chosen among emotional states of the Russell circumplex model;

two environmental concepts representing the outdoor temperature and luminosity (outdoor temperature and outdoor light); ∙ four musical service concepts where a service manages the music volume (volume service) whereas the remaining ones manage the music style to play (soft music, hard music, mid-music). The relationships among these concepts represent the influence values of the emotional and environmental features respect to the services activation’value. The Fig. 4 shows the FCM 𝐹 0 related to the music agent. The agent evaluates the environmental features (𝐼1 and 𝐼2 ) and the current user’s mood (𝐸𝑖 with 𝑖 = 1, . . . , 7) and decides which services (𝑆𝑖 with 𝑖 = 1, . . . , 4) have to be activated in order to satisfy user’s musical requirements. ∙

I1

year marked by particular weather patterns and daylight hours and to a particular time of day. Fig. 5 shows the timed automaton 𝑇𝑀 . Each automaton state represents the behaviour of the music agent during a well-defined time period of day. In particular, the state 𝑆1 represents the agent’s behaviour during the morning, the state 𝑆2 during the afternoon, the state 𝑆3 during the evening and, finally, the state 𝑆4 during the night. The transitions among the states apply a collection of cognitive operators that modify the weight of concepts relationships. This behaviour updating will adapt agent’s actions to user’s needs in different time.

I2 0.3

E1

0.4 0.6

E2

S3 E3

Fig. 5.

Music agent timed automaton

0.7 0.7 S1

0.2

0.7 E4

0.8

0.9

S2

0.5 0.8 0.4

E5

-0.6

Associated with the initial state 𝑆1 there is the fuzzy cognitive map 𝐹 0 (Fig. 4), whereas, the Fig. 6 shows the cognitive map 𝐹 1 associated with the state 𝑆2 obtained by applying the operators related to the automaton transition 𝑆1 → 𝑆2 on 𝐹 0 .

0.7

0.5

B. Usability Test E6

0.3

S4

-0.9 E7

Fig. 4.

The Fuzzy Cognitive Map 𝐹 0 related to the music agent.

TABLE I 𝐹 0 ’ S C ONCEPTS Concepts sad mood angry mood tense mood happy mood bored mood relaxed mood sleepy mood

Labels E1 E2 E3 E4 E5 E6 E7

Concepts outside temperature outside light volume service soft music service hard music service mid-music service

Labels I1 I2 S1 S2 S3 S4

In order to model the dynamic component of the agent’s behaviour, our approach uses a timed automaton named 𝑇𝑀 . Thanks to this component, agents are capable of dynamically changing their behaviour by updating their cognitive behaviour by means of the application of the operators belonging to the set 𝐶𝑜𝑝 . More in detail, the music agent exploits the automaton 𝑇𝑀 in order to adapt the relationships strength between environmental features and services to a period of

A usability test has been performed in order to evaluate the proposed cognitive approach in terms of user’s satisfaction. This test has been designed by providing a realistic environment, the Cognitive Assisted Living Testbed, based on the control network protocol known as Echelon Lonworks. Consisting of a single cognitive region located in a living room, the testbed framework provides the ideal environment to evaluate smart home services like the automatic settings of lighting, temperature, music, etc., under realistic circumstances. In particular, the cognitive region is composed by three different agents collection: light agents, temperature agents and music agents. At present, the framework is not equipped with sensors able to individuate the user’s emotional state. For this reason, the user communicates his mood to the test framework by exploiting a mobile device which implements a graphical user interface based on the our modified Russell’s model introduced in the section III. The usability test has been performed by inviting 20 users, chosen among researchers and students of University of Salerno, to live in our Cognitive Assisted Living Testbed. The main aim of this test is to evaluate agents’ behaviour during a temporal window in order to verify the dynamical aspects of agents’ decisions. In particular, this test chooses the day as the temporal window size and it split this window in four sequential time period named, respectively, morning, afternoon, evening, and night. In this way the user may

I1

TABLE II R ESULTS OF USABILITY TEST

I2 0.35

E1

0.45 0.6

E2

S3 E3

0.7 0.7 S1

0.8

0.9

0.7

0.5

0.3

S4

-0.9 E7

Fig. 6.

Realism Avg 6,6 7 6,7 6,6

Max 7 8 8 7

Usefulness Min Avg Max 7 7,7 9 8 8,5 9 7 8,1 9 6 7,5 9

VI. C ONCLUSIONS

0.45 E5

E6

Min 5 4 4 5

S2

0.55 0.8

-0.6

Appropriateness Min Avg Max 5 6,9 8 5 8 9 5 7,5 8 4 6,5 7

0.2

0.7 E4

Test Parts morning test afternoon test evening test night test

The Fuzzy Cognitive Map 𝐹 1 related to the music agent.

appreciate the time-depending behaviour of a cognitive agent. During the usability test, users rated services distribution by means of a predefined questionary and they proposed new ideas for eventual improvements. Once a user starts its interactions with the framework and services collections are distributed, he sets its satisfaction level by selecting a vote belonging in the real range [1, 10] (the higher the better) for the three following usability aspects: ∙ Appropriateness of Response which measures the effectiveness of the system’s behaviour in response to user’s emotional states (set by user) and environmental features (measured by the system). This aspect is directly related to the system’s design and particularly to modelling of the TAFCM; ∙ Realism of System’s Response which measures the lag between a user’s request and the corresponding (correct) system’s reply. This value is affected by the automaton’s transition functions (which typically have a negligible computational load); ∙ System Usefulness which measures the overall usefulness of the system with regard to the improving the life’s quality. Table II shows, for each aforementioned aspect, the minimum, maximum and average scores related to the four different day periods. These results show that proposed approach satisfies user’s requirements in a more than sufficient way. In the future, our idea is to improve experimental environment by integrating our approach with an automatic emotion detection system based on Russells model in order to individuate the users emotional state in a transparent way. The environment thus enriched will enable to perform a more realistic test case in order to verify the effective benefits of our system. However, our first experimental results provide a good validation of proposed approach.

Our proposal shown an innovative emotion-aware AmI architecture, based on the integration of methods of distributed artificial intelligence and cognitive modeling, whose main aim is to define a kind of intelligent environment evaluating human emotion experiences and providing people with proper emotional services. This architecture has been designed by exploiting a Multi-Agent System (MAS) approach and defining the so-called cognitive agents, i.e., intelligent agents which distribute emotional services satisfying people specific requirements thanks to innovative inference capabilities resulted from the joint exploitation of the Russell’s two-dimensional emotion model and Timed Automata based Fuzzy Cognitive Maps. As has been shown in experimental results, where a usability study test have been performed, the proposed approach maximize the system’s usability in terms of efficiency, accuracy and emotional response. R EFERENCES [1] E. Aarts and R. Roovers, “Embedded system design issues in ambient intelligence,” in Ambient Intelligence: Impact on Embedded Sytem Design, T. Basten, M. Geilen, and H. Groot, Eds. Springer US, 2004, pp. 11–29. [2] K. Lyytinen and Y. Yoo, “Issues and challenges in ubiquitous computing,” Communications of ACM, vol. 45, no. 12, pp. 63–65, 2002. [3] S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Pearson Education, 2003. [4] G. Marreiros, R. Santos, C. Ramos, J. Neves, P. Novais, J. Machado, and J. Bulas-Cruz, “Ambient intelligence in emotion based ubiquitous decision making,” 2007. [5] F. Nasoz, C. L. Lisetti, and A. V. Vasilakos, “Affectively intelligent and adaptive car interfaces,” Information Sciences, vol. 180, no. 20, pp. 3817 – 3836, 2010. [Online]. Available: http://www.sciencedirect.com/science/article/B6V0C50F8C0V-3/2/e3f97efb89d5154a76149c596243b717 [6] H. Hagras, “Embedding computational intelligence in pervasive spaces,” IEEE Pervasive Computing, vol. 6, pp. 85–89, July-September 2007. [7] J. Zhou and P. Kallio, “Ambient emotion intelligence: from businessawareness to emotion-awareness,” in Proceeding of 17th International Conference on Systems Research, Informatics and Cybernetics, 2005. [8] J. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, no. 39, pp. 1161–1178, 1980. [9] E. H. L. Aarts and B. Eggen, Eds., Ambient Intelligence in HomeLab. Neroc, 2002. [10] J. A. Kientz, S. N. Patel, B. Jones, E. Price, E. D. Mynatt, and G. D. Abowd, “The georgia tech aware home,” in CHI ’08: CHI ’08 extended abstracts on Human factors in computing systems. New York, NY, USA: ACM, 2008, pp. 3675–3680. [11] U. Rutishauser, J. Joller, and R. Douglas, “Control and learning of ambience by an intelligent building,” IEEE Trans. Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 35, no. 1, pp. 121–132, 2005. [12] R. Ross, “A statistic for circular scales,” Journal of Educational Psychology, no. 29, pp. 384–389, 1938.