Computer Science Review A survey on mobile

Computer Science Review 25 (2017) 79–100

Contents lists available at ScienceDirect

Computer Science Review journal homepage: www.elsevier.com/locate/cosrev

A survey on mobile affective computing Eugenia Politou *, Efthimios Alepis, Constantinos Patsakis Department of Informatics, University of Piraeus, Greece

article

info

Article history: Received 17 February 2017 Received in revised form 18 July 2017 Accepted 25 July 2017 Available online 8 August 2017 Keywords: Mobile affective computing Affect recognition Affect detection Smartphone sensors Mobile sensing

a b s t r a c t The spontaneous recognition of emotional states and personality traits of individuals has been puzzling researchers for years whereas pertinent studies demonstrating the progress in the field, despite their diversity, are still encouraging. This work surveys the most well-known research studies and the stateof-the-art on affect recognition domain based on smartphone acquired data, namely smartphone embedded sensors and smartphone usage. Inevitably, supplementary modalities employed in many eminent studies are also reported here for the sake of completeness. Nevertheless, the intention of the survey is threefold; firstly to document all the to-date relevant literature on affect recognition through smartphone modalities, secondly to argue for the full potential of smartphone use in the inference of affect, and thirdly to demonstrate the current research trends towards mobile affective computing. © 2017 Elsevier Inc. All rights reserved.

Contents 1.

2.

3.

4.

5.

Introduction....................................................................................................................................................................................................................... 1.1. Related research ................................................................................................................................................................................................... 1.2. Criteria for inclusion in the literature review .................................................................................................................................................... 1.3. Road map .............................................................................................................................................................................................................. Background........................................................................................................................................................................................................................ 2.1. Affective computing ............................................................................................................................................................................................. 2.1.1. The affective domain ............................................................................................................................................................................ 2.1.2. Core affect, emotion, mood, personality and sentiment .................................................................................................................... 2.1.3. Affect modelling, classification and measurement ............................................................................................................................ 2.2. Ubiquitous computing and mobile sensing........................................................................................................................................................ Mobile affective sensing and relevant research.............................................................................................................................................................. 3.1. Big-Five personality traits recognition ............................................................................................................................................................... 3.2. Recognition of distinct affective states ............................................................................................................................................................... 3.2.1. Ekman’s distinct states model recognition ......................................................................................................................................... 3.2.2. Stress recognition ................................................................................................................................................................................. 3.2.3. Recognition of happiness, boredom and other distinct affective states ........................................................................................... 3.3. Recognition of dimensional affective states ....................................................................................................................................................... 3.4. Recognition of wellbeing and human behaviour models .................................................................................................................................. Challenges for smartphone affective research................................................................................................................................................................ 4.1. Privacy................................................................................................................................................................................................................... 4.2. Informed consent ................................................................................................................................................................................................. 4.3. Data misuse .......................................................................................................................................................................................................... 4.4. Trust and engagement ......................................................................................................................................................................................... 4.5. Multimodal fusion ................................................................................................................................................................................................ 4.6. Resource constraints ............................................................................................................................................................................................ 4.7. Affect modelling and representation .................................................................................................................................................................. 4.8. Cultural differences .............................................................................................................................................................................................. 4.9. Cost ........................................................................................................................................................................................................................ Discussion and conclusions ..............................................................................................................................................................................................

author. * Corresponding E-mail addresses: [email protected] (E. Politou), [email protected] (E. Alepis), [email protected] (C. Patsakis). http://dx.doi.org/10.1016/j.cosrev.2017.07.002 1574-0137/© 2017 Elsevier Inc. All rights reserved.

80 80 80 81 81 81 81 81 82 84 84 85 85 85 88 89 90 90 91 91 92 92 92 93 93 93 94 94 94

80

E. Politou et al. / Computer Science Review 25 (2017) 79–100

References ......................................................................................................................................................................................................................... 96

1. Introduction Nowadays, various mobile sensing applications demonstrate the progress in Human–Computer Interaction (HCI) by exploiting big data to measure and assess human-behavioural modelling [1,2]. The continuous sensing of smartphone users and the understanding of their interactions with others and the environment is exploited by the modern mobile sensing applications to provide useful services concerning individual and community wellbeing. This research area conceptualises the notions of mobile affective sensing and computing [3,4] for which, during the past decade, extensive and innovative research has commenced worldwide. In the majority of the undertaken experiments the evolution of smart devices, like smartphones, has been utilised so as to collect data about their holders’ emotions, moods or sentiments, and to model and classify user’s emotional, personality or psychological states. In most of these cases, after extracting suitable features from the sensor data, various machine-learning and data mining techniques are applied to automatically recognise participants’ emotions; such as anger and happiness, mood; such as positive and negative mood, or personality traits; such as neuroticism or extraversion. With regard to data diversity and size, data acquired through smartphone sensors and utilities usually comes in high resolution and spans over multiple modalities such as location, proximity with other objects, collocation with other devices, diversity of contacts or touch behaviour. In addition, smartphone data collection can be done on an unprecedented scale with millions of users in parallel [5]. Given this sheer volume of affective data, many computer scientists and clinical psychologists argue that affective computing can alter the way modern psychology is performed nowadays by inferring people’s emotions, moods or states and by providing appropriate and timely interventions. As Miller states in his manifesto [6]: ‘‘smartphones can be used in modern psychology as a research method on its own, because they are ubiquitous, unobtrusive, intimate, sensor rich, computationally powerful, and remotely accessible and therefore they offer huge potential to gather precise, objective, sustained, and ecologically valid data on the real-world behaviours and experiences of millions of people, without requiring them to come into labs’’. 1.1. Related research Thus far, despite the remarkable progress in machine analysis of human affect and the immense related body of literature, the problem of recognition and understanding human behaviour and affective status through machines remains complex and hard to be tackled [7]. Apparently, the most prevailing research topics, occupied the scientific community of affect recognition over the past two decades, were the challenge of emotion recognition through visual and audio stimuli and, consequently, the fusion of these two modalities. One well-known survey on video and audio affect recognition is conducted in [8] where its authors analysed past decade’s efforts towards recognition of spontaneous affective expression by using audio and visual information. Heretofore research in emotion recognition based on speech and face expressions has yielded overly fruitful results and inevitably, the majority of existing surveys concentrate on documenting these basic modalities [9–16]. In addition to visual and audio, supplementary modalities

have been widely used to infer affect, such as physiological signals [17], text input [18,19], body gestures [20], keyboard strokes and mouse movements [21] and touch behaviour [22,23]. Recently, an overview of the state-of-the-art in audio-, visual- and textualbased multimodal affect recognition has been conducted [24]. Concerning affect recognition studies based on data acquired through smartphones, our research concluded that yet there are not any comprehensive surveys to cover the full range of the growing research efforts. Perhaps the most relevant work is the one carried out by Shmueli et al. [25] where some of the pertinent studies regarding the use of smartphone data for detecting users’ emotional states and personality traits are referenced, as well as the one cited in [26] that lists some studies for automatic personality recognition through smartphones. However, these reviews are neither extensive nor in-depth, since their main focus is, as long as the first is concerned, to survey recent advances in computational approaches and to demonstrate how trust is an important building block of computational social systems, whereas as long as the latter is concerned, to review the relevant progress on personality computing in general, and thus to indicatively list some pertinent research works regarding personality recognition through smartphone data. Predicting users’ discrete and isolated emotional states by utilising smartphone capabilities has been the objective of a small number of surveys. One such work concerning the emotional state of happiness is referenced in [27] where its authors compiled a survey of recent studies on happiness prediction by using smartphones and other intelligent devices, aiming at the better understanding of happiness determinants. Nevertheless, the survey examines a single state, happiness, and does not thoroughly cover all the studies performed for the full spectrum of human affective domain. Furthermore, a brief overview of the latest research works in the area of stress recognition can be found in both [28] and [29] whose authors report recent efforts in detecting stress based on data collected from smartphones. However, their works cover a narrow picture of the affective domain and its pertinent studies to date. Notwithstanding the previous reviews, the current survey attempts to enlighten readers in mobile affect recognition domain by exploring and documenting the full spectrum of relevant research works, something which, to the best of our knowledge, has not been examined before. Beyond citing all the relevant research on mobile affect recognition domain, the survey aims at exposing the full potential of smartphone use in the inference of affect by providing encouraging affect recognition results based on smartphone collected data. Last but not least, the survey contributes in the dissemination of recent progress and radical developments in the field by demonstrating the state-of-the-art and the current research trends towards mobile affective computing. 1.2. Criteria for inclusion in the literature review The literature review presented and discussed in this survey is based on an extensive search for relevant papers that have been published during the last 7 years, a period characterised with a significant increase in research on affective computing through smartphones and the evolution of mobile sensing technologies. For compiling the current survey, studies regarding the detection of emotions, moods, personalities or other behavioural characteristics, like wellbeing, based on smartphone derived data were taken into account. In this regard, we have deliberately omitted the study of methods which concentrate on the use of basic modalities and


do not exploit the capabilities of mobile phones. Hence, we focus on research that uses data acquired from smartphones, that is smartphone sensor data (like accelerometer or GPS) and/or smartphone usage data normally derived from using various mobile utilities; like Call/SMS logs, apps or screen usage. Inevitably, studies employing additional external modalities for inferring affect; like ECG or wrist bands, were also included in this survey, as long as they utilised smartphone derived data as well. 1.3. Road map This paper is organised as follows. A short introduction on the concepts of Affective Computing and Mobile Sensing is presented in Section 2. In this section, we also discuss the main components of Affective Computing, namely the affective domain and modelling. Section 3 surveys the most important studies carried out over the past 7 years with regard to automatic and spontaneous recognition of people’s affective states and personality traits through smartphones, whereas Section 4 discusses current trends and challenges in the field. Finally, Section 5 concludes the survey by providing summary statistics in terms of modalities and affective models used in the referenced studies. 2. Background Recent developments in mobile computing have resulted in great advances in the area of social, health and psychological sciences. As both the ongoing revolution of sensor capabilities embedded into the latest smartphones, characterising a research domain called Mobile Sensing, and the evolution of Affective Computing and HCI are taking place, the emerging products of these prominent research areas provide powerful tools to analyse people’s social behaviour. In return, scientists, having as an ultimate goal the individuals’ wellbeing and sustainable living, are keen on locating and isolating the unique social and personal conditions that impact people’s affective state and, vice versa, on discovering personality characteristics that affect social behaviour. 2.1. Affective computing Affective computing is a subfield of HCI named after the field of psychology in which ‘‘affect’’ is basically a synonym for ‘‘emotion’’. The fundamental concepts of affect recognition, interpretation and representation were firstly introduced by R.W. Picard in 1997 [30] who understood the potential of computers to recognise, understand, express and reproduce human emotions. As Picard elaborated on many works, in the interaction between human and computers, a device has the ability to detect and appropriately respond to its user’s emotions [31]. Indubitably, a computing device with today’s capacity could gather cues to user emotion from a plethora of sources, such as facial expressions, posture, gestures, speech, the force or rhythm of key strokes and the temperature changes of the hand on a mouse. All these sources could signify changes in users’ emotional states and can effectively be detected, interpreted and correlated by a computer [3]. As a matter of fact, the recognition and detection of human affect is usually referred to with the generic term ‘‘emotion detection’’ or ‘‘emotion recognition’’ which actually denotes the task of recognising and classifying a person’s emotion, such as anger, happiness or stress, across all possible channels of communication (modalities). Emotion recognition leverages techniques from multiple areas, such as signal processing, machine learning, and computer vision and may apply to a wide area of sciences, from psychometry and sociology studies to marketing and surveillance applications. Ultimately, the research field of HCI, and particularly affective computing, is about studying and developing systems and devices that are able to recognise, interpret, process, correlate and simulate human affects [32].

81

2.1.1. The affective domain Throughout psychology literature, the word ‘‘emotion’’ is often applied to a wide variety of phenomena, such as passions, sentiments, temperament, personality traits and moods. Although these words are regularly used interchangeably, they do in fact refer to specific and different experiential phenomena, all of which are used to be comprised under the generic term ‘‘affect’’ (or affective state) [33]. According to some psychology theorists, emotion, mood and affect are the three main constructs that constitute the affective domain and the study of this affective domain defines an active area of research referred as affective phenomena [34]. Although the differences between the notions of emotion, mood and affect are often indistinguishable, they are not insignificant. The history of associated research on affective states and phenomena is long, the proposed theories are prolific and diverged, and the associated literature is massive. Henceforth, only a modest amount of this information, geared towards the need to familiarise the reader with the terminology used throughout this survey, will be presented. 2.1.2. Core affect, emotion, mood, personality and sentiment Core affect. The term ‘‘core affect’’ was coined by Russell and Barrett in 1999 [35] where it was used ‘‘to refer to the most elementary consciously accessible affective feelings that need not be directed at anything’’. In other words, the states experienced as simply feeling good or bad, energised or enervated are called core affect and influence human reflexes, perception, cognition, and behaviour. According to [35], core affect ebbs and flows over the course of time and, in most of the cases, it is not necessarily consciously directed at anything. Russell again, in [36], proceeded in defining that the combination of the two dimensions of pleasure/displeasure (pleasure or valence) and activation/ deactivation (arousal or energy) is what is actually called core affect. He also explained that although core affect describes moods, it is simultaneously the core of all emotion-laden occurrent events and as such a ‘‘prototypical emotional episode’’ [35] includes a large change in core affect. For example, one cannot be ‘‘prototypically’’ afraid without feeling great displeasure and activation. Emotion and mood. The distinguishing characteristics of emotion and mood have already received much attention from the academic community and most scholars agree that these constructs represent closely related but distinct phenomena [37]. According to the descriptions defined by psychology theorists, emotions are elicited by something, are reactions to something and are generally about something [38]. On the other hand, one distinguishing feature of mood is that it typically lasts longer than an emotion and, unlike emotion which follows its eliciting stimuli closely or even instantaneously, a mood seems usually unrelated from its undermined cause. In other words, mood lasts longer than an emotion, is less specific, less intense, less likely to be triggered by a particular event and its cause may not always be easily identifiable [34,39–41]. For a comprehensive study on all the features that differentiate between emotion and mood an interested reader may refer to [37]. Personality. Since the early 1980s, an era characterised with the booming of interest in the emotion psychology and the in-depth investigation of emotions from a personality perspective, the two historically largely separate fields of personality psychology and emotion psychology are gradually being integrated, to the benefit of both fields [42]. Personality is an abstraction used to explain, in a consistent and coherent manner, an individual’s pattern of affects, cognitions, desires and behaviours. What one feels, thinks, wants and does, changes from moment to moment and from situation to situation, but shows a patterning across situations and over time that may be used to recognise, describe and even to understand a person [43]. Just as an emotion represents an integration of feeling,

82


action, appraisal and wants at a particular time and location, so does personality represent integration over time and space of these components. As Revelle and Scherer explain in [44], a helpful analogy is to consider that personality is to emotion what climate is to weather. That is, what one expects is personality and what one observes at any given moment is emotion. Personality traits. Trait theories have arisen in recent years and constitute a major approach to the study of human personality. According to this perspective, ‘‘personality traits’’ can be defined as habitual patterns of behaviour, thought and emotion [45], and have been popular models for quantifying personality. More precisely, the term ‘‘personality trait’’ or ‘‘dispositional trait’’ refers to broad, internal, and comparative features of psychological individuality that account for consistencies in behaviour, thought, and feeling across situations and over time [46]. In other words, personality traits are those intrinsic differences that remain stable throughout most of our life and define the constant aspects of our individuality. In some theories and systems, traits are something a person either has or does not have, but in many others, traits are dimensions such as extraversion vs. introversion, with each person rating somewhere along this spectrum [45]. Personality traits and emotions are notions so intimately tied that it is often difficult to be distinguished. Yet, it has only been in the last decades that systematic associations have been made between the structure of emotions and the structure of personality traits [47]. According to these theories, the emotion system is considered as a subsystem of personality, and therefore the emotional dispositions, at least those that are stable and general, are a species of personality traits [42]. Sentiment. On the other hand, a sentiment, according to the Cambridge dictionary [48], defines a thought, an opinion, or an idea based on a feeling about a situation, or a way of thinking about something. Sentiment analysis is an ongoing area of research in text mining and natural language processing fields that analyses people’s opinions, sentiments, evaluations and attitudes. Most of the times, since the identification of sentiment is often exploited for detecting polarity, they are combined under the same umbrella with the term ‘‘opinion mining’’ or even used as synonyms in some cases [19,49]. Overall, sentiment reflects the deeper psychological state of the holder, enabling people to reason why they like or dislike something [50]. In a glance. Desmet in his work [33] describes quite comprehensively, in terms of time and intention, the differentiations between the four main types of ‘‘affective states’’, that is emotions, moods, sentiments, and emotional or personality traits. According to this, emotions and moods are relatively short in duration while sentiments and personality traits display themselves over longer periods of time (i.e. dispositional). Emotions are directed at something, while moods are not directed at anything in particular. Emotions last a short time, ranging from seconds to minutes and they can be triggered by sights, smells, sounds, events or thoughts. Moods last longer than emotions, usually for hours or days, but again for a limited amount of time, they have combined causes rather than being elicited by a particular event, and they are not directed towards a particular object. On the other hand, sentiments are directed at something and they constitute our attitudes, likes and standards. Finally, emotional traits, which are often called ‘‘character’’ or personality traits, are personality characteristics that manifest for the long term. They are like moods, but persist for long enough time so that people can be characterised by their expression. When states become dispositional, they endure over time through different contexts, which is the case with sentiments and personality traits.

2.1.3. Affect modelling, classification and measurement The research in affect classification and measurement mainly focuses around two fundamental perspectives, one specifying that emotions, and affects in general, are discrete and separately identifiable (distinct-states or categorical approach) and the other one specifying that emotions can be characterised on a dimensional basis in groupings (dimensional approach). The distinction between the distinct-states approach and the dimensional approach is of fundamental importance since the former proposes that different emotions arise from separate neural systems, whereas the latter suggests that a common and interconnected neurophysiological system is responsible for all affective states [34,51]. In addition to the above two perspectives, there is also the appraisal approach for modelling emotions and the established Five Factor model for describing personality, all of which are discussed in the following paragraphs. Distinct-states models. Paul Ekman and his colleagues, based on Tomkins’ interpretation [52] of Darwin’s ideas of emotions and their expressions [53], concluded that there are six basic emotions: anger, disgust, fear, happiness, sadness and surprise [54]. Ekman explained that there are particular characteristics attached to each of these emotions, allowing them to be expressed in varying degrees. Each emotion acts as a discrete category rather than an individual emotional state. This theory also suggests that emotions are culturally universal [55,56] although many studies have challenged this universality [57–59]. Computer programmers often use as a guide Paul Ekman’s Facial Action Coding System [60], which is a common standard to systematically categorise the physical expression of emotions and it has been proven very useful to psychologists and animators. Additionally, a widely accepted research tool for investigating and measuring moods is the Profile of Mood States (POMS) [61], which remains one of the most frequently used measures of mood. It consists of a 65-item inventory that assesses six dimensions of the mood construct: anger, confusion, depression, fatigue, tension, and vigour. Nevertheless, there is no evidence, either explicit or implicit, in the theoretical basis of the POMS that these six distinct states collectively capture the entire content domain of mood [34]. Dimensional models. In contrast to distinct-states theories, the dimensional models of emotion propose that emotions can be distinguished in the form of two or three dimensional scales. The most popular dimensional models, described in the following paragraphs, are the Circumplex model, the Vector model, the PANAS model and the AD ACL model. The Circumplex Model of Affect, proposed by Russell in 1980 [51,62], specifies that all affective states arise from two fundamental neurophysiological systems: one relates to valence (a pleasure–displeasure continuum) and the other relates to arousal or alertness. These two orthogonal and bipolar dimensions – affective valence and perceived activation – define the affective space. Each affective state can be understood as a linear combination of these two basic constituents (valence and arousal) in different degrees, and in a manner that arousal represents the vertical axis, valence represents the horizontal axis, while the centre of the circle represents a neutral valence and a medium level of arousal. The Vector model appeared in 1992 and specifies that there is an underlying dimension of arousal and a binary choice of valence that determines direction [63]. This results in two vectors that both start at zero arousal and neutral valence and proceed as straight lines, one in a positive and the other one in a negative valence direction. In this model, high arousal states are differentiated by their valence, whereas low arousal states are more neutral and are represented near the meeting point of the vectors. The positive and negative affect schedule (PANAS) was proposed by Watson, Clark and Tellegen in 1988 [64] and it comprises a measure for general affective states. In the PANAS model, positive


affect is a dimension that ‘‘reflects the extent to which a person feels enthusiastic, active and alert’’. In contrast, negative affect has been described as a ‘‘general dimension of subjective distress and unpleasurable engagement’’. Consequently, the vertical axis represents low to high positive affect and the horizontal axis represents low to high negative affect. While the original PANAS model suggested that positive and negative affect are two separate systems (orthogonal), this hypothesis of complete independence between PA and NA has been rejected by following studies, such as [65]. Watson and Tellegen, the pioneers of PANAS model, promptly recognised the strengths and the limitations of dimensional models and therefore proposed that the affective domain could be described as having a hierarchical structure [66]. According to this hierarchical structure, broad dimensions can capture the differences and similarities between states on a macroscopic scale, whereas, for identifying in a microscopic scale the uniqueness of the different states, the distinct-states approach is also required. This position was further elaborated in [67] where Watson and Clark emphasised that these two basic approaches – dimensions and discrete affects – are not incompatible or mutually exclusive; rather, they essentially reflect different levels of a single, integrated hierarchical structure in such a manner that each of the higher order dimensions can be decomposed into several correlated yet ultimately distinct affective states [67]. The Activation Deactivation Adjective Check (AD ACL) model is a two-dimensional structure proposed by Thayer [41] and it has been used in several studies to assess affective responses to bouts of physical activity. The model again postulates two bipolar dimensions, one named energetic arousal (EA) which extends from energy to tiredness and the other, named tense arousal (TA) which extends from tension to calmness. While it has been acknowledged by researchers that this model fully overlaps the PANAS model, it has been also suggested that the structure of this multidimensional measure should approximate the circumplex model [68]. Notwithstanding the dimensional models described, some critics rejected the basic assumption that affect can be sufficiently described by two orthogonal dimensions and they advocated a three dimensional model in which valence (ranging from unpleasant to pleasant), calmness (ranging from restless/under tension to calm/relaxed) and energetic arousal (ranging from tired/without energy to awake/full of energy) form the basic dimensions. Although these dimensions are substantially correlated, they cannot be reduced to a two dimensional model [69,70]. For further reading, a thorough comparison of the dominant dimensional models of affect was made by Rubin and Talarico in [71], whereas, Ekkekakis [38] carried out an in depth categorisation of affect, mood, and emotion measurement. Appraisal-based models. The appraisal-based approach, originally coined by Arnold who first used the term ‘‘appraisal’’ [72], can be conceived as an extension to the dimensional approach. According to appraisal theory, emotions are generated through continuous, recursive subjective evaluation of both our own internal state and the state of the outside world [73]. Stated differently, emotions are extracted from our evaluations (appraisals or estimates) of events that cause specific reactions in different people [74]. The advantage of this approach is that it does not limit emotional states to a fixed number of discrete categories or to a few basic dimensions. Instead, it focuses on the variability of different emotional states, as produced by different types of appraisal patterns. Therefore, it is possible to differentiate between various emotions and to model individual differences [73]. The Big Five model. During the past century, many trait theories have been proposed in an attempt to describe and measure human personality. Particularly, discovering the core dimensions of personality and its encompassed theoretical perspectives as well as

83

assessing their taxonomy and measurement scales have been the subject of research for many psychology pioneers [75]. The most prevailing of all theories in measuring personality is the Five Factor model (FFM), also known as the Big Five personality traits [76– 79], which emerged to describe the essential traits that serve as the building blocks of personality. In the early 1990s, Goldberg proposed the Big Five model [79] based on common language descriptors of personality which are grouped together using a statistical technique called factor analysis. At about the same period more psychology theorists, like McCrae and John [77], concluded to about the same results by studying people from more than 50 different cultures. Nevertheless, the initial model was advanced by Tupes and Christal in 1961 [80] who claimed to have found just five broad factors of personality. The Big Five is a hierarchical model of personality traits with five broad categories (factors): Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (sometimes called Emotional Stability). Beneath each of the proposed global factor, a number of correlated and more specific primary factors are claimed. Thus, each bipolar factor (e.g., Extraversion vs. Introversion) summarises several more specific facets (e.g., Sociability) which, in turn, subsume a large number of even more specific traits (e.g., talkative, outgoing) [81]. In other words, the Big Five structure indicates that these five categories represent personality at the broadest level of abstraction, and suggests that most individual differences in human personality can be classified into these five broad domains where each domain (dimension) summarises a large number of distinct more specific personality characteristics. For example, the dimension of neuroticism includes the specific characteristics of mood swings, anxiety, moodiness, irritability and sadness [75]. For assessing the Big Five personality and measuring individual’s scores in the Big-Five dimensions several rating tools (instruments) have been developed. The most comprehensive instrument is Costa’s and McCrae’s NEO Personality Inventory, Revised (NEOPI-R) [82], a 240-item inventory which permits measurement of the Big-Five domains and six specific facets within each dimension. However, since this tool is too lengthy to be completed and hence unsuitable for many research purposes, numerous shorter versions have emerged and are commonly used. Among the popular shortened versions are the NEO Five-Factor Inventory (NEO-FFI) [83], which comprises of 60 items and is designed to take 10–15 min to complete, and the Big Five Inventory (BFI-44) [75], a 44-item self-report questionnaire. The briefest one and most widely used in situations where very short measures are needed is the Ten-Item Personality Inventory (TIPI) [81], a 10-item measure of the Big-Five dimensions. Besides those mentioned, researchers, acknowledging the cost involved in using these proprietary personality instruments, have collaborated for the development of a public domain fee-free instrument called the International Personality Item Pool (IPIP) [84]. Modelling wellbeing. Apart from the aforementioned popular models for measuring and assessing affective states, there are several popular theories about modelling wellbeing. Psychology literature is scattered with correlations between emotions, moods, affective states, personality traits and wellbeing and the body of research that investigates those relationships is huge [85–89]. In most of the times, wellbeing is often used in psychology interchangeably with the concept of happiness [90] whereas there are strong pieces of evidence that positive emotions trigger upward spirals towards enhanced emotional wellbeing [91]. Hence, wellbeing’s place in this survey should not be disregarded. As a matter of fact, it is the notion of Subjective Well-Being (SWB) that is used in research literature as a substitute for the term ‘‘happiness’’ [92]. SWB is a multidimensional approach of measuring wellbeing and it encompasses how people evaluate their own

84


lives in terms of cognitive and affective explanations. Although the tripartite model of SWB distinguishes between cognitive wellbeing (e.g., life satisfaction) and emotional wellbeing (positive emotions and negative emotions), distinctive correlates between emotional and cognitive wellbeing exist [93]. Ultimately, there are numerous and contradicting perspectives in the theory of wellbeing. For example, for a long time theorists believed that happiness, a notion often used for wellbeing, is a trait. Although this viewpoint has been criticised extensively, it is now commonly accepted that stable personality traits can influence wellbeing, and thus subjective wellbeing has both trait-like and state-like properties. The current working model of researchers in the field is that not only personality predisposes people to certain affective reactions but current events also influence one’s current levels of wellbeing [85]. 2.2. Ubiquitous computing and mobile sensing Up to 10 years ago, the basic approach of affective recognition process was to observe a person’s patterns of behaviour via sensors such as cameras, microphones or pressure sensors applied to objects the user is in contact with (mouse, chair, keyboard, steering wheel, toy) and use computers to associate these patterns with probable affective state information [94]. Nevertheless, the boom of mobile technology during the last decade has radically changed the way machines sense human emotions. Mobile devices do not only constitute just a telecommunication device in people’s everyday lives but, due to the recent technological advances, they are gradually replacing computers in all aspects of the digitised world. This large shift in the world of personal computing indicates clearly that users are willing to sacrifice performance in the name of portability and price, as smartphones are cheaper (since their cost is often folded into the cost of a multi-year contract with a mobile services provider), lightweight and can fit in a purse or pocket. Still, high end smartphone processors today are faster than PC processors from years ago, or very low end PC processors today and, according to the latest statistics, mobile web browsing has already overtaken the web browsing in desktops. Since the mid-1990s when IBM introduced ‘‘Simon’’ the first ever smartphone [95], today’s smartphones with multi-core CPUs and gigabytes of memory are placing more processing capabilities in individuals’ pockets than computers of past decades placed on people’s desktops [96]. Besides the impressive features of CPU and memory, modern smartphones are programmable devices equipped with a range of cheap though powerful embedded sensors, such as gyroscopes, GPSs, accelerometers or magnetometers, which enable the development of personal and community-scaled sensing applications [97]. This rise of rich-sensor smartphones has enabled the recent birth of mobile sensing, an emerging and exciting interdisciplinary research field which is part of the ubiquitous computing concept and requires significant advances in mobile computing, machine learning and systems design [98]. There are several types of mobile sensing such as individual, participatory, opportunistic, crowd, social, and on top of that, the object of sensing can either be people-centred or environment-centred [99]. Meanwhile, ubiquitous computing has already changed significantly the way computing and communication resources are used nowadays by fostering the principles of connectivity with any device, in any location and in any format [99]. Supporting these principles, ubiquitous computing pioneer, Weiser [100], who first coined the term ubiquitous, described the requirements to be met in order devices to be considered as ubiquitous, which are to be cheap, low-power computers that include equally convenient displays, a network that ties them all together and software systems implementing ubiquitous applications.

As a matter of fact, smartphones represent the first truly ubiquitous mobile computing devices since their mobility and afforded computational power allow users to interface directly and continuously with them, more than ever before, while their embedded sensors open up smartphones to new advances across a wide spectrum of application domains which till recently were possible only through wearable sensors [98]. Wearable sensors, although they are portable and promising, are still not viewed as personal companions. In contrast, sensor-enhanced smartphones always accompany users since they are willingly carried by a large fraction of people in developed countries and therefore constitute a rich information source [101,102]. Consequently, this mobile evolution does not only outline an era in which powerful machine-learning algorithms for statistical inferences using sensor data can be designed to run on commodity phones but, first and foremost, it facilitates the monitoring and analysis of human behaviour and social interactions on a large scale and in nearly real time [96,102]. In particular, the volumes of multimodal data collected from people’s daily use of smartphones through sources such as GPS, Call logs and Bluetooth, enable the development of data collection tools to record various behavioural aspects of users, ranging from how a device is used across different contexts to the analysis of spatial and social dimensions of users’ everyday lives [103]. This allows unobtrusive and cost-effective access to previously inaccessible sources of information on everyday social behaviour, such as physical proximity of people, phone usage and patterns of movement [101], and provides new opportunities to researchers as it allows them not only to understand the impact of context on user behaviour, but also to study individual differences, such as users personality. In turn, it can enable the design of communication features and multiple mobile applications that are tailored to the individual needs and preferences of a user [103]. Specifically, personality has been found to influence the behaviour of an individual in social interactions since personality traits play a central role in describing a person. Based on this proposition, several recent studies have investigated personality traits and their relationship to the use of the Internet and the social media [104]. Nevertheless, most of these studies neglected the fact that, unlike desktop computers, smartphones are small mobile devices whose nature as a primary communication tool should not be forgotten since people carry them around naturally and use them in the everyday management of social relationships [101]. In this context, Eagle et al. in [105] carried out an early research study demonstrating the potential of data collected from mobile phones to provide insight into the relational dynamics and behaviour patterns of individuals. Quite recently, scientists demonstrated that physiological parameters such as heart and breathing rates can be recovered from a smartphone via accelerometer measurements while the person is carrying it in different locations or using it during different activities [106]. Besides these potentials, mobile phone data, as described in the following chapter, can also provide meaningful associations between mobile phone usage and affective states, such as personality traits and emotions of individuals. 3. Mobile affective sensing and relevant research As described above, inferring, recognising and processing people’s affective information, such as emotions, moods and personality, based on features extracted from their smartphones has been the object of extensive research by computer scientists, information engineers, applied psychologists and other relevant disciplines and, on that ground, the associated literature is rich and diverse. Technologies and methods used for accomplishing the concerned tasks vary significantly both in terms of employed smartphone modalities as well as in terms of chosen machine learning algorithms to infer and classify affective status. On top of this, pilot


applications built for demonstrating the potentials of each model vary hugely in regard to experimental settings and environment. In the following paragraphs, the most prevailing, novel and promising works in the area of emotion recognition and personality inference through smartphones are presented, categorised by the affective model adopted to classify subject’s emotions, moods or personality, and described by the type of mobile usage or sensors employed to provide the necessary data for inference. An overall representation of the referenced studies, described by their classification model, the number of subjects involved, the duration of experiments and the categories of data collected through smartphones, is presented in Table 1. It is noted that the greyed out boxes represent data types not originated from the use of smartphones but rather from non-smartphone modalities, i.e. wearable devices. While affect recognition based solely on wearable sensors is a large and autonomous field of research and the focal point of several research teams worldwide, this survey references such modalities as long as they are part of an experiment where smartphone data are captured as well. 3.1. Big-Five personality traits recognition Modern psychologists and computer scientists are in a constant pursuit of automatic personality recognition and classification and to this end, they utilise all available information sources. For example, researchers in [107] analysed audio from meetings in order to classify participants according to the Big Five personality traits model. Quite recently, mobile sensing technology has been also employed in order to investigate similar phenomena. According to a study for the relationships between personality and mobile phone use [108], it was found that personality traits can explain patterns of mobile phone usage, for instance extraverts and perhaps disagreeable individuals were less likely to value incoming calls, while disagreeable extraverts also reported using mobile phones more and spent more time adjusting ringtones and wallpapers. Vice versa, the process of inferring personality traits from patterns of smartphone usage is the purpose of several research works in mobile affective computing. One such work is described in [109] where the authors analyse the relationship between smartphone usage and personality traits. The use of web, music, video, maps and other applications together with the traditional call and SMS usage, the proximity information derived from Bluetooth and the use of camera, are employed to demonstrate that aggregated features obtained from smartphone usage data can be indicators of the Big-Five personality traits. Additionally, TIPI questionnaire [81] is used to measure self-perceived personality and assess inference results which were found rather promising. The research team extended its work in [103] both in terms of experiment’s population and duration as well as in terms of the experimental framework which was enhanced in order to anonymise sensitive information extracted from usage logs and phone sensors. Similarly, in [110] the use of anonymised mobile phone call usage data to automatically infer users’ personality (as characterised by the Big Five model) in a privacy-preserving manner is described. The necessary information is extracted from call detail records, including variables obtained from social network analysis of the calls, and a model selection applied to come up with the selected features for predicting users’ personality. The chosen features included the duration of received phone calls, the number of received and placed phone calls as well as the number of SMSs and MMSs sent or received at different times of the day. Participants filled in a 50-item version of the IPIP public domain Big Five questionnaire [84] to collect ground truth about participants’ personality profile. Staiano et al. [111] carried on the work further by extending the social network structural properties for

85

the purpose of inferring and predicting personality and other psychological variables through contextual data collected from mobile phones. The data used in the study are proximity data derived from Bluetooth sensor, calls logs from which a social network is built and data coming from surveys where participants enter self-reported information about personality (Big Five) and relationships among subjects. To meet a similar goal for predicting users Big-Five personalities, and thus emotional stability, in [112] researchers demonstrated that user personality can be reliably inferred from basic information accessible from all commodity smartphones. The features used in the research fall under 5 broad categories: basic phone use (e.g., number of calls, number of texts), active user behaviours (e.g., number of call initiated, time to answer a text), location (radius of gyration, number of places from which calls have been made), regularity (e.g., temporal calling routine, call and text inter-time) and diversity (call entropy, number of interactions by number of contacts ratio). Participants completed the Big Five Inventory (BFI-44) [75] for measuring their personality and the classification reached up to 61% accuracy on a three-class problem on each of the five personality dimensions. 3.2. Recognition of distinct affective states Apart from inferring individuals’ personality, data collected through smartphones have been extensively used to deduce users’ distinct emotional states, such as happiness or anger, or to classify users in a two or three-class classification problem like stressed/not stressed or good/poor/neutral mood. An early work in emotion detection through smartphone acquired data is described in [113] where the authors propose a system that can recognise automatically high-level contexts, like users’ activity and emotion, by employing Bayesian networks which are commonly used to reason for reliable results in uncertain environments. The Bayesian network built was based on information collected from mobile devices, such as call and SMS logs, GPS coordinates and Bluetooth, and tried to infer 9 distinct emotions based on activity inference and, as the authors claim, without prior manual annotation. However, neither a justification for the affect recognition model used is provided nor the results of the emotion recognition accuracy are documented. 3.2.1. Ekman’s distinct states model recognition Following Ekman’s conclusions that there are six basic emotions (anger, disgust, fear, happiness, sadness and surprise), many research studies in inferring and classifying users’ emotions from smartphones have been inspired accordingly. For instance, in [114] EmotionSense, a mobile sensing platform for recognising user’s emotions based on built-in smartphone sensors, is presented. This platform gathers participants’ emotions as well as proximity and patterns of conversation by processing the outputs from the sensors of commodity smartphones. More precisely, the sensors employed were accelerometer, Bluetooth and GPS which, together with microphone inputs, were used to infer participants’ 4 emotional states contained in Ekman’s basic set, plus one neutral, based on users’ movements, location, proximity and conversation with other users. Ekman’s six basic emotions model, plus one neutral, was used in [115] where the authors built an Android application to recognise users’ emotions by inconspicuously collecting and analysing user-generated data from different types of smartphone sensors and utilities. The features extracted from smartphone data can be categorised into two types, the behaviour data types (such as typing speed, touch count and device’s shake) and the context of user data types (such as location, weather) and they were collected while users were using a certain application on their smartphones

86

Table 1 Smartphone affective recognition and the involved utilities/sensors. Duration in (d)ays, (w)eeks, (m)onths. Greyed out cells imply external data.

E. Politou et al. / Computer Science Review 25 (2017) 79–100 (continued on next page)

Table 1 (continued)


87

88


(i.e. writing tweets). The system then proceeded in the emotion detection of the seven emotional states by applying the extracted features to a Bayesian Network classifier for emotion recognition. Likewise, in [116] the authors proposed an emotion recognition framework that demonstrates how simple touching behaviours can be used for recognising smartphone users’ emotional states. Particularly, a demo Android application was implemented in order, first to monitor users’ touching behaviour by collecting sensory data from touch panel, accelerometer and gyroscope sensors, and then to infer users’ emotions classified by Ekman’s six basic emotions plus the neutral one. 3.2.2. Stress recognition Over the past few years, a respected amount of research has been carried out in regard to the spontaneous detection of stress in individuals [117]. Nevertheless, it was only the last 4–5 years that the use of smartphone acquired data for stress detection has begun to attract high interest by the scientific community. In particular, the work described in [118] focuses on the problem of stress detection based on behavioural metrics derived from users’ mobile phone activity and from additional indicators, such as weather conditions and personality traits collected through surveys. Mobile phone usage data consist of call logs, SMS logs, proximity data obtained by scanning near-by phones and other Bluetooth devices every five minutes and, along with personality and weather data, were used in a two-class stress recognition problem. Furthermore, the work conducted by Sano and Picard in [119] aimed at finding physiological or behavioural markers for the binary classification problem of stress recognition by using a wrist sensor (accelerometer and skin conductance sensors), mobile phone usage (calls, SMS, location and screen on/off) and user surveys. The authors applied correlation analysis to find statistically significant features associated with stress and they used machine learning to classify, with a satisfactory accuracy, whether the participants were stressed or not. The study has been further elaborated in [120] by increasing experiment’s sampling period and participants’ population to collect more intensive multi-modal data, including perceived stress, sleep, personality, physiological, behavioural and social interaction data, which are all important factors in inferring stress. The authors demonstrated successfully the capability of wearable and mobile devices to carry information that might be used by individuals to make better predictions about the impact of behavioural choices on academic performance, sleep, stress and mental health. Stress recognition can be also inferred more directly through cameras and microphones embedded in subjects’ smartphones. StressSense [121] is a voice based stress detection system for recognising the cognitive load related stress in job interviews and outdoor job execution tasks. The problem was modelled as a two-class detection problem, stressed versus neutral speech, with adequate results for both indoor and outdoor environments. It should be noted though, that while there are currently several research papers focusing on stress detection through cameras and microphones, we are not going to extensively describe them in the current survey since they are based on intrusive methods for acquiring data, such as recording audio and video of participants, rather than utilising more discreet ways for data extraction. Bauer et al. in [122] were addressing the problem of whether differences between stressful and non-stressful periods can be detected in information readily available on a smartphone. The study aimed at detecting time periods when a user is under continuous stress and for this purpose they used location data coming from GPS and WiFi sensors, social interaction data derived from Bluetooth, and call and SMS behaviour parameters about incoming and outgoing calls and SMSs contained in call and SMS logs. Rather

than trying to classify when a user is stressed or not, this work focused on discovering behavioural features for detecting behaviour modification of participants during the exam time. Hence, instead of providing overall classification accuracy, the authors provided analytical patterns of behaviour associated with location, social interaction and mobile usage. In [123] the authors presented a solution for assessing stress experience of people using features derived from smartphones and wearable chest belts. The study used information of audio, physical activity and communication data collected during workday from participant’s smartphones (through microphone, GPS and call logs), as well as heart rate variability data collected from a chest belt worn during sleep at night, in order to solve the three-level (low, moderate and high perceived stress) classification problem of stress recognition. The smartphone features used in the study were selected based, on the one hand, on the fact that people with neurotic personality trait have difficulty in managing stress [124], and on the other, on the features provided by Chittaranjan in [109] for inferring neurotic personality traits. It is worth noticing that although the authors’ original intentions were to include features from smartphone’s accelerometer sensor, address book, calendar and battery, none such feature was selected during the final feature selection process. This is due to the fact that in the real working environment where the system has been evaluated, ‘‘most people do not use their own smartphones for business purposes. Instead, the smartphone lies on the desk, and all business contacts and calendar events are stored on the computer’’. In the context of multimodal stress detection, the work carried out in [125] attempted to measure the levels of acute stress in humans by analysing their behavioural patterns when interacting with technological devices. To this end, a user environment, equipped with sensors and devices acquiring various kind of information from the user in a non-intrusive way, was set. While the user conscientiously interacts with the system by playing a mentally challenging mobile game, a parallel and transparent process takes place in which the extracted information is sent, in a synchronised way, to a platform that converts the sensory information into useful data and hence, allows for a contextualised analysis of the operational user data. Sources of information acquired from the mobile device include touch pattern, touch accuracy, touch intensity and touch duration along with hand gestures and movements from accelerometer. Supplementary, the amount of user movement, which represents how and how much the user is moving inside the environment, was extracted from the video camera. Although the classification accuracy of user’s touching behaviour as stressed or not stressed was considered satisfactory, the results cannot be expanded since the experiment took place in a controlled laboratory setting and not in the wild. Oriented towards the prediction of stress in working environments, researchers studied the use of only one smartphone built-in sensor, the accelerometer, together with supervised learning classifiers in order to detect behaviour correlating with subject’s stress levels [28]. According to the authors, the choice of accelerometer was made firstly because it raises fewer privacy concerns compared to other smartphone data, and secondly due to its low power consumption. The achieved results were close to those found throughout the literature, with the only difference being that here a single sensor was used. In the same context of predicting work related stress, work conducted in [126] aimed at showing that patterns of smartphone application usage are highly correlated with self reported stress levels and, consequently, they can be used to predict stress levels within the workplace context. For that purpose, a smartphone application was built and installed in participants’ smartphones having a twofold purpose, firstly to record application usage, like


emails, calendar and browsers, and secondly to capture perceived stress levels through prompted questionnaires. Detecting stress in working environments is the goal of another recent study [127] where authors, by employing transfer learning methodologies, proceeded in combining various sensing modalities to model stress behaviour based on non-obtrusive data obtained from smartphones. To collect information relevant to working environments a POMS scale questionnaire [61] was provided to the users and data derived from these self-reports were used as the ground truth. Measurements of stress levels were classified into three classes, namely low, moderate and high stress, and the extracted features from smartphone collected data pertained to physical activity level stemming from accelerometer sensor, location patterns stemming from WiFi sensor, social interaction stemming from verbal interaction captured by microphone, and social activity stemming from call and SMS usage, were used. Inspired from the two previously mentioned studies, the research team continued in classifying perceived stress of employees based on accelerometer data extracted from smartphones during phone conversations [29]. Their study differs from the two previous ones both in terms of using semi-supervised learning to complete the models for subjects with missing data as well as in terms of introducing intermediate models to predict mood variables to improve the accuracy of the predictions. In two more studies, presented in [128,129], two methods for stress assessment relying only on the analysis of how people use their smartphones are introduced. In the first one, researchers conducted an experiment in a controlled laboratory setting where only smartphone performed gestures on the phone e.g., ‘‘tap’’, ‘‘scroll’’, ‘‘swipe’’ and ‘‘text writing’’ were taken into account for the evaluation. Participants were provided with specific tasks and the stress levels were classified as a binary classification problem of stressed or not stressed. Next, in [129] the same team proceeded to an experiment in a non-controlled environment where features related to the smartphone usage, like the number of times people turn on and off the screen, the amount of time spent using social applications in smartphones, the number of screen touches, the physical activity and the light exposure were considered. The affect representation model for this experiment was a three class model for stressed, normal and relaxed affect states and the results showed that the most significant features were those related to application usage by individuals during the day, meaning that the type of application people use is strongly correlated with their stress level. Under the work carried out in [130] a system for predicting the negative states of depression, anxiety and stress based on mobile phone usage patterns, which included calls and application usage, was proposed. The system leverages machine learning techniques, as most of the rest surveyed papers does, and uses a binary scale representation, from low to high, for each of the measured emotions. One of the latest studies in stress recognition domain is the one specified in [131], which describes a mobile application for the early recognition and management of stress based on the continuous monitoring of heart rate variability obtained through a biosensor and contextual data, such as activity and location obtained through smartphone sensors. The application categorises stress levels of individuals into low, medium or high stress, but there are not any results provided for the evaluation of the proposed solution.

3.2.3. Recognition of happiness, boredom and other distinct affective states Beyond stress and Ekman’s basic emotions, many studies, like the one described in [113], focus on recognising isolated distinct

89

affective states of subjects, such as happiness, boredom or sadness, using binary or triadic affect classification. For instance, the authors of [132] built a model for automatic recognition of daily happiness as a 3-level classification problem (happy, neutral and unhappy) based on information about people’s activity detected through their smartphones. They used an extensive set of indicators obtained from mobile phone usage data (call logs, SMS and Bluetooth proximity data) as well as indicators coming from weather factors obtained through the web along with self-reported surveys about Big Five personality traits and daily happiness, and they provided compelling evidence that individual daily happiness can be reliably predicted from smartphone usage data and additional indicators. Similarly, the authors in [133] built a system for detecting when students become unhappy in the form of a binary classification problem (happy vs. sad). The system was designed in order to drive interventions according to its detection outcome so as mitigate the risks of depression. The collected data consisted, on the one hand, of physiological signals (Electrodermal Activity (EDA), skin temperature, and 3-axis accelerometer data) coming from a wearable sensor worn by the participants almost continuously for the duration of the experiment, and on the other, of gathered smartphone data coming from call logs, SMS messages, number of times the screen was turned on and off as well as location information derived from GPS and WiFi. Surveys about their behaviours, activities, and wellbeing were also completed twice daily by the participants. Furthermore, the study not only examined which features provide the most information about happiness and how they affect it, but it also investigated the relationship between happiness and other components of wellbeing, such as health, stress, and energy. Recently, Pielot et al. in [134] demonstrated the detection of boredom by utilising mobile phone usage. The authors showed that a user-independent machine-learning model of boredom – leveraging features related to recency of communication, usage intensity, time of day, and demographics – can infer boredom with quite satisfactory accuracy. Results indicated that boredom is highly correlated with smartphone usage extracted features, like the time elapsed since last call, the time elapsed since last SMS, the battery level, the bytes of transmitted data, whether or not the screen is covered and finally the usage of specific apps (email, browser, instagram etc.). Their findings suggest that being contacted by others is generally correlated with being less bored. Contacting others, however, is more likely to happen while being bored. They also concluded that the higher the usage intensity is, the higher the chance for the emotional state of boredom becomes. In the related and interdisciplinary context of emotion recognition and social network mining, the authors of [135] studied the problem of emotion prediction in social networks and proposed a method for modelling and inferring individuals’ mood based on the emotional states in the mobile social network. They analysed communication (by SMS text and call), calendar, alarm, WiFi signal and GPS location data coming from smartphones, as well as activity and mood information collected from user’s annotations on mobile network, in order to build a dynamic continuous factor graph model for predicting coarsely user’s mood (positive, negative and neutral). In another experiment [136], researchers employed machine learning techniques to recognise users’ emotional states from patterns of finger stroke behaviour when emotionally loaded pictures were presented on their smartphones. Although the recognition rate was adequate, the classification of the emotions was too coarse-grained since they classified them into the three broad states of positive, negative and neutral emotion. Furthermore, the experiment took place in a limited experimental setting and not inthe-wild for collecting real life data. A similar approach for using

90


finger stroke behaviour as an input modality for inferring user’s positive, negative or neutral affect is presented in [137]. Not long ago, Mottelson et al. [138] studied the implications of human affect on general purpose touch-based mobile interaction and showed that it is possible to detect mobile users’ positive and neutral affective states (negative was not elicited due to ethical concerns) using off-the-shelf machine learning techniques. The features used for the classification were based on smartphone acquired data in regard to the touching behaviour (speed, rotate, tap, precision etc.) of the user with his/her device after performing specific tasks. The authors conducted two empirical studies and their results reflected findings in experimental psychology by confirming that affect has direct behavioural links with smartphones interactions. 3.3. Recognition of dimensional affective states Several research studies for automatic recognition of people’s affective states through the usage of smartphone data have adopted a dimensional approach for modelling and measuring the moods and the emotions of participants. For instance, MoodSense [139] gathered information already available in smartphones (SMS, email, phone call, application usage, web browsing and location) and demonstrated that user mood can be inferred and classified into four major types with quite satisfactory accuracy. Of course, the accuracy was improved dramatically when the inference was based on the same participant’s training data. In order to quantify and represent mood, authors adopted the Circumplex Model of Affect. MoodScope [140], MoodSense’s successor, is a smartphone software system that goes significantly further in its methodologies, realisation, and evaluation than its predecessor and, although it does not solve the privacy problems arisen from the use of sensitive data, it takes privacy concerns into account by adopting a variety of data anonymisation techniques when capturing user to smartphone interactions. As MoodSense, MoodScope is a smartphone software system that statistically infers user’s daily mood with satisfactory accuracy, by analysing communication history and application usage patterns. In [141] a framework called MoodMiner for assessing and analysing mood in daily life is proposed. The assessment utilises mobile phone data (including acceleration, light, ambient sound, location and call log) in order to extract human behaviour patterns and to estimate daily mood. Although it is not explicitly stated in the paper, the model employed was similar to the Activation Deactivation Adjective Check (AD ACL) dimensional model for assessing user’s daily mood and the inference results of the classification algorithm were quite modest. Likewise, based on an affective model similar to Thayer’s AD ACL, HappyHour application [142], inspired by EmotionSense [114], employs a machine learning algorithm to infer user’s current emotion and uses this information to trigger suggestive feedback. HappyHour demo collects data from smartphone sensors (accelerometer and microphone), a smart shirt’s electrocardiogram and weather information and process them to infer emotional negative or positive states. When negative emotions are detected, the application, by exploiting user location through GPS sensor, timely suggests walking exercises while providing realtime information regarding nearby points of interest. In a recent study [143], researchers investigated the relationship between daily diversity of social communication and affect states people experience in their daily lives by exploiting linear regression statistical techniques. Two kinds of sensing technologies were used in order to capture information on subject’s social interaction: sociometric badges for face-to-face interaction and smartphones for mobile phone calls. In addition, daily experience sampling surveys were also conducted to collect affect states and

their corresponding traits while the short version of the Positive and Negative Affect Schedule (PANAS) model was used to evaluate the affect states of participants. Results were demonstrated in terms of variations within and between states and traits and they showed that communication diversity correlates with desirable affect states – e.g., an increase in the positive affect state or a decrease in the negative affect state – for some personality types, but correlates with undesirable affect states for others. 3.4. Recognition of wellbeing and human behaviour models Nowadays, several studies about gathering data from smartphones, in order to either depict human behaviour by implementing their own arbitrary model or to model wellbeing by adopting relevant wellbeing theories, are emerging. As a result, an increased amount of applications concerning the deduction and modelling of negative mental states, such as depression, based on smartphone behavioural and sensor data, have appeared. Whilst the inference of depression and other negative mental conditions is strongly related to the affective states of smartphone holders, we are not going to proceed to an extended reference of such works here as we feel that these specialised mental and health conditions should be studied in another dedicated survey. Still, for the sake of completeness, we do mention below some studies in regard to mobile affective computing and depression. As far as wellbeing is concerned, in [144] the overall multidimensional wellbeing is being automatically induced by monitoring persons’ physical activities, social interaction and sleep patterns. To do so, a prototype application called BeWell was developed to infer participants’ sleep, activity and social interactions by sampling three smartphone sensors, GPS, accelerometer and microphone, while at the same time is also monitoring phone recharging events and periods when phones are either stationary or in a near silent sound environment. Then, the application estimates multi-dimensional wellbeing scores that capture the relationship between behavioural patterns and health outcomes, by using existing guidelines provided by healthcare professionals. In the same context of inferring wellbeing, the authors of [133] went one step further by demonstrating a new developed technique (MTMKL) [145] in an attempt to answer the problem of using multimodal data for modelling complex internal states like wellbeing. For the data collection process, data acquired from participants’ smartphones (location, calls and SMS logs, communication over phone and SMS as well as each time the phone screen was turned on or off) were examined and, subsequently, features related with the number of unique contacts with whom the participants interact and other features related to the timing, duration, and frequency of phone activities, were selected. Additionally, data from a wearable biosensor which measured skin conductance, skin temperature and acceleration, were collected. Participants also submitted daily self-reports on their activities, sleeping, social interactions and other aspects of their lives. The study intelligently combined data from multiple modalities and provided significant performance improvements over both traditional classifiers and multiple kernel learning algorithms, whereas it was able to classify each of the wellbeing dimensions within a single model. Likewise, researchers in [146] tried to discover human behaviour based on people’s smartphone life log data and to build a behaviour model which can be used for human identification. Personal data collected consist of location information, battery status, nearby Bluetooth sensors, historical data, such as contacts, call logs and SMS logs, and continuous sensed data, such as data coming from accelerometer, gyroscope and magnetic field smartphone embedded sensors. In contrast to the previous studies referenced in the current survey, this study gathered combined data by the majority of the available sensors and sources, as it can be seen


in Table 1, and therefore constitutes by far the most multimodal study we have discovered. However, not all modalities contributed evenly in the outcome of the model, since when one or more features/sensors were removed, the observed accuracy remained nearly unchanged. Moturu et al. in [147] explored social interactions and their effects on wellbeing and life satisfaction by focusing on mood, sleep and sociability associations derived from face-to-face social interactions. They analysed smartphone generated social communication data based on Bluetooth proximity detection and self-reported mood and sleep data, in order to find relationships between mood, sleep and sociability. In particular, they detected, among many other interesting observations, that individuals with lower overall sociability show poor moods more often, a statistically significant result derived from quantified social interaction data. In the context of depression, social interaction was also employed in [148] for the detection of behaviour change in people with depression. To this end, an Android application that collects data from mobile sensors, including noise amplitude (from microphone), location, WiFi, light intensity (from ambient light sensor) and movement (from accelerometer), was developed. Furthermore, the application also captured several device states, such as screen on/off, applications currently running and battery-charging state. The study, even if it did not fully model depression given its small sample size, it successfully identified specific user behaviours related to depression. Lastly, one of the first attempts to detect major depressive disorders and unipolar depression is listed in [149] where a contextsensing system is built, on the one hand, for predicting patients’ mood, emotions, cognitive/motivational states, activities, environmental and social context based on smartphone collected data, and on the other, for guiding appropriate interventions when a depressive state is detected. More precisely, data used for inference were those collected directly by the GPS, WiFi, Bluetooth, accelerometer and ambient light phone sensors, as well as contextual data acquired from information available on the phone, such as time/day and activities of the phone’s operating system (e.g., recent calls, active phone applications). Moreover, participants were periodically prompted to self-report their states using momentary assessment on the phone and the sensor data acquired at these times were paired with simultaneously labelled state data to identify specific user states (e.g., mood, concentration) from sensor values. For every state, a machine learning algorithm generated a participantspecific model for predicting that state from sensed data in the future. Nevertheless, as far as mood inference was concerned, predictive capability was fairly poor and hence not reported. 4. Challenges for smartphone affective research Since the early days of affective computing in the late 1990s, many theorists confronted each other, and still do, over its principles. In fact, the pioneer of affective computing Rosalind Picard had to face many difficulties and to overcome great obstacles until the notion of affective computing would be endorsed and become popular within the wider academic circles, given the clear lack of interest for emotions in the computer scientific community at that time [150] as well as the absence of appropriate means for detecting them adequately. Notwithstanding the efforts made by Picard and her peers in affective computing, there are still numerous open research questions and considerations, both ethical and technical, and multiple paths to be explored before affective technology is applicable for everyday use. And although technology constantly evolves, providing evidence that technical challenges will be confuted at some point, the discussion for the ethical challenges faced are still in their infancy. Ethical questions are constantly raised when

91

technology precedes its time and researchers often face dilemmas of following innovative technological approaches while ensuring appropriate levels of ethics. As described by Miller in the smartphone psychology manifesto [6], one of the main disadvantages of smartphones are the ethical challenges in obtaining truly informed consent, protecting participant privacy and anonymity and reducing liability risks. In view of the vigorous and ongoing, for over a decade, discussions on technical and ethical considerations deriving from affecting computing [150,151], some of its key aspects and challenges will be presented and discussed below. Nonetheless, it should be noted that the list should by no means be interpreted as exhaustive. 4.1. Privacy Privacy is considered a major challenge not only in mobile affective computing but in the broader area of smartphone sensing. Respecting and preserving user’s privacy is perhaps the most essential task for all mobile data mining systems due to the fact that people are usually sensitive when their personal data are captured and used, especially if the data reveal users’ location, communications or other sensitive information. This concern grows dramatically when health or affective data are collected. In a pilot study led in 2016 in Australia regarding preliminary insight into individuals’ perceptions towards sharing their personal health data with researchers [152], anonymity of personal health data was regarded as very or extremely important by nearly 90% of participants and was the primary constraint on the sharing of health data. Moreover, in order for the participants to donate their personal data to a public scientific database, the majority of them required privacy assurance. A commonly adopted practice in research studies, whether they are about affective or other personal data, is that ideas and hypotheses have to be tested by a pilot application loaded in participants’ smartphones. Yet, releasing a smartphone application to the public requires compliance with ethical codes as well as privacy and security constraints that protect users. On the contrary, the majority of the referenced papers in Section 3 of this survey are not considering privacy and security constraints at all in their design, and even those that do take some steps to meet these requirements, the undertaken measures are not as satisfactory as it would have been expected, given the rapid progress of countermeasures found throughout the relevant literature. Undeniably, data derived from mobile sensors can reveal information that most users would not have exposed willingly if they had been asked, like call and SMS conversations, photographs and location data, whereas the continuous and passively collection of personal information from smartphone devices could be considered by many privacy rights advocates as a clear personal intrusion and a major threat to human rights. But not all kinds of mobile data contribute equally to the potential threats. In Fig. 1 all the smartphone data sources, utilised by the referenced studies in the current survey for inferring affective states, are taxonomised based on the risk they entail in disclosing sensitive personal information and are classified according to their privacy invasiveness into low, medium or high risks. Having said that, it should be stated that not all referenced studies handle smartphone data sources with the same degree of invasiveness. For instance, some studies use call logs to calculate aggregated information, such as the number of calls and their duration, while others access more sensitive information, like the contacts list, to calculate the diversity of social communication by counting the time allocated to each contact. Beyond the aforementioned privacy concerns, issues of privacy are raised not only regarding participants in a study but regarding others in their vicinity due to some sensors’ high sensitivity, like camera and microphone [104]. In many cases, people not

92


Fig. 1. Taxonomy of sensors and utilities according to privacy invasiveness.

broad area of research and for specific purposes) whereas, in some few cases, GDPR will permit researchers not to obtain consent at all. Nevertheless, when research studies concern sensitive information such as emotions, moods or personality characteristics, there should be a flexible balance between specific consent, which strangulates research, not consent, which trespasses individual’s privacy or another type of consent. That being said, it should be noted that informed consent is something that researchers intentionally skip during their experiments in order to improve their quality and to avoid a bias of people attitudes and interaction styles. People are expected to act differently when they know that they are being monitored or surveilled [165,166] and they do not reveal their actual emotions as they might feel that they could be judged about their reactions. 4.3. Data misuse

participating in a study, and hence not carrying on an emotion sensing application in their smartphones, may be monitored by just sitting next to a participant or calling/messaging him/her, and thus their personal data are becoming part of the experiment. Another interesting aspect of privacy in smartphone sensing applications is related to those data that do not only describe the current affective state of the individuals, but his/her predicted future states [153,154]. This raises a common question, pointed out by many researchers, regarding the ownership of the information extracted from the personal data of an individual. Although there are existing approaches which can help with the problem of privacy (e.g., cryptography, privacy-preserving data mining, anonymisation), they are often insufficient both in terms of technological and in operational issues [155]. For instance, as far as anonymisation is concerned, it has been shown in many works that there is no such thing like a foolproof anonymisation [156,157] since almost all information can be defined as ‘‘personal’’ when combined with enough other relevant data [158]. Nonetheless, collecting such fine-grained personal data as those captured almost continuously by smartphone sensors, can potentially lead to further privacy exposure since privacy in mobile devices is rather complicated [159] and many things that might seem irrelevant now, when correlated with other data and context, may reveal a lot of sensitive information. All in all, it is obvious that users have to make decisions concerning the disclosure of their personal information on the basis of a difficult trade-off between data protection and the advantages stemming from data sharing [160]. To the extent that the balance between the benefits of using personal data and the individual privacy rights are under continuous debate by scientists from law, engineering and psychology disciplines [161–163], the problem of data privacy in the era of ‘‘big data’’ will sustain. 4.2. Informed consent Informed consent is a concept highly interrelated to privacy. In medicine, informed consent is a process for obtaining permission before conducting a healthcare intervention on a person. Yet, informed consent is also adopted in the field of computer science when applications require the collection and processing of data from live subjects. In applications developed for research purposes, subjects are not always aware of the goals of the study and may not fully comprehend the actual data collected. Hence, informed consent should be required before individuals confirm their participation and, on top of that, researchers needs to limit the reuse of the acquired data to preserve informed consent [101]. According to the General Data Protection Regulation (GPDR) [164], which will take effect in European Union on 2018, the notion of a broad consent is to be allowed in research studies (consent is given to a

Another major ethical consideration of affect recognition technology, when it will become widely used, is its potential to be easily used by organisations or individuals to infer, manipulate, exploit or influence people’s emotions. As a result, this poses the ethical question of who owns the power in influencing people’s opinions and emotions [167]. The undesirable effects of such dangerous situations have been already fictionally depicted in Orwell’s ‘‘1984’’ and many are afraid that the book may be proved more predictive that author’s original intentions. Acknowledging the above ethical concerns, the European Data Protection Supervisor (EDPS) with the support of an Ethics Advisory Group initiated a work on the new digital ethics in EU and globally [168] in order to reconsider the ethical dimension of the relationships between human rights, technology, markets and business models and their implications for the rights to privacy and data protection in the digital environment. Additionally, the European Group on Ethics in Science and New Technologies (EGE), an independent, pluralist and multidisciplinary body which advises the European Commission on ethical aspects of science and new technologies, is working towards the preparation and implementation of Community legislation or policies for ethical questions arising from science and new technologies [169]. Yet, these are just some small steps in the right direction. 4.4. Trust and engagement The challenge of gaining people’s trust and engagement in participating in a smartphone emotion detection study is not trivial and most of the time is tightly coupled with privacy issues. This is because opportunistic smartphone sensing techniques [155] where mobile devices are used automatically without active and conscious human intervention, could be perceived as invasive by many participants as it has been already discussed in Section 4.1. Worst still, trust is usually at stake when, as already discussed in Informed Consent Section 4.2, participants never learn the purpose of the study in which they were participated in; instead, they are either misled into believing that they are participating into a study with false purpose, or they have never been informed for the secondary use of their data when the data collection process has been performed in an implicit manner (which means that they have been initially collected for another primary purpose, but now they are used for a secondary one) [170]. In these situations, a level of mistrust and doubt may arise and the necessary human participation might not be achieved in the long term. The situation of mistrust is deteriorated further if we take into account that people are typically not involved in the life-cycle of their own personal data, which results in a lack of understanding of who uses their data and for what reason [160]. As Picard states in [171], emotion research studies rarely benefit their participants. Of

E. Politou et al. / Computer Science Review 25 (2017) 79–100 Table 2 Used representation models.

course, they may pay a modest sum in cash or credits but considering that many participants have real emotion-related needs, yet few of them usually learn anything from participating. Due to security and privacy constraints, in many cases researchers hide all the individual participants’ information. But Picard wonders if there was a way to protect participants’ privacy and at the same time let them learn about themselves, without increasing scientific workload beyond the current one. Otherwise, the lack of incentives for users to participate in mobile affective sensing can lead to a lack of participation and, therefore, data [172]. As proposed by Rana et al. in [173], a possible strategy to gain trust is transparency. This could be achieved by visually demonstrating the end-to-end application – from data collection to data processing and to data dissemination – verifying that privacy is preserved and access control is provided in every step of the experiment. Additionally, some participants would only participate when they can identify and understand the validity and relevance of the affective application [174]. Hence, outlining and demonstrating how the proposed study or application could transform or assist participants in their everyday life or in treating negative emotional states, contributes significantly to the added value of the affective computing research.

93

Yet, the problem of data fusion for multimodal affect recognition continues to puzzle researchers and, for the time being, forms another barrier that limits the full integration of all possible modalities in a homogeneous system [73]. Nevertheless, it should be highlighted that, apart from a small number of studies like [145], all efforts for solving the problem of multimodal fusion do concentrate on the traditional quite ‘‘intrusive’’ modalities of video and audio [13,16,17,180–182], ignoring, as Gunes and Hung describe in [183], the ‘‘new kids in the block’’ such as bio signals and ‘‘the myriad ways of mobile sensing technology and smartphone sensor revolution provide (location sensing, acceleration sensing etc.) readily in our pockets’’. Hence, the multimodal fusion of smartphone captured data is a research path open for exploration. 4.6. Resource constraints On top of the aforementioned challenges, any algorithms preserving privacy or other security solutions as well as utilising effective multimodal techniques and existing complex technical solutions should be implemented in an energy and power saving approach by keeping in mind that mobile phones have limited operation time and processing power. Additionally, mobile affective applications should be implemented with a higher portability to address interoperability issues, since mobile devices may not only be equipped with different mobile platforms and operation environments [184], but with different sensor capabilities as well. Consequently, given the diversity of mobile devices in availability and capabilities, modelling and predicting the energy and processing requirements to accomplish a particular task remains a complex issue [175]. 4.7. Affect modelling and representation

4.5. Multimodal fusion Affective information, like emotions and personality traits, can be inferred from various communication channels such as facial and body expressions, speech, text and embedded smartphone sensors or biosensors. However, a challenging issue that impacts the effectiveness of affective technology is the fusion of these modalities for building competent multimodal affective recognition systems. A multimodal affective recognition system is a system that, via multiple inputs, retrieves affective information from various types of sources and associates input data with a finite set of affective states. The problems in regard to the implementation of an effective system with multimodal fusion are plenty and they mainly concern the great variations of data in terms of structure and content, the varied velocity of data reception, the different sampling rate and quality of received data and the continuous growing of data size [175]. While there are several techniques of fusion, such as featurelevel fusion, decision-level fusion, and data-level fusion [176], choosing an optimal fusion type is challenging. Computer scientists have shown that, by applying modern machine learning techniques, the combination of different classifiers has been more effective than single classifiers, as long as they are sufficiently diverse [177]. On that ground, research about multimodal fusion for the effective combination of acquired data, has been blooming over the past years [24]. Indeed, several studies have commenced regarding the multimodal fusion of distinct modalities for affect recognition [17,178] and many system architectures have been proposed in order to provide viable multimodal real time emotion recognition [176,179]. Recently, some cloud based big data approaches have been also proposed in an attempt to provide viable solution for this challenging issue [2].

Hitherto, immense research has been undertaken by scholars in forming sound theoretical foundations and systematic guidelines for emotion modelling [185]. Despite the continuous effort of affective computing scientists for transitioning affective computing into a more scientific discipline by validating existing emotion models and developing systematic guidelines [186] and standards [187] for affect modelling, the theoretical foundations of affective computing are yet under continuous development [188]. Considering the currently available models for representing the affect information, and examining those referenced in Section 3 and in Table 1, Table 2 is constructed to summarise the models used in this survey. By taking a closer look in Table 2, one can observe the following: 1. The number of studies employing a multi-dimensional affective model representation for recognising affect underperforms significantly the studies employing distinct-state approaches or binary and triadic classification models to represent their affective space. This is due to, on the one hand, the effectiveness of categorising input samples into a number of affective classes and, on the other, the difficulty of applying dimensional models in affective recognition systems. As described in a survey by Gunes et al. [73], there is a number of questions that need to be examined before dimensional affect models become suitable in automatic affect recognition systems. 2. From Table 2, it is apparent that the absence of appraisalbased models in automatic emotion recognition is an open issue for research due to the fact that appraisal-based approach requires complex, multicomponential and sophisticated measurements of change [73,189]. Despite this complexity, some advocate that the use of appraisal models in affective computing will provide a number of benefits for

94


automatic recognition [190] and, recently, research on this direction has been commenced [191]. 3. While many studies are employing the six ‘‘basic’’ emotions proposed by Ekman (anger, disgust, fear, joy, sadness and surprise), there is a plethora of affect computing studies that extend the set of Ekman’s emotions by including more distinct states, such as bored, contented, excited, nervous, relaxed or upset, and even further emotions more applicable to affect-sensitive learning environments, such as confusion, frustration, flow, curiosity and anxiety [192]. Hence, it is evident that broadening the affective space of categorical approaches for affective computing should be further examined. 4. Lastly, it is apparent that the most commonly used representation for affective states in computing is the binary or triadic representation that uses two or three–class states model (like happy, not happy and neutral) due to its modelling convenience in machine learning systems. However, psychology researchers argue that this simplistic representation is not suitable for real world affective recognition environments. Therefore, approaches using rough and fuzzy sets [193,194] could potentially fill this gap as such methods can be adopted by machine learning algorithms [195] and in fact have been already used by some researchers in the field [196–199]. Interestingly enough, the approach of soft sets [200] has not yet received the expected focus from the research community, despite its potential to further improve the affective modelling. There are also some other minor issues on affective modelling that have not yet been adequately addressed by the affective computing community, like the appropriate evaluation of affective systems and the influence of context in affect detection [192]. 4.8. Cultural differences In affect recognition, a potential ethical limitation in research studies comes from the fact that perceptions of emotions and personality are not universal but they are highly dependent on the cultural and conceptual framework [58,59,201]. Additionally, previous studies have shown that some cultures regard emotions as feelings of individuals whereas other cultures regard them inseparable from the feelings of a group [57]. According to these perspectives, a reasonable challenge is how an affective recognition system should be designed and built in a cultural transparent way. 4.9. Cost Last but not least, a – for the time of being – barrier preventing the wide spread of affective applications is the fact that smartphones, together with their accompanied mobile telecommunication services (e.g., 3G, 4G), are still relatively expensive, something that undermines the first requirement for ubiquitous computing as it has been specified by its pioneer Weiser, that is to be cheap [100], and potentially may affect research results due to insufficient participation. Therefore, unless devices and their telecommunication services are provided to the participants at the cost of the research projects, this barrier will always cause problems of sample selectivity and will restrict the extended application of the study in-the-wild. 5. Discussion and conclusions In this survey we attempted to document all research works and experiments in mobile affective computing, and particularly in smartphone affect recognition, which have been undertaken

Table 3 Affect recognition objective. Objective Stress Emotion Wellbeing and user behaviour Personality Mood

Percentage 35.7% 33.3% 14.3% 11.9% 4.8%

over the past few years. The majority of the studies were multimodal, since more than one data inputs were used to infer the affect information of participants, albeit some data did not originate from smartphone modalities but from additional inputs such as wearable chests or web information (described in greyed out boxes in Table 1). The frequency distribution of each referenced smartphone or external modality is illustrated in Fig. 2 where it can be seen that the most commonly employed data source for inferring affect across all modalities were users’ phone calls. Further, user questionnaires or data annotation techniques were adopted in the majority of the studies (78.6%) in order to obtain ground truth data for evaluating the detection results. To the extent that smartphone usage data are concerned, as illustrated in Fig. 3, the most commonly used mobile source was by far the call logs from user’s smartphones (26.4%), followed by the SMS logs (22%) and the data collected from the application logs used in the smartphones (14.3%). In terms of embedded sensor data, the most widely used sensor in the referenced studies was the GPS sensor (27.6%), followed by the accelerometer sensor (22.4%) and the Bluetooth (14.5%). The inference of emotion, mood, personality trait, wellbeing, behaviour or the much broad mental state of depression, were the recognition goal of the included studies and at the same time, the main criterion for listing them in this survey. Table 3 illustrates the degree to which each affect objective occupied the referenced literature. While a large proportion of the studies, almost 35.7%, were about inferring user’s stress levels, as it was expected given the place of stress as a dominant factor in our modern society affecting our expression and behaviour, the recognition of emotion and mood states occupied academics to the same extend (38.1%). Inferring personality of participants’ was the subject of about 12% of the referenced papers, whereas only a small proportion targeted at wellbeing and human behaviour inference. While the affect representation model used varied significantly among the referenced studies, it can be observed that most researchers were in favour of distinct states model, i.e. Ekman’s six basic states model with some deviations, and the binary or triadic representation of a single state, i.e. happy vs. unhappy or high vs. medium vs. high stress. Still, some dimensional models were also employed but they consist a minority. As far as the evaluation of the outcome is concerned, most of the studies measured the recognition rate in terms of accuracy, a measurement that showed a satisfactory degree of successful recognition results. However, the number of subjects involved in each study and the duration in which the data collection process took place vary dramatically from 1 subject and few days to 119 subjects and 17 months. On top of that, while most of the experiments took place in-the-wild, that is a non-controlled environment with real life data, some experiments have been carried out in the limited setting of a controlled laboratory. Hence, accuracy cannot be considered as a common evaluator factor across all included studies. It should be mentioned though, that few studies did not provide recognition results at all, while other studies evaluated their models in terms of MSE or F-measure, or in terms of a single accuracy for every affective state. Thus, the need for a common standardised approach for evaluating the effectiveness of affect


95

Fig. 2. Distribution of smartphone and external modalities.

(a) Sensor data sources.

(b) Usage data sources.

Fig. 3. Frequency distribution for sensor and usage of smartphone data.

recognition studies in-the-wild has arisen and a thorough discussion on this issue should be initiated before any methods or solutions get adopted by the scientific community. Disregarding the above analysis of modalities, affect models and evaluation measures across all referenced studies, the initial intention of this survey was not so to exclusively list all the pertinent research on mobile affect recognition domain but mainly, on the one hand, to provide compelling evidence that the era of smartphone affective computing has begun and, on the other, to demonstrate the current trends on smartphone affect recognition. A secondary goal was to present the results of mobile affect recognition which, although not as much standardised as they would have been in a much more mature scientific domain, they are still promising and fruitful and hence should not be neglected neither from computer scientists nor from psychology theorists. Concerning the latter, it should be emphasised that the inferred affective information can be proved extremely useful for a wide range of mental health applications, either by triggering interventions when they detect declining affective states like those described in [133,149] for detecting depression, or by providing appropriate recommendations depending on users’ mood like those described in [142] where

users’ emotions are inferred and used to trigger suggestive feedback. Therefore, as the authors of a paper on research priorities for artificial intelligence highlight [202], given the great potential of artificial intelligence (and thus affective computing as well) for providing beneficial and robust systems to society, it is valuable to investigate how to reap its benefits while avoiding potential pitfalls. Evidently, as affective computing is stepping out of the laboratory to the real world, people’s perception of it will fundamentally be both challenged and change, whereas understanding the links between people’s emotions and behaviour in different contexts will guide the next generation of real-world applications to accommodate cognitive and emotional requirements [172]. In this process, smartphones would play a central role and according to Miller’s smartphone psychology manifesto [6]: ‘‘the question is not whether smartphones will revolutionise psychology but how, when, and where the revolution will happen’’. Nevertheless, although artificial intelligence and HCI are rapidly evolving and many advocate that emotional-aware smartphones will be a commonplace in the days to come, mobile affect recognition systems face a number of substantial challenges before they become a prevailing choice for research and industry. Most of the challenging issues are of moral nature and as such there is an urge

96


for scientists involved to understand the ethical impact of affective computing to our everyday lives. According to Beavers [203], the moral dimensions of affective computing come primarily from the fact that it can intensify dispositions already at play in human life whereas progress in affective computing could jeopardise our wellbeing due to the potentials for abuse, some of which include manipulation, misinterpretation, confusion, etc. Therefore, setting a privacy baseline and providing a common framework for performing all the associated research is of utter importance. Undeniably, the discussion of ethical issues in artificial intelligence, in which affective computing is part of, is so indispensable that, at the time of writing, the IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems has already published the first version of a framework document called ‘‘Ethically Aligned Design’’ [204] in an attempt: (a) to provide guidelines to scientists and industry for building beneficent and beneficial autonomous systems by considering and taking into account all the ethics implications and (b) to advance a public discussion of how these intelligent and autonomous technologies can be aligned to moral values and ethical principles that prioritise human wellbeing. According to IEEE Standard Association’s managing director, Konstantinos Karachalios, ‘‘by providing technologists with peer-driven, practical recommendations for creating ethically aligned autonomous and intelligent products, services, and systems, we can move beyond the fears associated with these technologies and bring valued benefits to humanity today and for the future’’ [205]. In conclusion, and notwithstanding the discussed ethical perplexity, affective computing, just as technology in general, is morally neutral. As Beavers states in [203], the morally-relevant factor is what it is used for and how it is used. Comparable to Mendel’s genetics, which in the first half of the 20th century were transformed to a lethal weapon onto Mengele’s hands, and similarly to Einstein’s nuclear physics, which became weapon of mass distraction onto politicians hands, affective computing’s morality will be effectively judged by its use. Meanwhile, up until artificial intelligence becomes an integral part to everyday household devices, research in affective computing will continue to fascinate and intrigue people’s minds. References [1] A. Campbell, T. Choudhury, From smart to cognitive phones, IEEE Pervasive Comput. 3 (11) (2012) 7–11. [2] Y. Baimbetov, I. Khalil, M. Steinbauer, G. Anderst-Kotsis, Using big data for emotionally intelligent mobile services through multi-modal emotion recognition, in: International Conference on Smart Homes and Health Telematics, Springer, 2015, pp. 127–138. [3] M. Rouse, affective computing. URL http://whatis.techtarget.com/definition/ affective-computing. [4] B. Priyantha, D. Lymberopoulos, J. Liu, Littlerock: Enabling energy-efficient continuous sensing on mobile phones, IEEE Pervasive Comput. 10 (2) (2011) 12–15. [5] A. Mehrotra, V. Pejovic, M. Musolesi, SenSocial: a middleware for integrating online social networks and mobile sensing data streams, in: Proceedings of the 15th International Middleware Conference, ACM, 2014, pp. 205–216. [6] G. Miller, The smartphone psychology manifesto, Perspect. Psychol. Sci. 7 (3) (2012) 221–237. [7] M. Pantic, A. Pentland, A. Nijholt, T.S. Huang, Human computing and machine understanding of human behavior: a survey, in: Artifical Intelligence for Human Computing, Springer, 2007, pp. 47–71. [8] Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition methods: Audio, visual, and spontaneous expressions, IEEE Trans. Pattern Anal. Mach. Intell. 31 (1) (2009) 39–58. [9] E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: A survey of registration, representation, and recognition, IEEE Trans. Pattern Anal. Mach. Intell. 37 (6) (2015) 1113–1133. [10] M. Pantic, L.J.M. Rothkrantz, Automatic analysis of facial expressions: The state of the art, IEEE Trans. Pattern Anal. Mach. Intell. 22 (12) (2000) 1424– 1445. [11] M.A. Nicolaou, H. Gunes, M. Pantic, Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space, IEEE Trans. Affective Comput. 2 (2) (2011) 92–105.

[12] H. Gunes, B. Schuller, M. Pantic, R. Cowie, Emotion representation, analysis and synthesis in continuous space: A survey, in: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), IEEE, 2011, pp. 827–834. [13] A. Jaimes, N. Sebe, Multimodal human–computer interaction: A survey, Comput. Vis. Image Underst. 108 (1) (2007) 116–134. [14] C.-H. Wu, J.-C. Lin, W.-L. Wei, Survey on audiovisual emotion recognition: databases, features, and data fusion strategies, APSIPA Trans. Signal Inf. Process. 3 (2014) e12. [15] M. El Ayadi, M.S. Kamel, F. Karray, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognit. 44 (3) (2011) 572–587. [16] M. Pantic, L.J. Rothkrantz, Toward an affect-sensitive multimodal humancomputer interaction, Proc. IEEE 91 (9) (2003) 1370–1390. [17] N. Sebe, I. Cohen, T. Gevers, T.S. Huang, Multimodal approaches for emotion recognition: a survey, in: Electronic Imaging 2005, International Society for Optics and Photonics, 2005, pp. 56–67. [18] E. C.-C. Kao, C.-C. Liu, T.-H. Yang, C.-T. Hsieh, V.-W. Soo, Towards text-based emotion detection a survey and possible improvements, in: International Conference on Information Management and Engineering, 2009., ICIME’09, IEEE, 2009, pp. 70–74. [19] B. Liu, Sentiment analysis and opinion mining, Syn. Lect. Hum. Lang. Technol. 5 (1) (2012) 1–167. [20] A. Kleinsmith, N. Bianchi-Berthouze, Affective body expression perception and recognition: A survey, IEEE Trans. Affective Comput. 4 (1) (2013) 15–33. [21] A. Kołakowska, A review of emotion recognition methods based on keystroke dynamics and mouse movements, in: 2013 6th International Conference on Human System Interactions (HSI), IEEE, 2013, pp. 548–555. [22] Y. Gao, N. Bianchi-Berthouze, H. Meng, What does touch tell us about emotions in touchscreen-based gameplay? ACM Trans. Comput.-Hum. Interact. (TOCHI) 19 (4) (2012) 31. [23] M.J. Hertenstein, R. Holmes, M. McCullough, D. Keltner, The communication of emotion via touch., Emotion 9 (4) (2009) 566. [24] S. Poria, E. Cambria, R. Bajpai, A. Hussain, A review of affective computing: From unimodal analysis to multimodal fusion, Inf. Fusion 37 (2017) 98–125. [25] E. Shmueli, V.K. Singh, B. Lepri, A. Pentland, Sensing, understanding, and shaping social behavior, IEEE Trans. Comput. Soc. Systems 1 (1) (2014) 22–34. [26] A. Vinciarelli, G. Mohammadi, A survey of personality computing, IEEE Trans. Affective Comput. 5 (3) (2014) 273–291. [27] A. Muaremi, B. Arnrich, G. Tröster, A Survey on Measuring Happiness with Smart Phones, in: International Workshop on Ubiquitous Health and Wellness (UbiHealth), 2012. [28] E. Ceja, V. Osmani, O. Mayora, Automatic stress detection in working environments from smartphones’ accelerometer data: a first step, IEEE J. Biomed. Health Inform. (2015). [29] A. Maxhuni, P. Hernandez-Leal, E.F. Morales, L.E. Sucar, V. Osmani, A. MuńozMeléndez, O. Mayora, Using intermediate models and knowledge learning to improve stress prediction, in: Applications for Future Internet, Springer, 2017, pp. 140–151. [30] R.W. Picard, R. Picard, Affective Computing, Vol. 252, MIT press Cambridge, 1997. [31] R.W. Picard, Toward computers that recognize and respond to user emotion, IBM Syst. J. 39 (3.4) (2000) 705–719. [32] J. Tao, T. Tan, Affective computing: A review, in: International Conference on Affective Computing and Intelligent Interaction, Springer, 2005, pp. 981–995. [33] P. Desmet, Designing Emotions, Delft University of Technology. Department of Industrial Design, 2002. [34] P. Ekkekakis, Affect, mood, and emotion, in: Gershon Tenenbaum, Robert C. Eklund, Akihito Kamata (Eds.), Measurement in Sport and Exercise Psychology, Human Kinetics, Campaign (IL), 2012, pp. 321–332. [35] J.A. Russell, L.F. Barrett, Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant., J. Pers. Soc. Psychol. 76 (5) (1999) 805. [36] J.A. Russell, Core affect and the psychological construction of emotion., Psychol. Rev. 110 (1) (2003) 145. [37] C. Beedie, P. Terry, A. Lane, Distinctions between emotion and mood, Cogn. Emotion 19 (6) (2005) 847–878. [38] P. Ekkekakis, The measurement of affect, mood, and emotion: A guide for health-behavioral research, Cambridge University Press, 2013. [39] P. Ekman, Emotions revealed: Recognizing faces and feelings to improve communication and emotional life, Macmillan, 2007. [40] D. Hume, Emotions and moods, Organ. Behav. (2012) 258–297. [41] R.E. Thayer, The Biopsychology of Mood and Arousal, Oxford University Press, 1990. [42] R. Reisenzein, H. Weber, Personality and emotion, in: The Cambridge Handbook of Personality Psychology, 2009, pp. 54–71. [43] W. Revelle, Experimental approaches to the study of personality, in: Handbook of Research Methods in Personality Psychology, 2007, pp. 37–61. [44] W. Revelle, K.R. Scherer, Personality and emotion, in: Oxford Companion To Emotion and the Affective Sciences, 2009, pp. 304–306.

E. Politou et al. / Computer Science Review 25 (2017) 79–100 [45] J.P. Villanueva, Personality Traits: Classifications, Effects and Changes, Nova Science Publishers, 2010. [46] D.P. McAdams, B.D. Olson, Personality development: Continuity and change over the life course, Annu. Rev. Psychol. 61 (2010) 517–542. [47] R.R. McCrae, P.T. Costa, Adding Liebe und Arbeit: The full five-factor model and well-being, Pers. Soc. Psychol. Bull. 17 (2) (1991) 227–232. [48] Cambridge Dictionary. URL http://dictionary.cambridge.org/dictionary/english/ sentiment (Accessed: 07.01.17). [49] E. Cambria, B. Schuller, Y. Xia, C. Havasi, New avenues in opinion mining and sentiment analysis, IEEE Intell. Syst. 28 (2) (2013) 15–21. [50] E.H. Hovy, What are sentiment, affect, and emotion? Applying the methodology of michael zock to sentiment analysis, in: Language Production, Cognition, and the Lexicon, Springer, 2015, pp. 13–24. [51] J. Posner, J.A. Russell, B.S. Peterson, The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology, Dev. Psychol. 17 (03) (2005) 715–734. [52] S.S. Tomkins, Affect imagery consciousness: Volume I: The positive affects, Vol. 1, Springer publishing company, 1962. [53] C. Darwin, 1965. The Expression of the Emotions in Man and Animals, John Marry, London, UK, 1872. [54] P. Ekman, An argument for basic emotions, Cogn. Emotion 6 (3–4) (1992) 169–200. [55] P. Ekman, W.V. Friesen, M. O’Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W.A. LeCompte, T. Pitcairn, P.E. Ricci-Bitti, et al., Universals and cultural differences in the judgments of facial expressions of emotion., J. Pers. Soc. Psychol. 53 (4) (1987) 712. [56] P. Ekman, D. Keltner, Universal facial expressions of emotion, Calif. Mental Health Res. Dig. 8 (4) (1970) 151–158. [57] T. Masuda, P.C. Ellsworth, B. Mesquita, J. Leu, S. Tanida, E. Van de Veerdonk, Placing the face in context: cultural differences in the perception of facial emotion., J. Pers. Soc. Psychol. 94 (3) (2008) 365. [58] M. Gendron, D. Roberson, J.M. van der Vyver, L.F. Barrett, Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture, Emotion 14 (2) (2014) 251. [59] B. Mesquita, N.H. Frijda, Cultural variations in emotions: a review, Psychol. Bull. 112 (2) (1992) 179. [60] P. Ekman, W.V. Friesen, Facial action coding system, 1977. [61] D. McNair, M. Lorr, L. Droppleman, Manual for the Profile of Mood States, Educational and Industrial Testing Service, San Diego, CA, 1971. [62] J.A. Russell, A circumplex model of affect, J. Pers. Soc. Psychol. 39 (6) (1980) 1161–1178. [63] M.M. Bradley, M.K. Greenwald, M.C. Petry, P.J. Lang, Remembering pictures: pleasure and arousal in memory, J. Exp. Psychol. Learn. Mem. Cogn. 18 (2) (1992) 379. [64] D. Watson, L.A. Clark, A. Tellegen, Development and validation of brief measures of positive and negative affect: the PANAS scales, J. Pers. Soc. Psychol. 54 (6) (1988) 1063. [65] J.R. Crawford, J.D. Henry, The Positive and Negative Affect Schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample, Br. J. Clin. Psychol. 43 (3) (2004) 245–265. [66] D. Watson, A. Tellegen, Toward a consensual structure of mood, Psychol. Bull. 98 (2) (1985) 219. [67] D. Watson, L.A. Clark, Measurement and mismeasurement of mood: Recurrent and emergent issues, J. Pers. Assess. 68 (2) (1997) 267–296. [68] P. Ekkekakis, E.E. Hall, S.J. Petruzzello, Evaluation of the circumplex structure of the activation deactivation adjective check list before and after a short walk, Psychol. Sport Exerc. 6 (1) (2005) 83–101. [69] G. Matthews, D.M. Jones, A.G. Chamberlain, Refining the measurement of mood: The UWIST mood adjective checklist, Br. J. Psychol. 81 (1) (1990) 17– 42. [70] P. Wilhelm, D. Schoebi, Assessing mood in daily life, Eur. J. Psychol. Assess. 23 (4) (2007) 258–267. [71] D.C. Rubin, J.M. Talarico, A comparison of dimensional models of emotion: Evidence from emotions, prototypical events, autobiographical memories, and words, Memory 17 (8) (2009) 802–808. [72] M.B. Arnold, Emotion and Personality, Columbia University Press, 1960. [73] H. Gunes, M. Pantic, Automatic, dimensional and continuous emotion recognition, Int. J. Synth. Emotions 1 (1) (2010) 68–99. [74] P.C. Ellsworth, K.R. Scherer, Appraisal processes in emotion, Handb. Affect. Sci 572 (2003) V595. [75] O.P. John, S. Srivastava, The Big Five trait taxonomy: History, measurement, and theoretical perspectives, Handb. Pers.: Theory Res. 2 (1999) (1999) 102– 138. [76] L.R. Goldberg, The structure of phenotypic personality traits., Am. Psychol. 48 (1) (1993) 26. [77] R.R. McCrae, O.P. John, An introduction to the five-factor model and its applications, J. Pers. 60 (2) (1992) 175–215. [78] L.R. Goldberg, Language and individual differences: The search for universals in personality lexicons, Rev. Pers. Social Psychol. 2 (1) (1981) 141–165.

97

[79] L.R. Goldberg, An alternative ‘‘description of personality’’: the big-five factor structure, J. Pers. Soc. Psychol. 59 (6) (1990) 1216. [80] E.C. Tupes, R.E. Christal, Recurrent personality factors based on trait ratings, Tech. rep., DTIC Document, 1961. [81] S.D. Gosling, P.J. Rentfrow, W.B. Swann, A very brief measure of the Big-Five personality domains, J. Res. Pers. 37 (6) (2003) 504–528. [82] P.T. Costa, R.R. MacCrae, Revised NEO personality inventory (NEO PI-R) and NEO five-factor inventory (NEO FFI): Professional manual, Psychological Assessment Resources, 1992. [83] R.R. McCrae, P.T. Costa, A contemplated revision of the NEO Five-Factor Inventory, Pers. Individ. Differ. 36 (3) (2004) 587–596. [84] International Personality Item Pool. URL http://ipip.ori.org/ (Accessed: 15.02.17.). [85] E. Diener, E.M. Suh, R.E. Lucas, H.L. Smith, Subjective well-being: Three decades of progress, Psychol. Bull. 125 (2) (1999) 276–302. [86] W. Pavot, E. Diener, Happiness experienced: The science of subjective wellbeing, in: Oxford Handbook Happiness, 2013, pp. 134–151. [87] C.D. Ryff, Happiness is everything, or is it? Explorations on the meaning of psychological well-being., J. Pers. Soc. Psychol. 57 (6) (1989) 1069. [88] N. Schwarz, G.L. Clore, Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states, J. Pers. Soc. Psychol. 45 (3) (1983) 513. [89] R.M. Ryan, E.L. Deci, On happiness and human potentials: A review of research on hedonic and eudaimonic well-being, Annu. Rev. Psychol. 52 (1) (2001) 141–166. [90] R. Veenhoven, Freedom and happiness: A comparative study in forty-four nations in the early 1990s, Culture Subject Well-Beingpp, 2000, pp. 257–288. [91] B.L. Fredrickson, T. Joiner, Positive emotions trigger upward spirals toward emotional well-being, Psychol. Sci. 13 (2) (2002) 172–175. [92] P. Conceição, R. Bandura, Measuring subjective wellbeing: A summary review of the literature United Nations Development Programme (UNDP) Development Studies, 2008, Working Paper. [93] W. Tov, E. Diener, Subjective Well-being, in: The Encyclopedia of CrossCultural Psychology, Wiley, 2013 URL http://ink.library.smu.edu.sg/soss_ research/1395. [94] R.W. Picard, S. Papert, W. Bender, B. Blumberg, C. Breazeal, D. Cavallo, T. Machover, M. Resnick, D. Roy, C. Strohecker, Affective learning - a manifesto, BT Technol. J. 22 (4) (2004) 253–269. [95] IBM’s plans to ship Simon put on hold for time being. URL http://research.mi crosoft.com/en-us/um/people/bibuxton/buxtoncollection/a/pdf/press%20rel ease%20delay%201994.pdf(Accessed: 15.02.17). [96] N. Lathia, V. Pejovic, K.K. Rachuri, C. Mascolo, M. Musolesi, P.J. Rentfrow, Smartphones for large-scale behavior change interventions, IEEE Pervasive Comput. 12 (3) (2013) 66–73. [97] N.D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury, A.T. Campbell, A survey of mobile phone sensing, IEEE Commun. Mag. 48 (9) (2010) 140–150. [98] E. Miluzzo, Smartphone sensing, Ph.D. thesis, Dartmouth College Hanover, New Hampshire, 2011. [99] E. Macias, A. Suarez, J. Lloret, Mobile sensing systems, Sensors 13 (12) (2013) 17292–17321. [100] M. Weiser, The computer for the 21st century, Sci. Am. 265 (3) (1991) 94–104. [101] M. Raento, A. Oulasvirta, N. Eagle, Smartphones an emerging tool for social scientists, Sociol. Methods Res. 37 (3) (2009) 426–454. [102] D. Zhang, B. Guo, Z. Yu, The emergence of social and community intelligence, Computer 44 (7) (2011) 21–28. [103] G. Chittaranjan, J. Blom, D. Gatica-Perez, Mining large-scale smartphone data for personality studies, Pers. Ubiquitous Comput. 17 (3) (2013) 433–450. [104] S.D. Gosling, W. Mason, Internet research in psychology, Annu. Rev. Psychol. 66 (2015) 877–902. [105] N. Eagle, A.S. Pentland, D. Lazer, Inferring friendship network structure by using mobile phone data, Proc. Natl. Acad. Sci. 106 (36) (2009) 15274–15278. [106] J. Hernandez, D.J. McDuff, R.W. Picard, Biophone: Physiology monitoring from peripheral smartphone motions, in: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2015, pp. 7180–7183. [107] F. Pianesi, N. Mana, A. Cappelletti, B. Lepri, M. Zancanaro, Multimodal recognition of personality traits in social interactions, in: Proceedings of the 10th International Conference on Multimodal Interfaces, ACM, 2008, pp. 53–60. [108] S. Butt, J.G. Phillips, Personality and self reported mobile phone use, Comput. Hum. Behav. 24 (2) (2008) 346–360. [109] G. Chittaranjan, J. Blom, D. Gatica-Perez, Who’s who with big-five: Analyzing and classifying personality traits with smartphones, in: 2011 15th Annual International Symposium on Wearable Computers, IEEE, 2011, pp. 29–36. [110] R. de Oliveira, A. Karatzoglou, P. Concejero Cerezo, A. Armenta Lopez de Vicuña, N. Oliver, Towards a psychographic user model from mobile phone usage, in: CHI’11 Extended Abstracts on Human Factors in Computing Systems, ACM, 2011, pp. 2191–2196. [111] J. Staiano, B. Lepri, N. Aharony, F. Pianesi, N. Sebe, A. Pentland, Friends don’t lie: inferring personality traits from social network structure, in: Proceedings

98

[112]

[113]

[114]

[115]

[116]

[117]

[118]

[119]

[120]

[121]

[122]

[123]

[124]

[125]

[126]

[127]

[128]

[129] [130]

[131]

[132]

E. Politou et al. / Computer Science Review 25 (2017) 79–100 of the 2012 ACM Conference on Ubiquitous Computing, ACM, 2012, pp. 321– 330. Y.-A. de Montjoye, J. Quoidbach, F. Robic, A.S. Pentland, Predicting personality using novel mobile phone-based metrics, in: International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction, Springer, 2013, pp. 48–55. K. Oh, H.-S. Park, S.-B. Cho, A mobile context sharing system using activity and emotion recognition with Bayesian networks, in: Ubiquitous Intelligence & Computing and 7th International Conference on Autonomic & Trusted Computing (UIC/ATC), 2010 7th International Conference on, IEEE, 2010, pp. 244–249. K.K. Rachuri, M. Musolesi, C. Mascolo, P.J. Rentfrow, C. Longworth, A. Aucinas, EmotionSense: a mobile phones based adaptive platform for experimental social psychology research, in: Proceedings of the 12th ACM International Conference on Ubiquitous Computing, ACM, 2010, pp. 281–290. H. Lee, Y.S. Choi, S. Lee, I. Park, Towards unobtrusive emotion recognition for affective social communication, in: 2012 IEEE Consumer Communications and Networking Conference (CCNC), IEEE, 2012, pp. 260–264. H.-J. Kim, Y.S. Choi, Exploring emotional preference for smartphone applications, in: 2012 IEEE Consumer Communications and Networking Conference (CCNC), IEEE, 2012, pp. 245–249. S. Greene, H. Thapliyal, A. Caban-Holt, A survey of affective computing for stress detection: evaluating technologies in stress detection for better health, IEEE Consum. Electron. Mag. 5 (4) (2016) 44–56. A. Bogomolov, B. Lepri, M. Ferron, F. Pianesi, A.S. Pentland, Daily stress recognition from mobile phone data, weather conditions and individual traits, in: Proceedings of the 22nd ACM International Conference on Multimedia, ACM, 2014, pp. 477–486. A. Sano, R.W. Picard, Stress recognition using wearable sensors and mobile phones, in: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on, IEEE, 2013, pp. 671–676. A. Sano, A.J. Phillips, Z.Y. Amy, A.W. McHill, S. Taylor, N. Jaques, C.A. Czeisler, E.B. Klerman, R.W. Picard, Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones, in: 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), IEEE, 2015 pp. 1–6. H. Lu, D. Frauendorfer, M. Rabbi, M.S. Mast, G.T. Chittaranjan, A.T. Campbell, D. Gatica-Perez, T. Choudhury, StressSense: Detecting stress in unconstrained acoustic environments using smartphones, in: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, ACM, 2012, pp. 351–360. G. Bauer, P. Lukowicz, Can smartphones detect stress-related changes in the behaviour of individuals?in: 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), IEEE, 2012, pp. 423–426. A. Muaremi, B. Arnrich, G. Tröster, Towards measuring stress with smartphones and wearable devices during workday and sleep, BioNanoScience 3 (2) (2013) 172–183. J.C. McElroy, A.R. Hendrickson, A.M. Townsend, S.M. DeMarie, Dispositional factors in internet use: personality versus cognitive style, MIS Q. (2007) 809– 820. D. Carneiro, J.C. Castillo, P. Novais, A. Fernández-Caballero, J. Neves, Multimodal behavioral analysis for non-invasive stress detection, Expert Syst. Appl. 39 (18) (2012) 13376–13389. R. Ferdous, V. Osmani, O. Mayora, Smartphone app usage as a predictor of perceived stress levels at workplace, in: Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2015, pp. 225–228. A. Maxhuni, P. Hernandez-Leal, L.E. Sucar, V. Osmani, E.F. Morales, O. Mayora, Stress modelling and prediction in presence of scarce data, J. Biomed. Inform. 63 (2016) 344–356. M. Ciman, K. Wac, O. Gaggi, iSenseStress: Assessing stress through humansmartphone interaction analysis, in: 2015 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), IEEE, 2015, pp. 84–91. M. Ciman, K. Wac, Individuals’ stress assessment using human-smartphone interaction analysis, IEEE Trans. Affective Comput. PP (99) (2016) 1–15. G.C.-L. Hung, P.-C. Yang, C.-C. Chang, J.-H. Chiang, Y.-Y. Chen, Predicting Negative Emotions Based on Mobile Phone Usage Patterns: An Exploratory Study, JMIR Research Protocols, Vol. 5, 2016, (3). U. Reimer, E. Laurenzi, E. Maier, T. Ulmer, Mobile Stress Recognition and Relaxation Support with SmartCoping: User-Adaptive Interpretation of Physiological Stress Parameters, in: Proceedings of the 50th Hawaii International Conference on System Sciences, 2017. A. Bogomolov, B. Lepri, F. Pianesi, Happiness recognition from mobile phone data, in: 2013 International Conference on Social Computing (SocialCom), IEEE, 2013, pp. 790–795.

[133] N. Jaques, S. Taylor, A. Azaria, A. Ghandeharioun, A. Sano, R. Picard, Predicting students’ happiness from physiology, phone, mobility, and behavioral data, in: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, 2015, pp. 222–228. [134] M. Pielot, T. Dingler, J.S. Pedro, N. Oliver, When attention is not scarcedetecting boredom from mobile phone usage, in: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 2015, pp. 825–836. [135] J. Tang, Y. Zhang, J. Sun, J. Rao, W. Yu, Y. Chen, A.M. Fong, Quantitative study of individual emotional states in social networks, IEEE Trans. Affective Comput. 3 (2) (2012) 132–144. [136] D. Dai, Q. Liu, H. Meng, Can your smartphone detect your emotion?in: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), IEEE, 2016, pp. 1704–1709. [137] S. Bhattacharya, A predictive linear regression model for affective state detection of mobile touch screen users, Int. J. Mobile Hum Comput. Interact. (IJMHCI) 9 (1) (2017) 30–44. [138] A. Mottelson, K. Hornbæk, An affect detection technique using mobile commodity sensors in the wild, in: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 2016, pp. 781–792. [139] R. LiKamWa, Y. Liu, N.D. Lane, L. Zhong, Can your smartphone infer your mood, in: PhoneSense Workshop, 2011, pp. 1–5. [140] R. LiKamWa, Y. Liu, N.D. Lane, L. Zhong, MoodScope: building a mood sensor from smartphone usage patterns, in: Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, ACM, 2013, pp. 389–402. [141] Y. Ma, B. Xu, Y. Bai, G. Sun, R. Zhu, Daily mood assessment based on mobile phone sensing, in: 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks, IEEE, 2012, pp. 142–147. [142] P. Carmona, D. Nunes, D. Raposo, D. Silva, J.S. Silva, C. Herrera, Happy hourimproving mood with an emotionally aware application, in: 2015 15th International Conference on Innovations for Community Services (I4CS), IEEE, 2015, pp. 1–7. [143] A. Alshamsi, F. Pianesi, B. Lepri, A. Pentland, I. Rahwan, Network diversity and affect dynamics: The role of personality Traits, PLoS One 11 (4) (2016) e0152358. [144] N.D. Lane, M. Mohammod, M. Lin, X. Yang, H. Lu, S. Ali, A. Doryab, E. Berke, T. Choudhury, A. Campbell, Bewell: A smartphone application to monitor, model and promote wellbeing, in: 5th International ICST Conference on Pervasive Computing Technologies for Healthcare, 2011, pp. 23–26. [145] N. Jaques, S. Taylor, A. Sano, R. Picard, Multi-task, multi-kernel learning for estimating individual wellbeing in; Proc. NIPS Workshop on Multimodal Machine Learning, Montreal, Quebec, 2015. [146] R. Mafrur, I.G.D. Nugraha, D. Choi, Modeling and discovering human behavior from smartphone sensing life-log data for identification purpose, Hum.Centric Comput. Inf. Sci. 5 (1) (2015) 1. [147] S.T. Moturu, I. Khayal, N. Aharony, W. Pan, A. Pentland, Using social sensing to understand the links between sleep, mood, and sociability, in: 2011 IEEE 2015 15th International Conference on Third International Conference on Privacy, Security, Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), IEEE, 2011, pp. 208–214. [148] A. Doryab, J.K. Min, J. Wiese, J. Zimmerman, J.I. Hong, Detection of Behavior Change in People with Depression, AAAI, 2014. [149] M.N. Burns, M. Begale, J. Duffecy, D. Gergle, C.J. Karr, E. Giangrande, D.C. Mohr, Harnessing context sensing to develop a mobile intervention for depression, J. Med. Internet Res. 13 (3) (2011) e55. [150] R.W. Picard, Affective computing: from laughter to IEEE, IEEE Trans. Affective Comput. 1 (1) (2010) 11–17. [151] R. Cowie, Ethical issues in affective computing, in: The Oxford Handbook of Affective Computing, 2015, p. 334. [152] J. Chen, A. Bauman, M. Allman-Farinelli, A Study to Determine the Most Popular Lifestyle Smartphone Applications and Willingness of the Public to Share Their Personal Data for Health Research, Telemedicine and E-Health, 2016. [153] V. Pejovic, M. Musolesi, Anticipatory mobile computing for behaviour change interventions, in: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, ACM, 2014, pp. 1025–1034. [154] V. Pejovic, M. Musolesi, Anticipatory mobile computing: A survey of the state of the art and research challenges, ACM Comput. Surv. (CSUR) 47 (3) (2015) 47. [155] A. Kapadia, D. Kotz, N. Triandopoulos, Opportunistic sensing: Security challenges for the new paradigm, in: 2009 First International Communication Systems and Networks and Workshops, IEEE, 2009, pp. 1–10.

E. Politou et al. / Computer Science Review 25 (2017) 79–100 [156] P. Ohm, Broken promises of privacy: Responding to the surprising failure of anonymization, UCLA Law Rev. 57 (2010) 1701. [157] D. McMillan, A. Morrison, M. Chalmers, Categorised ethical guidelines for large scale mobile HCI, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2013, pp. 1853–1862. [158] L. Sweeney, Matching Known Patients to Health Records in Washington State Data, 2013, arXiv Preprint arXiv:1307.1370. [159] C. Spensky, J. Stewart, A. Yerukhimovich, R. Shay, A. Trachtenberg, R. Housley, R.K. Cunningham, SoK: Privacy on Mobile Devices–It’s Complicated, in: Proceedings on Privacy Enhancing Technologies, Vol. 2016, 2016, (3), pp. 96–116. [160] J. Staiano, N. Oliver, B. Lepri, R. de Oliveira, M. Caraviello, N. Sebe, Money walks: a human-centric study on the economics of personal mobile data, in: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 2014, pp. 583–594. [161] O. Tene, J. Polonetsky, Privacy in the age of big data: a time for big decisions, Stanford Law Review Online Vol. 64, 2012, p. 63. [162] G. King, Ensuring the data-rich future of the social sciences, Science 331 (6018) (2011) 719–721. [163] M. Musolesi, Big mobile data mining: good or evil? IEEE Internet Comput. 18 (1) (2014) 78–81. [164] Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance) European Union, OJ L 119. URL http://data.europa.eu/eli/reg/ 2016/679/oj (Accessed: 15.02.17). [165] M. Bateson, D. Nettle, G. Roberts, Cues of being watched enhance cooperation in a real-world setting, Biol. Lett. 2 (3) (2006) 412–414. [166] V. Griskevicius, J.M. Tybur, B. Van den Bergh, Going green to be seen: status, reputation, and conspicuous conservation, J. Pers. Soc. Psychol. 98 (3) (2010) 392. [167] R. Hellström, HaaS–Happiness as a Service–The future of Emotion Regulation Linkedin Article, 2015, (Accessed: 15.02.17). [168] EDPS starts work on a New Digital Ethics European Data Protection Supervisor PRESS RELEASE. URL http://secure.edps.europa.eu/EDPSWEB/webdav/ site/mySite/shared/Documents/EDPS/PressNews/Press/2016/EDPS-2016-05EDPS_Ethics_Advisory_Group_EN.pdf (Accessed: 15.02.17). [169] Introduction to the EGE. The European Group on Ethics in Science and New Technologies (EGE). URL https://ec.europa.eu/research/ege/index.cfm (Accessed: 15.02.17). [170] B. Guo, Z. Yu, X. Zhou, D. Zhang, From participatory sensing to mobile crowd sensing, in: 2014 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), IEEE, 2014, pp. 593– 598. [171] R.W. Picard, Emotion research by the people, for the people, Emotion Review 2 (3) (2010). [172] E. Kanjo, L. Al-Husain, A. Chamberlain, Emotions in context: examining pervasive affective sensing systems, applications, and analyses, Pers. Ubiquitous Comput. 19 (7) (2015) 1197–1212. [173] R. Rana, M. Hume, J. Reilly, R. Jurdak, J. Soar, Opportunistic and context-aware affect sensing on smartphones, IEEE Pervasive Comput. 15 (2) (2016) 60–69. [174] M.X. Patel, V. Doku, L. Tennakoon, Challenges in recruitment of research participants, Adv. Psychiatr. Treat. 9 (3) (2003) 229–238. [175] R.K. Ganti, F. Ye, H. Lei, Mobile crowdsensing: current state and future challenges, IEEE Commun. Mag. 49 (11) (2011) 32–39. [176] J. Wagner, E. André, F. Jung, Smart sensor integration: A framework for multimodal emotion recognition in real-time, in: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, IEEE, 2009, pp. 1–8. [177] A. Vinciarelli, M. Pantic, H. Bourlard, Social signal processing: Survey of an emerging domain, Image Vis. Comput. 27 (12) (2009) 1743–1759. [178] E. Alepis, M. Virvou, K. Kabassi, Mobile education: Towards affective bimodal interaction for adaptivity, in: Third International Conference on Digital Information Management, 2008, ICDIM 2008, IEEE, 2008, pp. 51–56. [179] H. Mousannif, I. Khalil, The human face of mobile, in: Information and Communication Technology-EurAsia Conference, Springer, 2014, pp. 1–20. [180] C. Vinola, K. Vimaladevi, A survey on human emotion recognition approaches, databases and applications, ELCVIA Electron. Lett. Comput. Vis. Image Anal. 14 (2) (2015) 24–44. [181] J. Wagner, E. Andre, F. Lingenfelser, J. Kim, Exploring fusion methods for multimodal emotion recognition with missing data, IEEE Trans. Affective Comput. 2 (4) (2011) 206–218. [182] C. Busso, Z. Deng, S. Yildirim, M. Bulut, C.M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, S. Narayanan, Analysis of emotion recognition using facial expressions, speech and multimodal information, in: Proceedings of the 6th International Conference on Multimodal Interfaces, ACM, 2004, pp. 205–211.

99

[183] H. Gunes, H. Hung, Is automatic facial expression recognition of emotions coming to a dead end? The rise of the new kids on the block, Image Vis. Comput. 55 (1) (2016) 6–8. [184] W.Z. Khan, Y. Xiang, M.Y. Aalsalem, Q. Arshad, Mobile phone sensing systems: A survey, IEEE Commun. Surv. Tutor. 15 (1) (2013) 402–427. [185] J. Broekens, Modeling the experience of emotion, Creating Synthetic Emotions Through Technological and Robotic Advancements, 2012, p. 1. [186] E. Hudlicka, Guidelines for designing computational models of emotions, Int. J Synth. Emotions (IJSE) 2 (1) (2011) 26–79. [187] M. Schröder, P. Baggia, F. Burkhardt, C. Pelachaud, C. Peter, E. Zovato, EmotionML–an upcoming standard for representing emotions and related states, in: International Conference on Affective Computing and Intelligent Interaction, Springer, 2011, pp. 316–325. [188] E. Hudlicka, H. Gunes, Benefits and limitations of continuous representations of emotions in affective computing: introduction to the special issue, Int. J. Synth. Emotions 3 (1) (2012) 1–6. [189] D. Grandjean, D. Sander, K.R. Scherer, Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization, Consciousness Cogn. 17 (2) (2008) 484–495. [190] M. Mortillaro, B. Meuleman, K.R. Scherer, Advocating a componential appraisal model to guide emotion recognition, Int. J. Synth. Emotions (IJSE) 3 (1) (2012) 18–32. [191] G.-J. de Vries, P. Lemmens, D. Brokken, S. Pauws, M. Biehl, Towards emotion classification using appraisal modeling, in: Psychology and Mental Health: Concepts, Methodologies, Tools, and Applications, IGI Global, 2016, pp. 552– 572. [192] R.A. Calvo, S. D’Mello, Affect detection: An interdisciplinary review of models, methods, and their applications, IEEE Trans. Affective Comput. 1 (1) (2010) 18–37. [193] Z. Pawlak, Rough sets and fuzzy sets, Fuzzy Sets and Systems 17 (1) (1985) 99–102. [194] H.-J. Zimmermann, Fuzzy set theory, Wiley Interdiscip. Rev. Comput. Stat. 2 (3) (2010) 317–332. [195] E. Hüllermeier, Fuzzy methods in machine learning and data mining: Status and prospects, Fuzzy Sets and Systems 156 (3) (2005) 387–406. [196] M.S. El-Nasr, J. Yen, T.R. Ioerger, Flame-fuzzy logic adaptive model of emotions, Auton. Agents Multi-Agent Syst. 3 (3) (2000) 219–257. [197] Y. Chen, Y. Chen, Affective computing model based on rough fuzzy sets, in: 2006 5th IEEE International Conference on Cognitive Informatics, Vol. 2, IEEE, 2006, pp. 835–838. [198] R.L. Mandryk, M.S. Atkins, A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies, Int. J. Hum.Comput. Stud. 65 (4) (2007) 329–347. [199] J.L. Salmeron, Fuzzy cognitive maps for artificial emotions forecasting, Appl. Soft Comput. 12 (12) (2012) 3704–3710. [200] H. Aktaş, N. Çağman, Soft sets and soft groups, Inform. Sci. 177 (13) (2007) 2726–2735. [201] R.E. Jack, O.G. Garrod, H. Yu, R. Caldara, P.G. Schyns, Facial expressions of emotion are not culturally universal, Proc. Natl. Acad. Sci. 109 (19) (2012) 7241–7244. [202] S. Russell, D. Dewey, M. Tegmark, Research priorities for robust and beneficial artificial intelligence, AI Mag. 36 (4) (2015). [203] A.F. Beavers, J.P. Slattery, On the moral implications and restrictions surrounding affective computing, in: Emotions and Affect in Human Factors and Human-Computer Interaction, 2017, pp. 143–161. [204] Ethically Aligned Design. URL http://standards.ieee.org/develop/indconn/ec/ autonomous_systems.html, organization = The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems, (Accessed: 15.02.17). [205] IEEE Ethically Aligned Design Document Elevates the Importance of Ethics in the Development of Artificial Intelligence (AI) and Autonomous Systems (AS), IEEE Press Release. URL http://standards.ieee.org/news/2016/ethically_ aligned_design.html (Accessed: 15.02.17).

Eugenia Politou holds a diploma degree in Electrical and Computing Engineering and an MSE degree in digital image processing and Internet databases both from Democritus University of Thrace, Xanthi, Greece. She is currently pursuing her Ph.D. degree in affective computing, human computer interaction and privacy from University of Piraeus. Since 1999 she has been working as a Researcher and a Software Analyst in Greece and in the UK for various national and European large-scale IT projects. She currently works as an IT Business Analyst at the e-Governance Directorate in the Greek Ministry of Health.

100

E. Politou et al. / Computer Science Review 25 (2017) 79–100 Efthimios Alepis received his B.Sc. in Informatics in 2002 and his Ph.D. in 2009, both from the Department of Informatics, University of Piraeus (Greece). He is Assistant Professor in the Department of Informatics, University of Piraeus since December 2013. He has authored/coauthored more than 60 scientific papers which have been published in international journals, book chapters and international conferences. His current research interests are in the areas of Object-oriented Programming, Mobile Software Engineering, Human–Computer Interaction, Affective Computing, User Modelling and Educational Soft-

ware.

Constantinos Patsakis is Assistant Professor at University of Piraeus. He holds a B.Sc. in Mathematics from the University of Athens, an M.Sc. in Information Security from Royal Holloway and a Ph.D. in Security from University of Piraeus. His main areas of research include cryptography, security, privacy and data anonymisation. He has several publications in peer reviewed international conferences and journals and participated in many national and European R&D projects. Previously, he worked as researcher at the UNESCO Chair in Data Privacy and Trinity College.