A Novel Fuzzy Based Home Occupant Monitoring System Using Kinect Cameras. Hossein Pazhoumand-Dar, Chiou Peng Lam, Martin Masek. School of ...
2015 IEEE 27th International Conference on Tools with Artificial Intelligence
A Novel Fuzzy Based Home Occupant Monitoring System Using Kinect Cameras
Hossein Pazhoumand-Dar, Chiou Peng Lam, Martin Masek School of Computer and Security Science Edith Cowan University Perth, Australia (h.pazhoumanddar, c.lam, m.masek)@ecu.edu.au improving the robustness of the approach in terms of extracting fuzzy attributes from numerical data. A further novelty is that, rather than dividing the day into a number of fixed time periods as in [8, 9], it is divided into a number of epochs, the duration of which is automatically learnt from the data to correspond to periods of activities of the occupant in a specific location. The rest of the paper is organized as follows: Section II briefly reviews relevant literature on abnormality detection in healthcare. Section III describes the proposed approach and details the steps of training the system and monitoring newly acquired data. The experimental evaluation of our technique is presented in Section IV followed by conclusions and future directions in Section V.
Abstract— In this paper, an approach is presented for the detection of abnormal behaviours in the activities of people living alone in their homes. The proposed approach takes input from Kinect cameras and extracts a number of visual attributes that represent the occupant’s location and orientation. A set of fuzzy logic parameters is first learnt from the training data. Next the proposed approach learns epochs of activities in each location and then generates models of normal behaviour patterns. Unusual behaviour is then detected in subsequent data by looking for patterns which differ from the learnt normal behaviours based on their time of occurrence, visual attributes, or duration. Experiments conducted showed the effectiveness of the proposed system. Keywords-behaviour monitoring; Kinect camera; fuzzy logic; activities of daily living; abnormality detection
II.
I. INTRODUCTION In many countries, the rising number of elderly people has resulted in demand for care services exceeding the resources of existing care providers [1]. This has driven interest in the development of automatic techniques that can monitor the wellbeing of elderly occupants. As there is no direct measure of wellbeing, studies [2] suggest using functional status by modelling activities of daily living (ADLs) of an individual. This model can then be compared to subsequent observed behaviour, where abnormal behaviour is then defined as any deviation from the modelled ADLs. A range of techniques [3, 4] have been proposed for modelling different aspects of the functional status of the elderly. Generally, the house is equipped with various types of sensors such as passive infrared sensors, magnetic switches, or video cameras. After collecting sensor data, normal behaviour of the occupant is modelled [5, 6]. This paper proposes an unsupervised method that first learns “normal” behaviour patterns of the occupant and then uses the learnt rules for detection of abnormal behaviour. To build the model, Kinect data captured during a fixed period of time from different locations in a home is used and different fuzzy attributes are extracted. Next the proposed approach learns epochs corresponding to different activities in each location and for each epoch, learns normal behaviour patterns by finding frequent occurrences among fuzzy attributes via the use of a fuzzy association rule mining algorithm [7]. The proposed approach extends upon existing fuzzy based approaches [8, 9] by applying a data driven technique to tune the parameters of the membership functions, 1082-3409/15 $31.00 © 2015 IEEE DOI 10.1109/ICTAI.2015.160
BACKGROUND
In recent years, many approaches have investigated the use of various sensor technologies, such as PIR and magnetic switches, for monitoring ADLs of the elderly in support of independent living [4]. Typically a model of normal behavior is estimated from the data, such as time spent in each room and number of sensor triggers. For example, in [10] the subject wears a RFID reader on their wrist and RFID tags are attached to different devices to estimate the interactions with the environment as a measure of ADLs. However, most of these approaches require equipping the house with many sensors during its construction. Another drawback is that they provide only rudimentary information whereas for detecting abnormal events such as falling down, more contextual information such as the orientation of the body is needed. To address these drawbacks, researchers suggested two main approaches. One is based on wearable sensors in which vital data, like heart or respiratory rates, and the orientation of body is acquired and monitored [11]. The other approach relies on the use of video cameras [12]. For example, Seki [9] used an omni-directional camera to obtain features extracted from the silhouette of a subject and to monitor behaviors using a fuzzy inference system. However, video cameras depend on appropriate lighting and wearable sensors rely on the occupant to consistently remember to wear them. Recently available low-cost 3D sensors, such as the Microsoft Kinect improve on traditional video cameras by operating on depth maps, providing information on people detected in the scene and offering light invariance. In the approach proposed by Rougier and Auvinet [13] Kinect 1128 1129
depth maps of a subject were used to detect fall events. They used a training data set of normal activities to put thresholds on two types of feature namely, the human centroid height relative to the ground and body velocity. Zhang and Tian [14] also proposed calculating deformation on the subject’s skeleton joint angles and their height to decide whether a fall has occurred using a tree based SVM classifier. Authors in [15] collected a dataset of depth maps for the health care domain and proposed new descriptors for classification of specific activities. In [16] a training dataset of depth maps were used to determine signatures of ADLs. To classify certain ADLs, a threshold was determined for signatures of each activity. However, given the wide range of activities and the considerable variability within particular activities at different time scales, these approaches are prone to many false classifications. In addition, labelling a large amount of training data requires much labour and time. III.
݊݅ݐܽݎݑܦೝ is the fuzzy set describing the expected duration of that behaviour pattern. Also, r is the ID of a particular Kinect camera and ܶௗ ೝ is the fuzzy set for the i-th epoch, learnt from training dataset dr. TABLE I.
AN EXAMPLE OF FUZZY RULE SET Antecedent
THE PROPOSED APPROACH
Index
r Time
rule1
1
rule2
Consequent
Xc
Yc
AR
Duration
ୢଵభ
low
low
medium
high
݊݅ݐܽݎݑܦଶభ
1
ୢଶభ
medium
high
high
high
݊݅ݐܽݎݑܦଵభ
rule3
2
ୢଵమ
medium
high
high
high
݊݅ݐܽݎݑܦଵౚమ
rule4
3
ୢଵయ
high
medium
medium
ڭ
ڭ
ڭ
ڭ
ڭ
ڭ
భ మ
ୣభ భ
medium ݊݅ݐܽݎݑܦୣౚయ ଶ
ڭ
ڭ
Determining these rules involves four steps: (1) defining the fuzzy sets for the observed attributes, where we use a data-driven technique to determine the mapping of observed feature attribute values to membership of linguistically labelled fuzzy sets (details in Section III.A.1), (2) For each location determining a fuzzy set of epochs corresponding to usual times where an occupant is active in a location (Section III.A.2), (3) Fuzzy association rule mining to identify frequent behaviour patterns as combinations of observed attributes within each epoch (Section III.A.3), and (4) Determining the expected duration of behaviour patterns within the epoch (Section III.A.4). Parameters of the proposed approach are learnt from a training dataset, Dtraining, consisting of sets of observations of normal behaviour from all Kinect cameras in the environment; that is, ܦ௧ ൌ ሼ݀ଵ ǡ ǥ ǡ ݀ோ ሽ, each ݀ (1 r R) comprising of all observations obtained from a specific Kinect camera r, where R indicates the total number of Kinect cameras. Each observation consists of a set of attributes ሼܽ ሽ, denoting the number of attributes, where in our case k=1,…,4, the attributes being ሺܺܿǡ ܻܿǡ ܴܣǡ ߠሻ. 1) Defining fuzzy sets for observed attributes: the role of this step is to discretise each attribute into a number of linguistic terms, which can then be associated with behaviours during ADLs. For instance, when the occupant is lying on the floor, the value of the orientation parameter, ߠ, might be mapped to the linguistic term “low”. As the range of each attribute value varies, the mapping of values to the terms needs to be determined. For each attribute ܽ , let ݀݉ሺܽ ሻ ൌ ሾ݈ೖ ǡ ݄ೖ ሿdenote its domain across all its observations in Dtraining. Hence, ݈ೖ and ݄ೖ in ݀݉ሺܽ ሻ are the minimum and maximum values of ܽ , respectively. A number (J) of fuzzy sets,݂ೖ , can be defined over the domain of each attribute ܽ , such that ܨೖ ൌ ሼ݂ଵೖ ǡ ǥ ǡ ݂ೖ ሽ. Each fuzzy set, ݂ೖ in ܨೖ , represents the j-th fuzzy set in ܨೖ and has an associated linguistic term as well as a membership function ߤ݂݆ ሺݔሻ such that ߤ݂݆ ሺݔሻ
For data acquisition, several Kinect cameras, each covering a different functional sub-area, are set up in the house. A common limitation of any technique involving the use of cameras is dealing with blind spots. To address this issue, we make sure that cameras are configured for maximising coverage and minimising areas of blind spot in the functional areas of activities. From each Kinect, observations are taken at one-second intervals, and those in which a person is detected are stored. Each stored observation includes the depth map, a binary silhouette mask of the occupant, a time stamp, and a camera ID. Three features are extracted from each recorded observation: the location of the detected occupant Centre of Gravity, calculated from the binary silhouette mask of the occupant as the pixel location ሺܺܿǡ ܻܿሻ, the Aspect Ratio (AR) of the 3D axis-aligned bounding box of the subject, defined as the ratio of the box height to the length of the width-depth diagonal, and Orientation (ߠ) of the occupant, between 0 and 90 degrees, obtained by fitting an ellipse to the silhouette of the occupant and measuring the angle of its major axis to the x-axis. These features, along with their associated time stamp and location (camera ID) are used in the training phase where normal behaviour patterns are learnt in the form of a set of fuzzy rules, and in a monitoring phase, where the approach attempts to identify differences to the expected learnt behaviour. Details of steps in the training and monitoring phases are described below. A. Training In this phase, normal behaviours of the occupant are modelled using a fuzzy rule set. Table I shows an example where each rule is in the form of: “IF Location is r AND Time is ܶௗ ೝ AND Xc is A1 AND Yc is A2 AND AR is A3 AND is A4 THEN Duration is
݊݅ݐܽݎݑܦೝ ̶ Here A1, A2, A3, and A4 are fuzzy sets of the captured attributes defining a frequent behaviour pattern and
݀݉ሺܽ݇ ሻ ՜ ሾͲǡ ͳሿ.
1130 1129
ܽ݇
ܽ݇
In our work, J was set to 5 for eachh fuzzy set, with linguistic terms of {Low, LowerMeddium, Medium, UpperMedium, High}, as is commonly useed by others [8]. Triangular membership functions, due to thheir simplicity of calculation and good performance when no information about the distribution of variable values iis available [17], were used for these fuzzy sets. The three parameters that definee a triangular membership function were learnt from thee training dataset for each attribute using the Fuzzy C C-Means (FCM) clustering algorithm [18]. This involves using FCM to cluster all values of an attribute ܽ into J clusters. From each cluster, the cluster boundaries and the location of the cluster centre are used to determine the cluuster membership function parameters. More specifically, leet the upper and lower bounds of cluster j ሺͳ ݆ ܬሻ, containing data points ܥ ൌ ሼଵ ǡ ǥ ǡ ሽ be defined by ݑೕ ൌ ݔܽܯሺܥ ሻ and
(a)
݈ೕ ൌ ݊݅ܯሺܥ ሻ, respectively. For the fuzzzy set ݂ೖ with cluster center ܿೕ , the membership functionn for element ݔis defined by (1).
ܽ݇
Ͳ݂݅ ݔ ݈ೕ
௫ିೕ ೕ ିೕ ௨ ି௫
(b)
݂݈݅ೕ ൏ ݔ ܿೕ
۔ೕ ݂݅ܿ ൏ ݔ ݑ ೕ ೕ ۖ௨ೕ ିೕ ۖ Ͳ݂݅ݑ ൏ ݔ ೕ ە
Figure 1. Learning membership functio on parameters: (a) shows the histogram of observation values for and (b b) shows results of step 1 using a clustering based membership function deefinition for this distribution.
(1)
Number of observations
ߤ݂݆ ሺݔሻ ൌ
ۓ ۖ ۖ
Fig. 1 (a) shows the distribution of Ʌ for 30 days of observations across three locations. T The results of clustering-based membership function for this attribute is shown in Fig. 1 (b). 2) Defining fuzzy sets for epochs of aactivities: In this step, we determine a set of epochs during which activities are expected to occur for each monitoredd location. Given that only observations where the occupannt is detected are recorded, for a set of observations from a pparticular Kinect camera, ݀ , the amount of activity in that cameras location within a particular time period can be obtained by counting the number of observations recorded in thhat period. In our implementation, we split the day into 24 hourly intervals, producing a time series. Fig. 2 shows an example of this time series for 30 days of data for the livving room in an experimental home. As this series is built oover a number of days, peaks typically correspond to distinct regular activities at a particular time, such as watching the nigghtly news. In our approach, the number of peakss ሺܰܲௗೝ ሻ in the series is detected using the MATLAB finndpeaks function, which defines a peak as a sample largger than its two neighbours, and we use it to estimate the number of different ADLs in ݀ . The observations aare clustered into ܰܲௗೝ clusters using the FCM algorithm, each cluster ݁ௗ ೝ representing an epoch of time, with the totaal set of epochs in ே ݀ ൌ ሼ݁ௗଵೝ ǡ ǥ ǡ ݁ௗೝ ೝ ሽ. Each epoch ݁ௗ ೝ in ݀ is then modelled uusing a fuzzy set ܶௗೝ using the observations belonging too the epoch to produce a Gaussian type membership funcction in order to
10000 8000 6000 4000 2000 0 0
2
4
6
8
10
12
14
16
18
20
22
24
Time of day (hours) Figure 2. An example of total hourly coun nt of observations for a living room area during 30 0 days.
capture the variations in the duratiion of each epoch. More specifically, let the mean and stand dard deviation of the time interval of cluster ݁ௗ ೝ (1 i ܰܲ ܲௗೝ ) be given by ݉ௗ ೝ and ߪௗ ೝ ǡrespectively. The associated membership m function for ܶௗ ೝ is defined as (2). ߤ ் ሺݔሻ ൌ ݁ ݔെ ೝ
ሺ௫ିೝ ሻ ఙ
ೝ
൨
(2)
This procedure for determining g epochs is repeated for each Kinect camera location in the training set and a different number of fuzzy sets are a obtained from each location, corresponding to the ADLs and their duration in that location. Fig. 3 shows the induced fuzzy sets associated with epochs from the data shown in n Fig. 2. As shown in Fig. 3, the occupant has three epochs of o major activities in the
1131 1130
epoch ୧ୢ౨ . It involved two steps to model the variations in the typical duration of each frequent behaviour. The process proceeds as follows.
living room. Also, this shows that the occupant performs more ADLs in this location after 1 PM since two different epochs are estimated after 1 PM.
First, for each behaviour pattern ܾೝ in ܲܨ , sequences ೝ of consecutive observations that correspond to that pattern
ೝ are found in ܦ௨௧௩௨௦ . This is done by sorting
ೝ observations in ܦ௨௧௩௨௦ according to their time stamp and then putting all consecutive observations having
the same attributes asܾೝ into a sequence, ܵ݁ݍೝ . Hence,
each frequent behaviourܾೝ in ܲܨ will be related to a set ݅
݅
durations for sequences in ݎݑ݅ݒ݄ܽ݁ܤ ೝ are calculated as
3) Identifying frequent behaviour patterns: In this step, the observations belonging to the cluster of every epoch, ݁ௗ ೝ , are examined to determine a set of frequent behaviour patterns. This is done using the fuzzy association rule mining algorithm presented in [7]. This algorithm examines different combinations of attributes in the observations and outputs fuzzy association rules with associated support values. The support value of each behaviour pattern indicates the proportion of observations in ݏௗ ೝ corresponding to that behaviour pattern. The set of generated fuzzy association rules is then pruned according to their support values using a threshold MinSupp. In the pruned rule set, each combination of fuzzy attributes in the antecedent and consequent parts of a rule is considered as a frequent behaviour pattern, indicated as
݉ೝ and ߪ ೝ , respectively. Using these two values, we
define a fuzzy set, ݊݅ݐܽݎݑܦೝ , over the time domain for describing the upper bound of the duration of behaviour
pattern ܾೝ . A z-shaped membership function [19] is
associated to this fuzzy set with two parameters ݑൌ ሺ݉ೝ ሻ
and ݒൌ ሺ݉ೝ ͵ߪ ೝ ሻ to characterise its break points, as given by (3).
ߤ
ܾೝ ,
where q indicates the q-th frequent behaviour pattern obtained for that epoch. These are used to generate a list of frequent behaviour patterns (ܲܨ ) of the occupant during ೝ
௨௧ ೝ
ሺݔሻ ൌ
ͳǡ ݔ ݑ ۓ ௫ି௨ ଶ ௨ା௩ ۖͳ െ ʹ ቀ ቁ ǡ ݑ൏ ݔ ௩ି௨ ଶ
ଶ
ʹ۔ቀ௫ି௩ ቁ ǡ ௨ା௩ ൏ ݔ ݒ ଶ ۖ ௩ି௨ Ͳەǡ ݒ ݔ
(3)
In addition to modelling duration of frequent behaviours, we also estimate a maximum duration for infrequent behaviour, ܣܧ , obtained from processing all ܦ௨௧௩௨௦ obtained for location r. That is, sequences of consecutive observations for all ܦ௨௧௩௨௦ for location r are obtained and the length of the longest one is used as the value for ܣܧ .
epoch ݁ௗ ೝ associated to location r. Once the set of frequent behaviour patterns is determined, the observations from each epoch ୧ୢ౨ are divided into two datasets:
ೝ , containing observations from 1- ܦ௨௧௩௨௦ epoch ୧ୢ౨ that have combination of attributes corresponding to a frequent behaviour pattern in ܲܨ .
B. Monitoring The monitoring phase takes the fuzzy rule set obtained during the training phase as input, and for each location, it checks whether the behaviour of the occupant is in the set of frequent behaviours (i.e. normal behaviour). For each observation, if the combination of attributes corresponds to a frequent behaviour pattern, the duration of the behaviour during consecutive observations is calculated and evaluated against the consequent part of the corresponding fuzzy rule for that behaviour pattern. Alternatively, if no frequent behaviour can be matched to the observation, the system will consider the duration of infrequent behaviours for the location of observation and wait for a specific amount of time. If the ongoing behaviour during that time remains unknown, the system will raise an
ೝ
ೝ
2- ܦ௨௧௩௨௦ , containing the rest of the observations in ୧ୢ౨ . These datasets are used in the next step to determine expected durations of the two behaviour classes. 4) Determining expected duration of behaviour patterns: This step uses the set of frequent behaviour fuzzy association rules ܲܨ , and the set of observations ೝ
ೝ
of sequences,ݎݑ݅ݒ݄ܽ݁ܤ ೝ ൌ ቄሺͳሻ݁ ݎ݀ݍǡ ǥ ǡ ሺሻ݁ ݎ݀ݍቅ. In the second step, the mean and standard deviation of
Figure 3. An example of Gaussian membership functions for time of observations in a living room area
ೝ ), for each matching these rules (i.e., ܦ௨௧௩௨௦
1132 1131
alarm. The process of the monitoring phase is shown in Fig. 4. In Fig 4, NormalDuration is to indicate the duration of a currently ongoing behaviour belonging to a frequent behaviour pattern andAbnormalDurationholds the duration of an ongoing behaviour not in the frequent behaviours list, which could be either an infrequent or an abnormal behaviour. 1. NormalDuration=0 2. AbnormalDuration=0 3. CurrentBehaviour=0 4. while (Monitoring) 5. if ( new observation ) 6. obtain ୫ 7. compute QueryBehaviour using (5) 8. if (QueryBehaviour)>0 9. AbnormalDuration=0 10. if CurrentBehaviour== QueryBehaviour 11. NormalDuration ++ 12. if ρ ሺܰ݊݅ݐܽݎݑܦ݈ܽ݉ݎሻ==0 ౚ ୈ୳୰ୟ୲୧୭୬౧ ౨
13. trigger Alarm 14. end if 15. else 16. CurrentBehaviour= QueryBehaviour 17. NormalDuration=1 18. end if 19. else 20. AbnormalDuration++ 21. NormalDuration=0 22. if AbnormalDuration>ܣܧ 23. trigger Alarm 24. end if 25. end if 26. end if 27. end while Figure 4. The algorithm for monitoring the occupant in the monitoring phase
For each new query observation obtained, line 6 extracts the attributes to obtain a feature vector ଵ ଶ ଷ ସ ହ ଵ ǡ ǡ ǡ ǡ ǡ ሽ. In ܱ , is the index of ܱ ൌ ሼ ଶ location and includes the time of observation, ଷ ସ ହ ǡ ǡ ǡ ǡ are the respectively. Components values associated with ܺ ǡ ܻ ǡ AR, and , respectively. Line 7 evaluates ܱ against the fuzzy ruleset using a fuzzy concept called firing strength [20]. In our case, firing strength of a rule is the degree of satisfaction of the antecedent of the rule by the elements of ܱ . More specifically, let ܸ ൌ ሼݒଵ ǡ ݒଶ ǡ ݒଷ ǡ ݒସ ǡ ݒହ ǡ ݒሽ be the set of variables in the antecedent of rule ݈݁ݑݎ and ܣൌ ሼ݂ଵ ǡ ݂ଶ ǡ ݂ଷ ǡ ݂ସ ǡ ݂ହ ǡ ݂ ሽ be a set of fuzzy sets associated to those variables, and also let ൛ߤభ ǡ ߤమ ǡ ߤయ ǡ ߤర ǡ ߤఱ ǡ ߤల ൟ be the set of membership functions of A, such that ߤೢ ሺ ݓൌ ͳǡ ڮǡሻ represents the membership function of ݂௪ . Given ܱ , the following
formula is used to calculate the firing strength of ݈݁ݑݎ in the training rule set. ௪ ሻ ݂ழை ǡ௨ வ ൌ ς௪ୀଵ ߤೢ ሺ
(4)
The best match to ܱ is given by the rule with the maximum firing strength (see (5)). ܳ ݎݑ݅ݒ̴݄ܾܽ݁ݕݎ݁ݑൌ ଵஸ୮ஸ ݂ழை ǡ௨ வ (5) Line 8 checks if ܳ ݎݑ݅ݒ݄ܽ݁ܤݕݎ݁ݑis greater than zero, implying that it contains the index of a rule characterising a behaviour pattern. If yes, in line 10 we check whether the behaviour of ̴ has also been observed in the previous observation. If yes, the duration for the ongoing identical (matched) behaviour, defined as NormalDuration,
ܳݎݑ݅ݒ݄ܽ݁ܤݕݎ݁ݑ (݈݁ݑݎொ௨௬௩௨ ). According to (3), the domain of the membership function for duration of behaviour is up to three standard deviations from the mean value of duration. So, by the time NormalDuration is no longer satisfying the consequent part of ݈݁ݑݎொ௨௬௩௨ , we are at the 99% confidence interval that the duration of the ongoing behaviour no longer belongs to the behaviour modelled by the rule and therefore, an alarm is raised (line 13). Alternatively, if ܳ ݎݑ݅ݒ݄ܽ݁ܤݕݎ݁ݑhas a new value corresponding to a new frequent behaviour pattern, this new value is then assigned to CurrentBehaviour (line 16) and NormalDuration will then be re-initialised to 1 (line 17). If ܱ cannot be matched to any rules in the fuzzy rule set, it means that this observed behaviour pattern of the occupant is either corresponding to one of the infrequent behaviours or is considered as an abnormal situation (that is, an unusual activity which is not part of the training data, such as fallen on the floor). If this condition persists during consequent observations, the system
ǡ AbnormalDuration, while monitoring the scene and if this counter reaches the location-specific threshold, ܣܧ , for camera location r, as was described in Section III.A.4, an alarm will be raised. IV. EXPERIMENTAL RESULTS Since no public dataset supplies continuous Kinect data for ADLs inside a residential home, we collected data for one month in a residence with a single occupant in order to test the effectiveness of the proposed approach. The activities were undertaken to simulate an elderly, retired occupant living alone. This dataset was captured using three Kinect cameras, with one placed in the kitchen, living room, and bedroom. This dataset consists of a training set and an unseen test set. The training set consists of nearly two million observations of behaviour patterns associated with typical (or normal) ADLs of the occupant. The test set holds some sequences of normal behaviours (i.e. typical ADLs)
1133 1132
(a)
(b)
(c)
Figure 5. Example of observations in the training dataset. (a) The occupant is sitting on the sofa in the living room, (b) sleeping in the bedroom, and (c) preparing a meal in the kitchen
evaluate the performance of the proposed approach during the monitoring phase. a) Scenarios of normal behaviour patterns: There are 20 sequences associated with normal behaviours. The proposed approach classified 18 of these sequences correctly (i.e. as normal behaviours). Some instances of scenarios for normal behaviours that were classified correctly are as follows: Scenario 1: Sitting on the sofa in the living room in the evening: Fig. 5 (a) shows an image from a sequence where the occupant sat on the sofa in the living room for 60 minutes. Monitoring this sequence triggered the corresponding rule in the rule base for Sitting on the sofa in the living room in the evening to be fired and resulted in this observation being considered normal at the end of the sequence. The duration of 60 minutes associated with this monitored sequence is also well within the bounds of the learnt value normally associated with this behavioural rule. Scenario 2: Seven hours of continuous sleep on the bed in the bed room during night: As another scenario of normal behaviour pattern, the image sequence including the usual behaviour pattern of sleeping on the bed for 7 consecutive hours during the night and early morning was tested. An image from this sequence is shown in Fig. 5 (b). The proposed approach determined this behaviour pattern normal since the combination of the attributes of this monitored sequence was within the bounds associated with the set of learnt frequent behaviours for the bedroom and hence, these observations triggered the associated rule to fire. The two misclassified sequences from this set were associated with behaviours that were not present in the training data for a specific location. Both sequences were first detected by the approach as belonging to the set of ܦ௨௧௩௨௦ and with subsequent monitoring, the duration of the respective behaviour also exceeded the corresponding ܣܧ , resulting in both being considered as an abnormal behaviour. For example, Fig. 7 (a) shows an image from a sequence involving 7 minutes of monitoring, where the occupant was crouching down on the kitchen floor while cleaning inside of the fridge. This activity was not present in the original training dataset. The combination of attributes associated with this activity was not within the bounds associated with the set of learnt frequent behaviours for the kitchen. Therefore, no rule triggered during the monitoring of this sequence and resulted in the approach triggering an alarm after the elapsed duration exceeds its
and abnormal events (e.g. occupant lying on the floor of the kitchen). Fig. 5 (a-c) shows examples of observations associated with sequences of various typical ADLs in the training set. A. Training in the experimental setup Training was performed by following the procedure described in Section III.A. Fig. 6 shows the plot of support level versus the number of behaviour patterns from processing the first epoch of each location. As observed in Fig. 6, the number of behaviour patterns with greater than 2% support in the kitchen, living room, and bedroom are 6, 7, and 15 respectively. To obtain frequent behaviour patterns for each epoch, we use MinSupp (see Section III.A.3). From examining plots for different epochs we see that they all have similar characteristics as shown in Fig. 6, that is support level flattens out and approaches zero after a specific number of rules. Thus, we have empirically chosen to pick rules with at least 2% support and accordingly set MinSupp to this value. Reducing MinSupp to a smaller value, such as 1%, increases the number of fuzzy rules and subsequent analysis of the classification results associated with using either values of MinSupp (1% or 2%) showed no significant difference between the two.
Support
14% 12% 10% 8% 6% 4% 2% 0%
Kitchen Living Room Bedroom
1 Number 5 of 9behaviour 13 patterns 17 21 29support 33 level 37 with a 25 specific Figure 6. The number of behaviour patterns with a specific level of support obtained for the three locations in the dataset
Subsequently, the numbers of fuzzy rules obtained for monitoring the locations of kitchen (r=1), living room (r=2), and bedroom (r=3) were 81, 95, and 60, respectively. Also, as part of the training phase, the values of ܣܧ୰ for these three locations were calculated from their corresponding ܦ௨௧௩௨௦ , with ܣܧଵ ൌ ʹͺͲ, ܣܧଶ ൌ ͵ͷ, and ܣܧଷ =167 seconds, respectively. B. Evaluating the proposed approach in the monitoring phase 40 sequences of different scenarios for normal and abnormal behaviours in the unseen test set were used to
1134 1133
corresponding ܣܧଵ for this location. By inttroducing a more comprehensive training dataset for typicaal ADLs of the occupant, however, the proposed approachh will be able to handle such situations correctly. b) Scenarios of abnormal behaviour patterns: There are 20 sequences representing differennt scenarios of abnormal activities. The approach classiffied all of these sequences correctly (as abnormal). Scenario 1: Lying and sitting on the flooor in the kitchen. Fig. 7 (b) and (c) show an image from two sequences of 10 minutes where the occupant was lying annd sitting on the floor in the kitchen, respectively. The combination of attributes obtained from observations associated with each of these sequences did not trigger any ruule in the set of learnt frequent behaviours, and as suchh the approach considered these behaviours as belonginng to the set of infrequent behaviours and the associated dduration for each of these sequences was monitored. The proposed approach raised an alarm after the elapsed duuration of these behaviours exceeded the learnt value (greatter than than 280 seconds (ܣܧଵ ሻ).
being considered normal. However, as time passed, the duration of this behaviour then ex xceeded the learnt value associated with this behavioural ru ule, thus its membership degree for the consequence part of the t rule was then reduced to zero, causing an alarm to be raised. This scenario is another example where using a more comprehensive training dataset consisting of typ pical ADLs with more variability would enable the propo osed approach to learn a more representative rule base for characterising the behaviour of the occupant more acccurately. Scenario 3: Performing a normal n activity at an abnormal time of day. As another example of an abnormal situation, we tested a sequence in which the occupant was walking around in the living room at 1 AM in the morning. Since no epoch of activity was learnt for that specific time in the living room data (see Fig. 3), although combination of other attributes corresponded to a frequent f learnt behaviour pattern of walking belonging to oth her epochs, no rule from the learnt rule set fired. Hence, the behaviour in this sequence fell into the set of infrequ uent behaviour and when the elapsed duration for this behaviiour was more than ଶ , the system raised an alarm. To measure accuracy of the proposed approach, we utilize the F-measure [21] which balances the two types of errors (false negatives and false po ositives) by calculating a geometrical mean of precision and recall: r ்
ܨൌʹൈ (a)
்ାி ் ்ାி
(b)
ൈ
் ்ାிே ் ሺሻ ்ାிே
In (6), TP and FP stand for the number n of test sequences of abnormal and normal behaviou urs, respectively, which caused an alarm to be raised. FN indicates the number of sequences of normal behaviours thaat are classified correctly. Considering all the unseen test sequences, s the approach showed an accuracy of 95% (TP=38 8, FP=2, and FN=0).
(c)
V. DISCUSSION AND D CONCLUSION This paper proposed an appro oach to monitor and to detect abnormal behaviour patternss of people living alone, based on fuzzy logic. The proposeed approach learns rules associated with frequent behaviour patterns of the occupant using a fuzzy association rule mining algorithm, and automatically detects abnormal beehaviours. Evaluation of the approach using sequences off normal and abnormal behaviours have shown the effectiveness of the proposed algorithm in terms of picking up vaarious abnormal events as well as correct classification of normal n behaviours. The evaluation has also highlighted the need for training data with greater variability within eaach group of ADLs for better classification. In the event of monitoring a currrent behaviour where the subject becomes occluded, the app proach will terminate the current behaviour. When the occcupant comes back into view, the approach will process the input as part of a new behaviour. That is, the previous beehaviour of the occupant gets terminated and is replaced by a newly matched behaviour the next time Kinect deteects the person.
(d)
Figure 7. Examples of images from unseen test sequeences for the kitchen. (a) An image from a sequence of a new behaviour off crouching down on the floor while cleaning inside the fridge. (b) and (c)) showing an image from abnormal situations of lying and sitting on the flloor respectively. (d) An image of a person crouching down on the floor to interact with objects inside the cabinet
Scenario 2: Performing a normal activitty for an unusual duration. Fig. 7 (c) shows one image from a test sequence of 10 minutes where the occupant was crouchhing down on the kitchen floor during the first epoch off the day. The combination of attributes in this seqquence actually corresponded to a frequent behaviour patterrn for that epoch in the kitchen since the occupant typically aat this time of the day will be putting away objects inside thee kitchen cabinet (putting dishes washed the previous nightt back inside the cabinet). Hence, the initial firing sttrength of the corresponding rule in the rule set resulted inn this observation
1135 1134
In future work we will: (i) investigate how to determine the parameter MinSupp automatically using a data-driven approach, (ii) further examine the role of separating the day into epochs on the classification accuracy, (iii) examine the impact of varying the number of fuzzy sets on the classification results. Furthermore, because the Kinect SDK allows us to track more than one person in the scene, our future work also involves extending the proposed approach to monitor ADLs of multiple occupants living in the same house.
[9]
[10]
[11]
REFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
N. Labonnote and K. Høyland, "Smart home technologies that support independent living: challenges and opportunities for the building industry–a systematic mapping study," Intelligent Buildings International, pp. 1-26, 2015. J. H. Shin, B. Lee, and K. S. Park, "Detection of abnormal living patterns for elderly living alone using support vector data description," Information Technology in Biomedicine, IEEE Transactions on, vol. 15, pp. 438-448, 2011. B. Kwolek and M. Kepski, "Improving fall detection by the use of depth sensor and accelerometer," Neurocomputing, 2015. S. A. Lowe and G. ÓLaighin, "Monitoring human health behaviour in one's living environment: a technological review," Medical engineering & physics, vol. 36, pp. 147168, 2014. N. Noury, P. Barralon, N. Vuillerme, and A. Fleury, "Fusion of Multiple Sensors Sources in a Smart Home to Detect Scenarios of Activities in Ambient Assisted Living," International Journal of E-Health and Medical Communications (IJEHMC), vol. 3, pp. 29-44, 2012. M. Munstermann, T. Stevens, and W. Luther, "A novel human autonomy assessment system," Sensors, vol. 12, pp. 7828-7854, 2012. C. M. Kuok, A. Fu, and M. H. Wong, "Mining fuzzy association rules in databases," ACM Sigmod Record, vol. 27, pp. 41-46, 1998. T. Martin, B. Majeed, B.-S. Lee, and N. Clarke, "Fuzzy ambient intelligence for next generation telecare," in Fuzzy Systems, 2006 IEEE International Conference on, 2006, pp. 894-901.
[12]
[13]
[14]
[15]
[16] [17] [18]
[19] [20] [21]
1136 1135
H. Seki, "Fuzzy inference based non-daily behavior pattern detection for elderly people monitoring system," in Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE, 2009, pp. 6187-6192. T. Gu, Z. Wu, X. Tao, H. K. Pung, and J. Lu, "epsicar: An emerging patterns based approach to sequential, interleaved and concurrent activity recognition," in Pervasive Computing and Communications, 2009. PerCom 2009. IEEE International Conference on, 2009, pp. 1-9. M. Kangas, A. Konttila, P. Lindgren, I. Winblad, and T. Jämsä, "Comparison of low-complexity fall detection algorithms for body attached accelerometers," Gait & posture, vol. 28, pp. 285-291, 2008. Z. Zhongna, C. Xi, C. Yu-Chia, H. Zhihai, T. X. Han, and J. M. Keller, "Activity Analysis, Summarization, and Visualization for Indoor Human Activity Monitoring," Circuits and Systems for Video Technology, IEEE Transactions on, vol. 18, pp. 1489-1498, 2008. C. Rougier, E. Auvinet, J. Rousseau, M. Mignotte, and J. Meunier, "Fall detection from depth map video sequences," in Toward Useful Services for Elderly and People with Disabilities, ed: Springer, 2011, pp. 121-128. C. Zhang, Y. Tian, and E. Capezuti, Privacy preserving automatic fall detection for elderly using RGBD cameras: Springer, 2012. B. Ni, G. Wang, and P. Moulin, "Rgbd-hudaact: A colordepth video database for human daily activity recognition," in Consumer Depth Cameras for Computer Vision, ed: Springer, 2013, pp. 193-208. L. J. Da Luz, M. Masek, and C. P. P. Lam, "Activities of daily living classification using depth features," 2013. W. Pedrycz, "Why triangular membership functions?," Fuzzy Sets Syst., vol. 64, pp. 21-30, 1994. J. C. Bezdek, R. Ehrlich, and W. Full, "FCM: The fuzzy cmeans clustering algorithm," Computers & Geosciences, vol. 10, pp. 191-203, 1984. R. C. Berkan and S. Trubatch, Fuzzy System Design Principles: Wiley-IEEE Press, 1997. D. Kukolj, "Design of adaptive Takagi–Sugeno–Kang fuzzy models," Applied Soft Computing, vol. 2, pp. 89-103, 2002. D. M. Powers, "Evaluation: from precision, recall and Fmeasure to ROC, informedness, markedness and correlation," 2011.
