Log-Viterbi algorithm applied on second-order hidden Markov model ...

Review

Log-Viterbi algorithm applied on second-order hidden Markov model for human activity recognition

International Journal of Distributed Sensor Networks 2018, Vol. 14(4) Ó The Author(s) 2018 DOI: 10.1177/1550147718772541 journals.sagepub.com/home/dsn

Yang Sung-Hyun1, Keshav Thapa1 , M Humayun Kabir2 and Lee Hee-Chan1

Abstract Recognition of human activities is getting into the limelight among researchers in the field of pervasive computing, ambient intelligence, robotic, and monitoring such as assistive living, elderly care, and health care. Many platforms, models, and algorithms have been developed and implemented to recognize the human activities. However, existing approaches suffer from low-activity accuracy and high time complexity. Therefore, we proposed probabilistic log-Viterbi algorithm on second-order hidden Markov model that facilitates our algorithm by reducing the time complexity with increased accuracy. Second-order hidden Markov model is efficient relevance between previous two activities, current activity, and current observation that incorporate more information into recognition procedure. The log-Viterbi algorithm converts the products of a large number of probabilities into additions and finds the most likely activity from observation sequence under given model. Therefore, this approach maximizes the probability of activity recognition with improved accuracy and reduced time complexity. We compared our proposed algorithm among other famous probabilistic models such as Naı¨ve Bayes, condition random field, hidden Markov model, and hidden semi-Markov model using three datasets in the smart home environment. The recognition possibility of our proposed method is significantly better in accuracy and time complexity than early proposed method. Moreover, this improved algorithm for activity recognition is much effective for almost all the dynamic environments such as assistive living, elderly care, healthcare applications, and home automation. Keywords Activity recognition, second-order hidden Markov model, log-Viterbi algorithm, time complexity, smart home

Date received: 8 August 2017; accepted: 5 March 2018 Handling Editor: Francesco Longo

Introduction People adopt independent lifestyles and emphasize the quality of their life. They wish to have easygoing and assisted living. Thus, the system with the intelligent ability to provide various services in accordance with user’s preference should be developed. But, in order to provide intelligent services, the system needs to understand and recognize human activity and behavior first.1,2 The objective of proposed work is to predict and recognize user activities in a smart home environment more conveniently and accurately with less complexity. If we able to create an accurate and fastest

activity recognition method, the smart home can provide the appropriate service in accordance with user desires. The activity of human represents the functional

1

Department of Electronic Engineering, Kwangwoon University, Seoul, Republic of Korea 2 Department of Electrical and Electronic Engineering, Islamic University, Kushtia, Bangladesh Corresponding author: Keshav Thapa, Department of Electronic Engineering, Kwangwoon University, Seoul 139-701, Republic of Korea. Email: [email protected]

Creative Commons CC BY: This article is distributed under the terms of the Creative Commons Attribution 4.0 License (http://www.creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (http://www.uk.sagepub.com/aboutus/ openaccess.htm).

2 status of the person. Human perform numerous activities from morning to night like brushing, toileting, eating, watching TV, listening music, cooking, sleeping, going outside, medication, and so on. The activity recognition in the real world is challenging due to diversity and complexity of user activities which affect the accuracy of recognition precisely. Many researchers have proposed many ideas and works but the accuracy and complexity still remain below the target in activity recognition evolution. The human activity recognition has broad application areas such as elderly care,3 remote health4 monitoring emergencies such as natural disasters, robotics,5 comfort applications in smart homes, and energyefficient urban spaces. Among all these, activity recognition in the smart home has gained its popularity. The number of existing study has focused on recognizing resident’s activities based on sensory data in the smart home.6 The activity recognition systems use several types of sensors, microphones, video cameras, RFID readers, wearable sensors, and embedded sensors with appliances to determine the state of the physical world. We mainly focused on using the external sensor rather than the inertial (wearable) sensor. These days a smartphone is also getting popularity as the sensing tools for monitoring purposes of individuals activity. Audio and video-based activity recognition are sophisticated to use because it requires processing of multidimensional data and may raise user’s privacy issue.7–9 Human activity recognition system needs access to sensors, computational process, and inferring mechanism to recognize the activity.10 The accuracy and computational complexity are the main parameters mostly considered for sustainable recognition of human activity. Inter-activity and intra-activity variations influence the accuracy and complexity. Likewise, errors in sensors, execution of activities, limited training data may impact the performance of recognition methods. Different types of machine learning approaches and probabilities reasoning algorithms have been proposed for accurate recognition. Conditional random field,11 Bayesian network,12,13 hidden Markov model (HMM),14 hidden semi-Markov model (HSMM),15 adaptive HMM16 techniques are probabilistic in nature and have been widely used for detecting the spatiotemporal aspects of the data. However, recognition accuracy is just satisfactory not better that actually have to be. And computational complexity is also very high for large dataset. This method of recognition is the datadriven approach. The data-driven approach has two discriminative and generative root approaches. Discriminative approaches are computationally efficient and good in prediction but this may suffer from overfitting. Generative approaches are flexible to uncertainty in the data but they required a large amount of data for recognition and prediction. Many

International Journal of Distributed Sensor Networks researchers used either of one, but some17 used both approaches. In this article, we present the log-Viterbi (LoV) algorithm applied on second-order HMM for recognition of activities in the smart home. The second-order HMM is an extension to the first-order HMM which contains additional information about previous activity that leads to better accuracy results. Second-order HMM relies on one observation in a current state and the transition probability function based on two previous states. Increment in order of secondorder HMM may lead to the computational complexity but use of logarithm function into the Viterbi algorithm eventually confront complexity problem. The LoV algorithm is the dynamic programming algorithm used to find the most likely sequence of hidden states by converting the products of a large number of probabilities into additions from observation sequence. Therefore, the complexity to process the product value is reduced which ultimately speed up the system. So for to our knowledge, the LoV algorithm applied into the second-order HMM makes our approaches noble from other existing one. We have evaluated the performance of activity recognition with regard to our proposed method using three fully annotated real-world datasets generated by Kyoto and Van Kasteren and hence, compared among other probabilistic approaches: Naı¨ ve Bayes (NB), conditional random field (CRF), HMM, and HSMM activity recognition algorithm. The remainder of the paper is organized as follows: related works of the activity recognition are in section ‘‘Related work.’’ Section ‘‘The proposed method’’ illustrates the proposed activity recognition method. The algorithm for the whole process is described in section ‘‘The proposed method.’’ Section ‘‘Evaluation and results’’ describes the experimental results and evaluation. The time complexity is covered in section ‘‘Time complexity’’ followed by conclusion in section ‘‘Conclusion.’’

Related work From late 90s to till date, human activity recognition is one of the interesting topics for many researchers worldwide. However, development of improving an accurate technique has some serious issues to meet the real-world condition. The realistic data collection, unobtrusive, portable, and inexpensive data acquisition system development, the design of extraction and reasoning algorithm, flexibility to support new and multi-users are the major challenges faced during activity recognition.18 The smart home project such as the Center for Advanced Studies in Adaptive Systems (CASAS)19 at Washington State University (WSU), Aware Home Research Initiative (AHRI)20 at Georgia Institute of

Sung-Hyun et al. Technology, Tim Van Kasteren,21 PlaceLab Massachusetts Institute of Technology (MIT)22 monitor resident’s activities, upgrading comfort by installing heterogeneous sensors but its efficiency depends only on the large number of sensor inputs and processing time. In a smart home, sensors play a vital role in observing and understanding human daily activities. Basically, the external sensors (pressure, light, heat, etc.) and wearable (accelerometer, ECG, etc.) sensors have been used for human activities recognition (cooking, sleeping, walking, running, etc.) in smart homes.23 Kasteren et al. used temporal probabilistic models (NB, HMM, CRF) to recognize the activities from the sensor data by dividing data into time-slices in constant length which may not be well suited for all activities. Tapia et al.1 used NB to learn different length timeslices for different activities during training data. Longrange dependencies between the observations within activity segments can be modeled by integrating sequential pattern mining, used to characterize the time spans during an activity execution, with the HSMM for the activity recognition. Video-based depth image compensation with colored markers and joint trajectory on HMM was implemented5 and analyzed with high accuracy but complexity remains high due to image processing. For the recognition of activity the label of activity class according to highest probability of sequences of the sensor technology is used in the NB classifier.24 Dynamic bayesian network (DBN) is tested to recognize activities by rebuilding the already learned models to use differences in activities for an incremental learning approach. Support vector mechanism (SVM)25–27 applied for classification of activities, considers redundancy of activities during feature selection process. Multilayer perceptrons25,28 and decision tree29 are used in activity classification. The combination of pattern clustering method such as K-pattern clustering algorithm an activity decision algorithm; an artificial neural network (ANN)

Figure 1. Block diagram of activity recognition.

3 featuring IoT on smart home environment30 showed some good accuracy but due to two-step process and that of IoT complexity, runtime may increase. Equally, decision tree algorithm also affords some better feasibility for more interpretability. For the big structured dataset, the neural network techniques such as deep learning31 and deep belief network are also widely used. Again restricted Boltzmann machine (RBM)32 and conditional restricted Boltzmann machine (CRBM)33 have been presented as probabilistic modeling tools. The discriminative models like condition random fields (CRFs)34 use independence assumption in which we learn the model parameters by optimizing conditional likelihood rather than joint likelihood. However, CRF is computationally uncontrollable and repeatedly depend on approximation techniques. Analysis of resident’s behavior in the smart home is done using the activity recognition. Audio35–video36,37 activity recognition is most suitable for healthcare and monitoring. However, the audio-visual method has privacy, complex, and pervasive issue.38 Deep neural network can only facilitate limited temporal modeling by executing into a fixed size sliding window of activity frame. Convolutional neural network (CNN); a broadly applied deep neural network, has the ability to extract features and can learn multiple layers of features but it is bit slower and complicated. Long–short term memory is recurrent neural networks with memory called cell to model the long range temporal dependencies in time series problems39 using the multimodal wearable sensor. In this work, we use probabilistic LoV algorithm applied on second-order HMM to overcome the time complexity of existing algorithm. Eventually, it also addresses the accuracy issue.

The proposed method The system architecture used in the activity recognition is shown in Figure 1. Although figure describes the

4

International Journal of Distributed Sensor Networks 3.

The observation distribution, also known as emission probability distribution and determined by bk = P(xt jyt )

where P(xt jyt ) is the probability of observation xt at state yt, bk observation probability. Second-order HMM finds a hidden state sequence which maximizes a joint probability P(x, y) of transition probability aijk and observation probability bk which is the probability that outcomes xt is observed in a state yt

Figure 2. State transition of second-order HMM.

overall system architecture of activity recognition, our main focus is on the development of reliable algorithm using second-order HMM. Hence, we proposed the LoV algorithm. Raw data are taken from the sensors. The feature is extracted from the sensor data. Leave one-day cross concept is considered for the evaluation, on which a single day data are used for testing. Meanwhile, 50% of the data are used as the training data while the remaining 50% for test data. LoV algorithm produces optimal path sequence of activity from given model. And finally, the desired activity is recognized.

Second-order HMM Normal HMM explains hidden state yt depends on the previous state yt1 together with observed variable xt at time t. However, in second-order HMM state at any time step t is dependent on their previous two states at t – 1 and t – 2, as shown in Figure 2. Unlike the transition probability distribution aij of normal HMM, transition probability distribution of second-order HMM is represented by aijk . There are three HMM elements that characterize the probability distributions as:

Pðx, yÞ =

N Y

pi aijk  bk

ijk = 1

= Pðy1 Þ

T Y

ð5Þ Pðyt jyt1 , yt2 Þ  P(xt jyt )

t=1

The parameters that maximize this joint probability are found by simply counting the number of occurrence of transitions, observations, and states. But the secondorder HMM parameters need to be adjusted to overcome the evaluation, decoding, and learning problems. New forward and backward functions are defined to address the evaluation problem in which probability of observation given in the model l = (pi , aijk , bk ) could be calculated. The forward function at (j, k) defines probability of observation X = x1 , x2 , x3 , . . . , xT and transition ajk between time t – 1 and t at ðj, kÞ = Pðx1 x2 x3    xT , yt1 , yt jlÞ, 2  t  T,

ð6Þ

1  j, k  N

at (j, k) can be obtained from at1 (i, j) where aij and ajk are two transitions between the states i to k at + 1 ðj, kÞ =

1.

ð4Þ

The initial state distribution

N X

½at ðj, kÞ  aijk  bk ðxt + 1 Þ

ð7Þ

i=1

2  t  T 1  j, k  N pi = PðiÞ

ð1Þ

The log is applied to the given function at + 1 (j, k)

where P(i) is the probability of being i as the initial state. log at + 1 ðj, kÞ =

2.

The transition probability distribution of k given i and j

N X

log½at ðj, kÞ  aijk  bk ðxt + 1 Þ

at + 1 ðj, kÞ = exp aijk = P(yt jyt1 , yt2 ) N X

ð8Þ

i=1

ð2Þ

N X

log at ðj, kÞ

i=1

ð9Þ

+ log aijk + log bk ðxt + 1 Þ aijk = 1,

1  i  N, 1  j  N

ð3Þ

k1

where N is the number of states in the model and yt is the hidden state at time t.

PðXjlÞ =

N X

aT ð i Þ

ð10Þ

i=1

Likewise, the backward algorithm bt (i, j) can be expressed in logarithm form as

Sung-Hyun et al.

5

bt ði, jÞ = P(xt + 1 xt + 2 xt + 3    xT jyt , l), 2  t  T  1, 1  j, k  N log bt ði, jÞ =

N X

  log bt + 1 ðjÞ  aijk  bk ðxt + 1 Þ ,

i=1

ð11Þ

ð12Þ

   jt ðj, kÞ = exp 1  i max  N log dt1 ði, jÞ + log aijk , 3  t  T, 1  j, k  N

And we terminate by computing the most probable final state yT

2  t  T  1, 1  j, k  N bt ði, jÞ = exp

N X

yT = 1 argmax i, j  N ½dt ðj, kÞ

log bt + 1 ðjÞ + log aijk ðxt + 1 Þ + log bk

i=1

ð13Þ

Proposed LoV algorithm The problem of decoding can be addressed using the Viterbi algorithm. However, time complexity seems to be increased. Therefore, LoV algorithm is on the count. On given model and observation sequence X = x1 , x2 , x3 , . . . , xT , we have to choose a corresponding state sequence Y = y1 , y2 , y3 , . . . , yT , which is optimal, and can best explains the observation to infer best state sequence. This optimality criterion maximizes the expected number of correct individual state sequence. The most likely state sequence is using probability of partial alignment ending at transition (j, k) at times (t – 1, t) To implement this, we define the variable dt ðj, kÞ = P(y1 , y2 , . . . , yt2 , yt1 = j, yt = k, x1 x2 x3    xt , jl), 2  t  T, 1  j, k  N

ð14Þ

dt (j, k) is the highest probability along a single path, at time t, which accounts for first t observations and ends in state k, by induction we have   dt ðj, kÞ = 1  i max  N dt1 ði, jÞ  aijk bk (xt ), 3  t  T1  j, k  N

ð15Þ

ð20Þ

ð21Þ

We can then compute the most probable sequence of states using backtracking or traceback approach.   yt = jt + 1 yt + 1 t = T  1, T  2, . . . , 1

ð22Þ

Maximum likelihood estimation The Baum–Welch algorithm is used37 to compute the learning or likelihood of training observation of the model. In second-order HMM, the forward-backward algorithm determines the expected number of state transitions and emissions based on a current model in Baum–Welch-based algorithm. These parameters will be used to re-estimate the model parameters using estimation formula for each iteration. This interaction continues until a convergence to stationary point occurs. The probability of being in state Yj at time t and Yk at t + 1 given the model l, then from definition of forward-backward algorithm is given as at ðjÞ  aijk  bk (xt + 1 )bt + 1 ðjÞ ð23Þ PN j=1 k at ðjÞ  aijk  bk (xt + 1 )bt + 1 ðjÞ

jt ðj, kÞ = PN

The probability of being in state Xj at time t, given the observation sequence Yj and the model l is gtðjÞ =

N X

jt ðj, k Þ

ð24Þ

k=1

The maximum likelihood of model parameter can be re-estimate as

Log of dt (j, k) be   log dt ðj, kÞ = 1  i max  N log dt1 ði, jÞ  aijk bk (xt ),

ð16Þ 3  t  T, 1  j, k  N   dt ðj, kÞ = exp 1  i max  N log dt1 ði, jÞ + log aijk + log bk ð17Þ   dt ði, jÞ = exp log pi + log aij + log bk (xt  1  i, j  N ð18Þ j3 ði, jÞ = 0   dt ðj, kÞ = exp 1  i max  N log dt1 ði, jÞ + log aijk + log bk ðxt Þ ,

p  i = gi ðiÞ PT1 = 1 jt ði, jÞ aij = Pt T1 t = 1 g t ði Þ PT2 1 jt ðj, kÞ aijk = Pt = T2 t = 1 gt ðjÞ PT gt ðkÞdt yt ðkÞ k = t = b P1 T t = 1 gt ðjÞ

ð25Þ ð26Þ

ð27Þ

ð28Þ

3  t  T, 1  j, k  N

ð19Þ

We also keep track of the most likely previous state for each possible state that we end up in probable path

Evaluation and results In our experiments, the state is considered to be the activity and observation data are the sensor data. We

6

International Journal of Distributed Sensor Networks

LoV algorithm 1. 2. 3. 4. 5.

6.

7. 8.

Input: Y is sets of Activity X is sets of Sensors (Observation) Initialize: Initial distribution p Transition distribution A Emission distribution B For yt2 Y do pi bi ðy1 Þ Calculate the initial probability a1 ðiÞ end for do for yt2 Y do calculate the forward P probability at time t + 1, exp Ni= 1 logat ðj, k Þ at + 1 ðj, k Þ + logaijk + logbk ðxt + 1 Þ where, 2  t  T, 1  j, k  N end for end for for calculate the backward probability at t from the given value at t + 1P exp Ni= 1 logbt + 1 ðjÞ bt ðI, jÞ + logaijk + logbk ðxt + 1 Þ Where, 2  t  T  1, 1  j, k  N end for for t = 2 to N do for yt2 Y do calculate the state  sequence,  exp max dt ðj, k Þ 1  i  N logdt1 ði, jÞ + logaijk + logbk (xt  calculate the most path,  probable   exp max jt ðj, k Þ 1  i  N logdt1 ði, jÞ + logaijk

9. end for 10. end for terminate by computing the most probable final state argmax yT 1  i, j  N ½dt ðj, k Þ 11. For t = N to 2 do backtracking  jt + 1 yt + 1 yt 12. end for 13. for yt2 Y do 14. for t = 1 to N do 15. max = 0 Probability of the forward-backward algorithm jt ðj, k Þ

t ðjÞ:aijk :bk (xt + 1 )bt + 1 ðjÞ PN aP N j=1

k

at ðjÞ:aijk :bk (xt + 1 )bt + 1 ðjÞ

16. end for g tðjÞ

N P

jt ðj, kÞ

k=1

17. end for 18. update: we can obtain the parameters of hidden Markov model gi ðiÞ p=p i PT1 j ði, jÞ =1 t PtT1 aij gt ðiÞ t = 1P T2 j ðj, k Þ Pt =T21 t A = aijk t=1

B = bk 19. end for 20. end for 21. end

g t ðj Þ

PT g ðk Þdt yt ðkÞ t=1 t PT t=1

gt ðjÞ

evaluate the proposed recognition model using three publicly available real-world datasets. The results of our approach are compared with NB, HMM, CRF, and HSMM. Furthermore, we put in front the activity level performance analysis through the confusion matrices. Leave one-day cross concept is considered for evaluation in which a single day data are used for testing, while the remaining for training. The process is repeated for all days and an average is calculated. The parameter taken for evaluation is F-score and accuracy. Finally, the time complexity is calculated as per experimental processing.

Evaluation terminology We define evaluation terminologies as precision, recall, F-score, and accuracy. F-score is the harmonic mean of precision and recall. These terminologies are measures using true positive (TP), false positive (FP), and false negative (FN) through confusion matrix F  score =

2 3 Precision 3 Recall Precision + Recall

Precision = Recall =

TP 3 100 TP + FP

TP 3 100 TP + FN

Accuracy =

TP 3 100 N

N denotes the number of states. The F-score and accuracy values near to 1 highlight the best performance of proposed method otherwise indicates downfall of performance.

Datasets Many smart home datasets are publicly available; among them, the Kasteren datasets and WSU CASAS are very famous datasets for evaluation of any smart home projects for activity recognition. Overview of Kasteren datasets and WSU CASAS is shown in Table 1. Kasteren datasets contain three smart homes datasets (Kasteren-House-A, Kasteren-House-B, and Kasteren-House-C), but we chose only two of them. Kyoto1 is one of the many WSU CASAS datasets we selected. The Kasteren dataset is collected from performance performed by one resident. However, WSU CASAS comprises of multiple residents, which means WSU CASAS datasets provide high inter-subject variations.

Sung-Hyun et al.

7

Table 1. Dataset information used in the evaluation of proposed method.

Rooms Sensors Activities Residents Period Instances Activities performed

Kasteren-House-A

Kasteren-House-C

Kyoto1

3 14 10 1 25 days 245 Leaving, toileting, showering, sleeping, breakfast, dinner, drinking

6 21 16 1 20 days 257 Eating, brushing teeth, shaving, medication, breakfast, dinner, relaxing, dishwashing, toileting, drinking, snacks, laundering, lunch, sleeping, leave house

4 38 5 20 15 days 120 Calling, washing hand, eating, cooking, cleaning

Table 2. Confusion matrix for Kasteren-House-A.

Leaving Toileting Showering Sleeping Breakfast Dinner Drinking Precision

Leaving

Toileting

Showering

Sleeping

Breakfast

Dinner

Drinking

Recall

82.5 0 0 0 0 0 6.5 92.70

0 88 15 0 0 0 0 85.44

0 12 85 0 0 0 1.2 86.56

0 0 0 91.3 0 0 0 100.00

0 0 0 1.3 79.5 13 6.9 78.95

0 0 0 3.2 15.1 77 4.4 77.23

17.5 0 0 4.2 5.4 10 81 68.59

82.50 88.00 85.00 91.30 79.50 77.00 81.00

Table 3. Confusion matrix for Kasteren-House-C.

Eating (1) Brushing teeth (2) Shaving (3) Medication (4) Breakfast (5) Dinner (6) Relaxing (7) Dishwashing (8) Toileting (9) Drinking (10) Snacks (11) Laundering (12) Lunch (13) Sleeping (14) Leave house (15) Others (16) Precision

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Recall

72 0 0 0 6 4.3 0 0 0 6.2 2.3 0 9 0 0 2.2 70.59

0 78 3 0 0 0 0 2.3 7.2 0 0 0 0 0 0 1.5 84.78

0 5 85 0 0 0 0 0 0 0 0 7.2 4.6 2 0 5.1 78.05

0 0 0 81.6 0 1 0 0 0 4.3 1.3 0 4 0 0 3 85.71

8 0 0 0 67 5.2 0 0 0 2.1 2.5 0 0 0 0 6 73.79

5 0 0 0 5.5 79 0 0 0 1.5 0 0 0.5 0 0 4.2 82.55

0 0 0 1.9 0 0 92 0 0 0 0 0 0 2 0 3.3 92.74

0 0 0 0 0 2.8 0 81 0 0 1.6 0 0 0 0 1.9 92.78

0 7 1.5 0 0 0 0 5.6 78 0 0 0 0 0 0 3.2 81.85

0 0 0 0 4.5 0 0 3.6 0 77.5 0 0 4.9 0 0 3.7 82.27

5 0 0 0 3 0 0 0 0 1.6 78 0 2 0 0 0 87.05

0 0 1.2 0 0 0 0 0 0 0 0 81.8 0 0 0 5.9 92.01

7 0 0 0 5 3 0 0 0 0 5 0 69 1 0 2 75.00

0 0 0 10 0 0 2.7 0 0 0 0 0 0 92 0 6.5 82.73

0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 3 96.94

3 10 9.3 6.5 9 4.7 5.3 7.5 14.8 6.8 9.3 11 6 3 5 48.5 30.37

72.00 78.00 85.00 81.60 67.00 79.00 92.00 81.00 78.00 77.50 78.00 81.80 69.00 92.00 95.00 48.50

Activity recognition analysis Table 2 presents confusion matrix of Kasteren-HouseA activities. The proposed method recognizes activities with an average accuracy of more than 85%. Breakfast and dinner activities have 13% of their instances shared among each other because they are being performed at the same location (kitchen). Likewise, toileting and

showering share 12% of instances as they share the same location (restroom). Drinking generates most of the confusion with other activity as drinking can be done while taking breakfast, dinner, leaving activities concurrently. Sleeping activity has highest accuracy of 91.3% as none other activity is done while sleeping. Table 3 shows confusion matrix of activities in Kasteren-House-C dataset. All the activities have been

8

International Journal of Distributed Sensor Networks

Table 4. Confusion matrix for Kyoto1.

Calling Washing hand Eating Cooking Cleaning Precision

Calling

Washing hand

Eating

Cooking

Cleaning

Recall

95 0 6.5 0 0 93.6

0 96 0 3.7 2 94.395

2 0 89.4 2.3 0 95.41

3 1.6 4.1 94 1 90.6

0 2.4 0 0 97 97.5855

95 96 89.4 94 97

Table 5. Performance evaluation of smart home dataset with proposed method. Datasets

Method

Precision

Recall

F-score

Accuracy

Kasteren-House-A

NB CRF HMM HSMM Proposed method NB CRF HMM HSMM Proposed method NB CRF HMM HSMM Proposed method

76.61 79.52 81.23 83.31 84.21 78.45 86.22 80.23 89.65 94.33 54.36 65.63 66.25 77.36 80.58

69.11 72.23 68.2 81.82 84.21 73.56 87.18 90.55 95 94.28 53.55 68.96 64.35 80.23 78.46

72.67 75.70 74.15 82.56 84.21 75.93 86.70 85.08 92.25 94.30 53.95 67.25 65.29 78.77 79.51

71.56 75.12 73.25 83.89 83.47 74.88 86.12 85.01 90.26 94.28 53.12 66.92 65.01 77.32 78.46

Kyoto1

Kasteren-House-C

NB: Naı¨ve Bayes; CRF: conditional random field; HMM: hidden Markov model; HSMM: hidden semi-Markov model.

recognized with higher accuracy. Some of the activities like dinner have 79% activity instance recognition accuracy. It has 5%, 5.5%, 1.5%, 0.5% and 4.2% error activities recognized as eating, breakfast, dishwashing, lunch, and other, respectively. 85% of shaving activity is recognized correctly but 5%, 7.2%, 4.6%, and 5.1% activity is confused with brushing teeth, toileting, doing laundry, and other. There occurred a confusion between different activities it is because of sharing same location and platform, for example, any eating activity like lunch, dinner, and breakfast, commonly occurs in the kitchen. Also share common sensor like gas sensor, heat sensor, and so on. The Kasteren-House-C has large confusion activities; therefore, F-score and accuracy are lower than Kasteren-House-A and Kyoto1. Other similar activities such as drinking, eating, lunch, snack exhibit a similar trend of sharing error and making confusion among each other. Kyoto1 dataset confusion matrix is shown in Table 4 whose recognition accuracy is more compared to other two datasets. Calling activity has 95% correctly recognized but still 6.5% of eating error activity because the call can be taken while doing both activities at the same time. Washing hand has 96%, while 3.7% and 2% of the cooking and cleaning confusion since washing hand can be done next or in between

cleaning and cooking. The cooking activity is recognized 94% correctly but still creates confusion with 3%, 1.6%, 4.1%, and 1% of calling, washing hand, eating, and cleaning, respectively. The overall accuracy of the proposed method on Kyoto1 has 94.3% to recognize all of the activities. In eating activity, 89.4% instances are correctly identified, while 4%, 2% of instances are erroneously identified as wash hands, calling and 2.3% for cooking. Although accuracy is high, to find actual recognition distribution, a large number of the dataset could be needed. Kasteren and CASAS data have the least number of dataset and instances, therefore, actual distribution is easy to find out. On analysis, our proposed method to the available dataset proves to be more reliable and effective than other methods.

Activity recognition comparison Performance evaluation of the Kasteren-House-A dataset on our proposed method is shown in Table 5. Activities like breakfast, dinner, and drinking again like toileting and showering are performed in the same location that means location sensor is shared in that location. Thus, there appeared less inter-class variation. Precision, recall, F-score, and accuracy in Kasteren-

Sung-Hyun et al.

9

Figure 3. The F-score comparison of the proposed method with the existing approaches in the activity level performance: (a) Kasteren-House-A, (b) Kyoto 1, and (c) Kasteren-House-C.

House-A are obtained as 84.21%, 83.47%, 0.84%, and 85.47% respectively, from our proposed method. The resultant values are satisfactory than the existing approaches NB, CRF, and HMM, which means location sensor is shared in that location. The resultant values are satisfactory than the existing approaches NB, CRF, HMM, and HSMM. In Kasteren-House-C precision, recall F-score and accuracy of the proposed method are 80.58%, 78.46%, 0.79%, and 78.46%, respectively. The precision, recall, F-score, and accuracy of Kasteren-House-C is less than that of inter-class variation but these values are high enough compare with other approaches. Table 3 shows the confusion matrix of the Kasteren-House-C. Kyoto1 is the CASAS datasets which have high intra-class and inter-subject variations and activity class is discriminative. Kyoto1 shows the excellent performance of our proposed method securing F-score of 0.94 and accuracy of 94.28%. The precision and recall of Kyoto1 are 94.33% and 94.28%, respectively, and remain higher on our proposed method compare to other four approaches.

All the activity recognition comparison with other is clearly shown in Table 5.

Activity level performance comparison The activity level performance of proposed method is compared with existing method NB. HMM, CRF, and HSMM are given in Figure 3. We measured the F-score of each activity on different datasets Kasteren-HouseA, Kasteren-House-C, and Kyoto1 individually. Individual activity performance also shows better accuracy. Kasteren-House-A shown in Figure 3(a) has the accuracy of 85.65%, while Kasteren-House-C has 80.87% of accuracy as displayed in Figure 3(c) and Kyoto1 achieve an accuracy of about 95.67% as in Figure 3(b). We can still earn high accuracy but imbalance in the dataset, where the activities are done more frequently. The used dataset also has problem of overlapping with a minimum number of activity instances and sharing of a different sensor. However, the F-score solves this kind of drawbacks to produce reliable

10

International Journal of Distributed Sensor Networks Acknowledgements The work reported in this article was conducted during the sabbatical year of Kwangwoon University in 2016.

Declaration of conflicting interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD Figure 4. The time complexity comparison of proposed method with the HMM.

Keshav Thapa

https://orcid.org/0000-0003-1329-2826

References dataset. The proposed method has highest F-score of 0.84, 0.79, and 0.94 in Kasteren-House-A, KasterenHouse-C, and Kyoto1, respectively, compare to existing methods. This shows that our method can also be least affected by the problem of intra-class and interclass instances.

Time complexity Time complexity appears on calculating the joint probability of hidden state sequences with observed series of activity. The regular time complexity of HMM is N2L for N states and NN state transition probabilities, 2N output probabilities of an output sequence of length L. The proposed method has the time complexity of (LN2 log N) which means the time complexity is reduced by log N. The effectiveness of the reduced time complexity is shown in Figure 4.

Conclusion We have presented LoV algorithm using second-order HMM to recognize human activities in the smart home. Proposed method takes two previous activities and the current activity observation for recognition. Three real datasets are used to evaluate the recognition performance of the proposed method with other recognition methods. The proposed method shows F-score of 0.84, 0.79, and 0.94 in Kasteren-House-A, Kasteren-HouseC, and Kyoto1, respectively, compare to other existing methods. In addition, it reduced time complexity by log N as compared to others. Therefore, accuracy and time complexity of recognition are better than other approaches. In future, we can increase the accuracy much more applying machine learning schemes for learning model parameters.

1. Tapia EM, Intille S and Larson K. Activity recognition in the home using simple and ubiquitous sensors. In: Ferscha A and Mattern F. (eds) Pervasive Computing. Berlin: Springer, 2004, pp.158–175. 2. Mynatt ED, Essa I and Rogers W. Increasing the opportunities for aging in place. In: Proceedings of the ACM conference on universal usability, Arlington, VA, 16–17 November 2000, pp.65–71. New York: ACM. 3. Van Kasteren TLM, Englebienne G and Krose BJA. An activity monitoring system for elderly care using generative and discriminative models. Pers Ubiquit Comput 2010; 14(6): 489–498. 4. Hong YJ, Kim IJ, Ahn SC, et al. Mobile health monitoring system based on activity recognition using accelerometer. Simul Model Pract Theory 2010; 18(4): 446–455. 5. Nergui M, Yoshida Y and Yu W. Human gait behavior interpretation by a mobile home HealthCare robot. J Mech Med Biol 2012; 12: 1240021. 6. Lin YJ, Latchman HA, Lee M, et al. A power line communication network infrastructure for the smart home. IEEE Wirel Commun 2002; 9(6): 104–111. 7. Duong T, Phung D, Bui H, et al. Efficient duration and hierarchical modeling for human activity recognition. Artif Intell 2009; 173(7–8): 830–856. 8. Ward JA, Lukowicz P, Troster G, et al. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE T Pattern Anal 2006; 28(10): 1553–1567. 9. Aggarwal JK and Cai Q. Human motion analysis: a review. Comput Vis Image Und 1999; 73(3): 428–440. 10. Abowd GD and Mynatt ED. Charting past, present and future research in ubiquitous computing. ACM T Comput: Hum Int 2000; 7(1): 29–58. 11. Vail DL, Veloso MM and Lafferty JD. Conditional random fields for activity recognition. In: Proceedings of the 6th international joint conference on autonomous agents and multiagent systems, Honolulu, HI, 14–18 May 2007, pp.1–8. New York: ACM.

Sung-Hyun et al. 12. Gong S and Xiang T. Recognition of group activities using dynamic probabilistic networks. In: Proceedings of the ninth IEEE international conference on computer vision, Nice, 13–16 October 2003, pp.472–479. New York: IEEE. 13. Oliver N, Horvitz E and Garg A. Layered representations for human activity recognition. In: Proceedings of the fourth IEEE international conference on multimodal interfaces, Pittsburgh, PA, 16 October 2002, pp.3–8. New York: IEEE. 14. Rabiner L. A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 1989; 77(2): 257–286. 15. Duong TV, Bui HHD, Phung DQ, et al. Activity recognition and abnormality detection with the switching hidden semi-Markov model. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, San Diego, CA, 20–25 June 2005, pp.838– 845. New York: IEEE. 16. Li Z, Wei Z, Yue Y, et al. An Adaptive hidden Markov model for activity recognition based on a wearable multisensor device. J Med Syst 2015; 39: 57. 17. Fahad LG and Rajaranjan M. Integration of discriminative and generative models for activity recognition in smart homes. Appl Soft Comput 2015; 37: 992–1001. 18. Fatima Fahim M, Lee YK and Lee S. Analysis and effects of smart home dataset characteristics for daily life activity recognition. J Supercomput 2013; 66(2): 760–780. 19. CASAS Dataset, http://casas.wsu.edu/datasets/ 20. Bell C, Fausset C, Farmer S, et al. Examining social media use among older adults. In: Proceedings of the 24th ACM conference on hypertext and social media, Paris, 1–3 May 2013. New York: ACM. 21. Van Kasteren TLM. Datasets for activity recognition, https://sites.google.com/site/tim0306/datasets 22. Alam MR, Reaz MBI and Ali MAM. A review of smart homes—past, present, and future. IEEE T Syst Man Cy C 2012; 42(6): 1190–1203. 23. Aipperspach R, Cohen E and Canny J. Modeling human behavior from simple sensors in the home. In: Fishkin KP, Schiele B, Nixon P, et al. (eds) Pervasive computing. Berlin: Springer, 2006, pp.337–348. 24. Chernbumroong S, Cang S, Atkins A, et al. Elderly activities recognition and classification for applications in assisted living. Expert Syst Appl 2013; 40(5): 1662–1674. 25. Fleury MV and Noury N. SVM-based multimodal classification of activities of daily living in health smart homes: Sensors, algorithms, and first experimental results. IEEE T Inf Technol B 2010; 14(2): 274–283. 26. Munguia Tapia E. Using machine learning for real-time activity recognition and estimation of energy expenditure.

11

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

39.

PhD Dissertation, Department of Architecture, Massachusetts Institute of Technology, Cambridge, MA, 2008. Namand Y and Park J. Child activity recognition based on cooperative fusion model of a triaxial accelerometer and a barometric pressure sensor. IEEE J Biomed Health 2013; 17(2): 420–426. Parkka J, Ermes M, Korpipaa P, et al. Activity classification using realistic data from wearable sensors. IEEE T Inf Technol B 2006; 10(1): 119–128. Organero MM and Ruiz-Blazquez R. Time-elastic generative model for acceleration time series in human activity recognition. Sensors 2017; 17(2): E319. Bourobou STM and Yoo Y. User activity recognition in smart homes using pattern clustering applied to temporal ANN algorithm. Sensors 2015; 15: 11953–11971. Larochelle H and Bengio Y. Classification using discriminative restricted Boltzmann machines. In: Proceedings of the 25th international conference on machine learning, Helsinki, 5–9 July 2008, pp.536–543. New York: ACM. Hinton GE. Training products of experts by minimizing contrastive divergence. Neural Comput 2002; 14: 1771–1800. Xing Z, Pei J and Keogh E. A brief survey on sequence classification. ACM SIGKDD Explor Newsl 2010; 12(1): 40–48. Van Kasteren TLM, Noulas A, Englebienne G, et al. Accurate activity recognition in a home setting. In: Proceedings of the 10th international conference on Ubiquitous computing, UbiComp 2008, pp.1–9. Ward JA, Lukowicz P, Troster G, et al. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE T Pattern Anal 2006; 28(10): 1553–1567. Uddin Jalal MZ and Kim TS. Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE T Consum Electr 2012; 58(3): 863–871. Jalal A, Kamal S and Kim D. Human depth sensorsbased activity recognition using spatiotemporal features and hidden Markov model for smart environments. J Comput Netw Commun 2016; 2016: 8087545. Humayun Kabir M, Robiul Hoque M, Keshav T, et al. Two-layer hidden Markov model for human activity recognition in home environments. Int J Distrib Sens N. Epub ahead of print 12 January 2016. DOI: 10.1155/ 2016/4560365. Ordo´n˜ez FJ and Roggen D. Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 2016; 16: 115.