Using Hidden Markov Models for Accelerometer-Based Biometric Gait ...

Using Hidden Markov Models for Accelerometer-Based Biometric Gait Recognition Claudia Nickel, Christoph Busch

Sathyanarayanan Rangarajan

University of Applied Sciences Darmstadt Darmstadt, Germany [email protected]

Technical University of Denmark Kopenhagen, Danmark [email protected]

Abstract—Biometric gait recognition based on accelerometer data is still a new field of research. It has the merit of offering an unobtrusive and hence user-friendly method for authentication on mobile phones. Most publications in this area are based on extracting cycles (two steps) from the gait data which are later used as features in the authentication process. In this paper the application of Hidden Markov Models is proposed instead. These have already been successfully implemented in speaker recognition systems. The advantage is that no error-prone cycle extraction has to be performed, but the accelerometer data can be directly used to construct the model and thus form the basis for successful recognition. Testing this method with accelerometer data of 48 subjects recorded using a commercial of the shelve mobile phone a false non match rate (FNMR) of 10.42% at a false match rate (FMR) of 10.29% was obtained. This is half of the error rate obtained when applying an advanced cycle extraction method to the same data set in previous work. Keywords-biometrics; gait recognition; hidden markov models; accelerometers; authentication on mobile devices

I.

INTRODUCTION

Nowadays, mobile phones are used for multiple purposes like internet-access, e-mails, calendar, mobile banking and so on. This results in a large amount of personal information stored on these devices such as addresses, appointments and business details in addition to phone numbers. Inspite of this, most mobile phones are not sufficiently protected. A survey by Breitinger and Nickel [1] reported that in only 13% of the cases authentication is required after a stand-by-phase. As a result of this, all data is directly accessible by every person procuring physical access to that device. The same survey also showed that the reason for this is that the authentication methods offered by mobile phones are not well accepted by users. Most authentication methods require the user to remember either a PIN or password which has to be entered for authentication. As this implies extra work for the users, many of them are not willing to set the authentication settings of their phones in a way that frequent authentication is required. As mobile phones in addition have the possibility to access servers using singlesign-on for transitional access to further services, this situation results in significant security risks.

Manuel Möbius atip GmbH Frankfurt, Germany [email protected]

This situation can be improved by offering an unobtrusive authentication method which does not require extra-work for the user. Accelerometer-based gait recognition is such a method. Accelerometers are commonly integrated into many mobile phones and can be used to record the gait of the user. The feasibility of using gait as a biometric feature has already been shown for different application scenarios using different sensors. The research can be categorized into three general groups: machine vision based, floor sensor based and wearable sensor based gait recognition [2]. Vision based gait recognition, where the gait is recorded using video cameras, can be used for surveillance scenarios. Major research in this area has been done by Nixon [3]. Middleton et al. [4] proposed the use of floor sensors for acquiring the gait data which offers a suitable authentication method for entrance control. Gait recognition using wearable sensors was first proposed by Ailisto et al. in 2005 [5] and further developed by Gafurov [6]. They used high-quality dedicated accelerometers which were placed on the hip, arm or ankle to record the acceleration while the subjects were walking. Only recently researchers started to use mobile phones to record the accelerometer data [7]–[10]. From the recorded data a periodic repetition can be seen after two steps. This is called a cycle and is used as a basis for computing the feature vector in many former proposed accelerometer-based gait authentication methods. Irregular cycles and unclear boundaries between two cycles (see section III for an example) are possible reasons for failure of cycle extraction methods and increase the error rates in these methods. To overcome the limitations of the previous work, the method proposed in this paper is based on Hidden Markov Models [11] that are trained for a specific subject. The clear advantage is that no error-prone cycle extraction has to be performed. The proposed method was tested with data collected using a mobile phone. Although this data has a lower sampling rate than the testing data recorded using dedicated accelerometers in [5], [6] (around 40 samples per second instead of around 100), we can achieve suitable error rates. These results show that accelerometer-based gait recognition is a reasonable supplement to existing authentication methods on mobile phones. The main advantage is the unobtrusiveness.

While the user is walking with the phone he is directly authenticated without any extra input. To allow an authentication in the case where the user is not walking, this authentication method needs to be combined with some knowledge-based authentication method like PIN or other active biometrics like fingerprint. This paper is organized as follows: Section II explains how the data was collected. In section III, examples of the collected data are shown and the principle of feature extraction used in previous methods is sketched. The following section IV, gives an overview of Hidden Markov Models and in section V the applied Hidden Markov Model Toolkit (HTK) is described. The experiments conducted are discussed along with their corresponding results in section VI followed by conclusions and future work in VII. II.

DATA COLLECTION

The data used in this article was collected using a standard G1 mobile phone [12] which contains piezo-resistive accelerometers for measuring acceleration in three directions. The G1 uses the android platform and a software was written for this platform to access the accelerometer and write the data from the sensor to a text file. The sampling occurs at nonequidistant intervals with a mean sampling rate of 42 samples per second. This was obtained for each of the three directions, namely x, y and z (see Fig. 1). Due to the limitations of the android SDK it is not possible to record data at a fixed sample rate. This is because, a new acceleration value is recorded only when there is a change in acceleration thereby resulting in different time intervals between successive recordings. While recording the gait data the phone was placed in a pouch, attached to the belt of the subject on the right-hand side of the hip. Further, the phone was positioned horizontally in such a way that the screen points to the body and the upper part of the phone points towards the walking direction (see Fig. 1). The walking distance for each subject was about 37 meters down the hall on a flat carpet surface (see Fig. 2). At the end of the hall-way the subjects had to wait for 2 seconds, turn around, wait again and then walk back the same distance. In total, complete data sets of 48 subjects participating in the collection were recorded (see table I for age and gender

Figure 1. Phone attached to the subject and the three axes in which acceleration is measured.

Figure 2. Photograph of the walking setting.

TABLE I.

AGE AND GENDER DISTRIBUTION OF DATA SUBJECTS

30

unknown

male

1

1

25

9

2

female

0

5

4

0

1

total

1

6

29

9

3

distribution). For each of them two sessions were captured on two different days (mean time lag was two weeks), wearing normal pair of shoes. The subjects were told to walk as normal as possible, which means that different subjects can walk at different speeds. III.

DATA DESCRIPTION

Fig. 3 shows an example of the recorded data. The upper part shows the vertical acceleration which does also contain the gravity. The y-acceleration corresponds to the forwardbackward movement and the z-acceleration (at the bottom) shows the lateral acceleration. It can be seen that the first part of the recorded data also contains the phase when the phone is placed in the hip-pouch and the time interval the subject waits before starting the walk. The two larger sections of high acceleration contain the data when the subject was walking. The data inbetween shows the standing – turning around – standing at the end of the hall-way. Only the real walking data (called walk) is used for further processing. Two walks can be extracted from one data set (indicated by the dashed vertical lines in Fig. 3). Each subject took part in the data collection twice resulting in four walks for each subject. Further, the length of each walk differs depending on the walking speed of the subject. The mean duration of one walk was 26.5 second (with standard deviation of 2.8 seconds) and on average 1108 samples were recorded per walk (with standard deviation of 116). Former reported methods (e.g. [13]) would search for extrema in these walks to extract the cycles which are then used to calculate the feature vectors. In the left image of Fig. 4 these cycles can be clearly seen, going from one minimum to the next one and containing two steps. This method clearly fails for the data on the right hand part of Fig. 4 which is highly irregular and does not contain clear minima. The method proposed in this paper overcomes this problem by using Hidden Markov Models instead of cycle extraction as explained in the next section.

Figure 3. Recorded accelerometer data.

IV.

HIDDEN MARKOV MODELS

Hidden Markov Models (HMM) have been frequently used for modeling time series data. They were introduced during the mid-60’s by Baum et al. [14]. In a Markov model the states, corresponding to a physical event, of the model are visible to the observer. On the other hand, in HMMs, only the output of the model is visible and the states are not observable, in other words are hidden [11]. Lately, HMMs have had a wide range of applications due to the fact that they are easy to build and manipulate. The most prominent of these applications is speech recognition, where HMMs are the backbone of most commercial speech recognition systems. An excellent tutorial for the use of HMMs in speech recognition systems is provided in [15]. Apart from speech recognition, HMMs are also used in a number of pattern recognition applications such as sign language recognition [16] and handwriting recognition [17]. As mentioned earlier, the objective of this paper is to apply HMMs to another challenge, namely biometric gait recognition. Hidden Markov Models can be defined using several parameters. The key parameters are: •

Number of States: A finite number of states of the model.

•

Transition Probability Matrix: At each time instance, a new state is reached based on a transition probability distribution that depends on the previous state.

•

Observation Function: After each transition, an observation symbol is produced as output according to a probability distribution associated to the current state.

For building and training HMMs, the Hidden Markov Model Toolkit (HTK) was used, which is discussed in the next section. V.

HMM-TOOLKIT

Hidden Markov Model Toolkit (HTK) is a software toolkit for building and manipulating Hidden Markov Models [18]. It

Figure 4. Example of good and bad gait data.

consists of a set of tools and library modules which facilitate the training and testing of HMMs. Though HTK was originally developed for the purpose of speech recognition at the Machine Intelligence Laboratory of the Cambridge University Engineering Department (CUED), it has been widely used in applications such as speech synthesis [19], handwriting recognition [20] etc. Full details about HTK and its tools are provided in [18] and details about the design and philosophy of HTK can be found in [21]. An overview of the HTK tools used in this paper is shown in Fig. 5 and can be best understood by going through the following steps: A. Data Preparation In order to build HMMs, a set of gait data files and their associated classifications are needed. In our experiments, we classified subjects into two groups namely the enrollee and the rest-of-all group. As mentioned earlier, the gait (consisting of acceleration values in x-, y- and z-direction) of each subject was stored in text files. These cannot be used directly by HTK and hence were converted to native HTK format before they can be used for training. This was done by converting the data into a matrix with 26 columns which contains all x-values in the first rows, followed by all y-values and all z-values. It was payed attention that no row contains data of two different directions. The tool HLIST was then used to cross verify the results of this conversion. B. Training Phase In this phase, the topology required for individual HMMs is first defined with the help of a prototype definition. In our experiments, we used a prototype consisting of five to eight

correct classifications for each subject. The results produced by HRESULTS and HVITE were used in the calculation of false match rate and false non match rate of the system which is described in more detail in the following section. VI.

ANALYSIS AND RESULTS

The data used for the experiments consists of four walks for each of the 48 subjects as stated in Section II and III. These walks were interpolated with a fixed sample rate of 200 samples per second and further divided into parts of 3 seconds each which resulted in a total of 1651 walk sections. This configuration has been empirically determined to deliver best results. Depending on the speed of a subject’s walk this resulted in 28 to 37 data sets (walk sections) per subject. In order to make sure that all the subjects have an equal number of walk sections, 28 walk sections per subject (total of 1344 walk section for 48 subjects) were used in our experiments. The accelerations in x-, y- and z-direction were transformed as described in subsection V-A.

Figure 5. Processing stages.

states (depending on the experiment) where the first and the last state are non-emitting i.e. they do not have an observation function. Further, a single gaussian observation function with diagonal matrices was used for each state [18]. These functions are entirely defined by mean and variance vectors for which a vector size of 26 was used. These vectors along with the transition probability matrix were initialized with arbitrary values. This HMM prototype was then given as an input to the HCOMPV tool which generates the initial set of models. The actual parameters are then re-estimated using the HEREST tool, which performs embedded training using the Baum-Welch algorithm [15], [18] for the given training set. Further, the tool HHED was used to improve the recognition rate by introducing multiple gaussian mixtures into each HMM state. C. Recognition Phase For testing purposes the HVITE tool was used. HVITE uses a modified version of the Viterbi algorithm known as the token passing algorithm for recognition [18] [22]. It takes as input a dictionary which describes how each subject can be classified, a recognition network which is generated using the HPARSE tool, the HMM definitions produced by HEREST during the training phase and finally a list of walk sections for testing purpose. The output produced by HVITE is again a text file which contains a list of walk sections of the user (provided for testing) along with the class into which they fall (either enrollee or rest-of-all). D. Analysis Phase Performance evaluation of the recogniser is carried out in this phase. For this purpose, the HRESULTS tool was used. HRESULTS compares the labels produced by HVITE with the correct reference labels and returns the percentage of

The goal of the analysis is to know how often the enrolled user is not recognized by his phone and how often a foreign subject is recognized as the enrolled user resulting in unauthorized access. Therefore we did 48 tests, in each test a different subject was chosen as the enrollee. This subject should be recognized by the system as the authorized subject and the HMM needs to be trained accordingly. All the other subjects should be rejected (identified as belonging to the restof-all group) by the system. As mentioned earlier, for each of these two groups one left-right-HMM was created. Different test using five to eight states and four to six gaussian mixtures per emitting state were done. For training these two HMMs, the first 20 walk sections of the enrollee and all 840 walk sections of 30 subjects from the rest-of-all group were used. The remaining walk sections of the enrollee and the rest-of-all group were left for testing purposes. This process was repeated by picking a different subject as enrollee in each run. For each chosen enrollee this results in a different number of false matches (a subject of the rest-of-all group was identified as enrollee) and false non matches (the enrollee was not recognized as such) as output of HVITE (see section V-C). On a Intel(R) Core(TM)2 Duo CPU [email protected], 2.53Hz with 4GB RAM the test for one subject for a seven state HMM (5 mixtures) took on average 0.73 seconds (standard deviation was 0.22). The obtained results were used to calculate the overall False Match Rate (FMR) and False Non Match Rate (FNMR) as follows. In order to get realistic results with respect to our application scenario, we introduced a majority voting method for the calculation of error rates. In this method, the results obtained for each enrollee are first analyzed. Only if more than half of their walk sections is rejected, it is considered as a false reject. Similarly, we account for a false accept only when at least half of the walk sections of a subject belonging to the rest-of-all group is identified as an enrollee. Table II shows the results obtained when using different number of states (ranging from five to eight) and different numbers of mixtures per state (ranging from four to six). One

TABLE II.

RESULTS OBTAINED FOR DIFFERENT NUMBERS OF STATES AND GAUSSIAN MIXTRUES USING ALL TEST DATA.

States 5 5 5 6 6 6 7 7 7 8 8 8

Mixtures 4 5 6 4 5 6 4 5 6 4 5 6

FMR 5.76% 4.53% 4.17% 12.38% 6.37% 3.92% 12.38% 6.62% 2.70% 6.62% 3.80% 1.35%

FNMR FNMR+FMR 27.08% 32.84 16.67% 21.20 14.58% 18.75 8.33% 20.71 14.58% 20.95 12.50% 16.42 2.08% 14.46 10.42% 17.04 12.50% 15.20 22.92% 29.54 18.75% 22.55 29.17% 30.52

TABLE III. RESULTS OBTAINED WITH A 7-STATE HMM AND DIFFERENT NUMBERS OF GAUSSIAN MIXTURES WHEN USING 24 SECONDS OF TEST DATA.

No. of mixtures 4 5 6

FMR 15.68% 10.29% 4.77%

FNMR 2.08% 10.41% 12.50%

FNMR+FMR 17.76 20.70 17.27

can see that the 7-state HMM always gives best results in terms of sum of FMR and FNMR. In the previous stated results, all data which was not used for training was used for testing. Therefore, for the genuine test the majority voting is based on 24 seconds of data (8 sections of 3 seconds each) and for the impostor test the majority voting is based on 84 seconds of data (28 sections of 3 seconds each). The results shown in table III were obtained when using eight walk sections (i.e. 24 seconds) of both the enrollee and the subjects belonging to the rest-of-all group for testing. Only the results of the best performing 7-state HMM are listed.

starts. For each walk the most typical cycle (in terms of dynamic time warping distance) was used as feature vector. Distance calculation was again done using dynamic time warping. The equal error rate obtained was around 20% which underlines the quality of the here proposed method using Hidden Markov Models which halved this error rate. VII. CONCLUSIONS AND FUTURE WORK One application of accelerometer-based gait recognition is the authentication of users to their mobile phones. Nevertheless, most research papers state results which were obtained using data collected by dedicated high grade accelerometers. The proposed methods focus on extracting the cyclic repetitions which occur after two steps. When using the data collected from a standard mobile phone with a lower sampling rate, these methods do not deliver acceptable results. Inspired by research done in the area of speaker recognition we applied Hidden Markov Models to the gait data collected using a mobile phone. After interpolating the data to get a fixed sampling rate (200 samples per second) and dividing it into parts of fixed length (3 seconds) it was directly used for training and testing respectively. In contrast to cycle extraction methods these processing steps do not require specific properties of the data (e.g. distinctive minima). This makes the proposed method more suitable for the test data resulting in a good false non match rate of 10.42% at a false match rate of 10.29%. Future work will focus on further decreasing these error rates by applying different pre-processing methods and using different HMM configurations. Instead of using raw data for classification more sophisticated features will be extracted and used as input to the HMMs. As the used test data is recorded under constrained conditions, we plan to directly implement the method on the phone to confirm the results in a more realistic scenario with a larger number of subjects and a larger amount of data per subject. This implementation will also show the feasibility of the described method for authentication on mobile phones.

From these results, it can be seen that when changing the number of mixtures, the sum of FMR and FNMR stays nearly the same but the fractions of FMR and FNMR change. Hence, the optimal number of mixture components can be chosen based on the usability (i.e. lower FNMR) and security (i.e. lower FMR) requirements of the system. When comparing the FMR in table II and III one can see that the FMR is decreasing when the results are based on a longer time period (more test data). This decrease is further vizualized in figure 6. The concurrent decrease of the FNMR needs to be confirmed using a database containing a larger amount of data per subject. The 7-state HMM with 5 mixtures per emitting state gives us the equal error rate of approximately 10%. This is directly comparable to the results stated in [7]. In that paper a cycle extraction method was applied to exactly the same data set used for the tests described here. After time interpolation and filtering the data the average cycle length was determined. This was used to identify minima which correspond to cycles

Figure 6. False match rate depending on the length of the compared data (7-state HMM).

ACKNOWLEDGMENT This work was supported by CASED (www.cased.de). REFERENCES [1]

F. Breitinger and C. Nickel, “User Survey on Phone Security,” in BIOSIG 2010 - Proceedings of the Special Interest Group on Biometrics and Electronic Signatures, 2010. [2] D. Gafurov, “A survey of biometric gait recognition: Approaches, security and challenges,” in Proceedings of Annual Norwegian Computer Science Conference, 2007. [3] M. S. Nixon, J. N. Carter, J. M. Nash, P. S. Huang, D. Cunado, and S. V. Stevenage, “Automatic gait recognition,” in Biometrics - Personal Identification in Networked Society. Kluwer, 1999, pp. 231–250. [4] L. Middleton, A. A. Buss, A. I. Bazin, and M. S. Nixon, “A floor sensor system for gait recognition,” in Fourth IEEE Workshop on Automatic Identification Advanced Technologies, 2005. [5] H. J. Ailisto, M. Lindholm, J. Mäntyjärvi, E. Vildjiounaite, and S.-M. Mäkelä, “Identifying people from gait pattern with accelerometers,”Biometric Technology for Human Identification II, vol. 5779, no. 1, pp. 7–14, 2005, vTT Electronics, Finland. [6] D. Gafurov, “Performance and security analysis of gait-based user authentication,” Ph.D. dissertation, Faculty of Mathematics and Natural Sciences, University of Oslo, 2008. [7] M. O. Derawi, C. Nickel, P. Bours, and C. Busch, “Unobtrusive UserAuthentication on Mobile Phones using Biometric Gait Recognition,” in Sixth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, 2010. [8] J. Frank, S. Mannor, and D. Precup, “Activity and gait recognition with time-delay embeddings,” in AAAI Conference on Artificial Intelligence, 2010. [9] J. Kwapisz, G. Weiss, and S. Moore, “Cell phone-based biometric identification,” in Biometrics: Theory Applications and Systems (BTAS), 2010 Fourth IEEE International Conference on, 2010, pp. 1 –7. [10] S. Sprager, “A cumulant-based method for gait identification using accelerometer data with principal component analysis and support vector machine,” in Sensors, Signals, Visualization, Imaging, Simulation and Materials, 2009, pp. 94–99.

[11] L. Rabiner and B. Juang, “An introduction to hidden markov models,” ASSP Magazine, IEEE, vol. 3, no. 1, pp. 4–16, April 1986. [12] G1 ([online; last visited 2011-01-14]) http://www.google.com/intl/en us/mobile/android/hpp.html. [13] L. Rong, D. Zhiguo, Z. Jianzhong, and L. Ming, “Identification of individual walking patterns using gait acceleration,” in Bioinformatics and Biomedical Engineering, 2007. ICBBE 2007. The 1st International Conference on, July 2007, pp. 543–546. [14] L. E. Baum and T. Petrie, “Statistical inference for probabilistic functions of finite state markov chains,” in The Annals of Mathematical Statistics, vol. 37, no. 6, pp. pp. 1554–1563, 1966. [15] L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” pp. 267–296, 1990. [16] T. E. Starner and A. Pentland, “Visual recognition of american sign language using hidden markov models,” Master’s thesis, Massachusetts Institute of Technology, 1995. [17] A. Schlapbach and H. Bunke, “Using hmm based recognizers for writer Identification and verification,” in IWFHR ’04: Proceedings of the Ninth International Workshop on Frontiers in Handwriting Recognition. Washington, DC, USA: IEEE Computer Society, 2004, pp. 167–172. [18] S. J. Young, G. Evermann, M. J. F. Gales, T. Hain, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. C. Woodland, The HTK Book, version 3.4. Cambridge, UK: Cambridge University Engineering Department, 2006. [19] H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. Black, and K. Tokuda, “The HMM-based speech synthesis system (HTS) version 2.0,” in Proc. 6th ISCA Workshop on Speech Synthesis (SSW-6), Aug. 2007. [20] J. Domenech, A. H. Toselli, A. Juan, E. Vidal, and F. Casacuberta, “An off-line HTK-based OCR system for isolated handwritten lowercase letters,” in Proceedings of the IX Spanish Symposium on Pattern Recognition and Image Analysis, vol. II, 2001, pp. 49–54. [21] S. Young and S. Young, “The HTK Hidden Markov Model Toolkit: Design and Philosophy,” Entropic Cambridge Research Laboratory, Ltd, vol. 2, pp. 2–44, 1994. [22] A. Viterbi, “Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,” Information Theory, IEEE Transactions on, vol. 13, no. 2, pp. 260 – 269, apr 1967.