WiFi Fingerprint Based, Indoor, Location-Driven

WiFi Fingerprint Based, Indoor, Location-Driven Activities of Daily Living Recognition 1

Bang Wu, 1Zixiang Ma, 1Stefan Poslad, 2Yidong Li

1

IoT Lab, Queen Mary University of London, London, UK School of Computer and Information Technology, Beijing Jiaotong University (BJTU), Beijing, PR China Contact Email: [email protected]

2

Abstract— Indoor, location-driven human Activities of Daily Living (ADLs) are important indicators of people’s physical and mental wellbeing. In terms of Indoor Position Systems (IPS), much work focuses on using WiFi to determine location as these signals are more prevalent and available, and to increase the location determination accuracy, rather than to use WiFi fingerprint positioning systems to recognise location-driven ADLs (LD-ADL). This paper proposes a novel Multi-Dimensional Dynamic Time Warping (MD-DTW) based method for fusing sequences of points and Received Signal Strength Indicator (RSSI) trajectories or sequences. Since the number of wireless (Access Points (APs) being used can affect the positioning accuracy, we also integrated the use of MD-DTW with the use of Interval Overlap Degree (IOD) to improve AP selection. We designed 9 activities (340 paths) and verify the recognition performance of our method in an office type environment at Queen Mary. The results show that the recognition accuracy for 9 activities for MD-DTW is on average 66.29% higher than that using a coordinate points method alone, and on average 2.8% higher than using the RSSI feature sequence method alone when the number of APs is more than 16. Moreover, the average accuracy of MD-DTW is 99.98% for Class 1 (office space) activities, 99.81% for Class 2 (kitchen) activities and 99.95% for the combined activities, when their AP quantities are more than 49, 36, 50, respectively. Keywords— Activities During daily Life (ADL); Human Activity recognition (HAR); WiFi Fingerprint; RSSI; Wireless Access Point (AP) selection; Interval Overlap Degree (IOD).

I. INTRODUCTION Learning and recognizing human activities, e.g. ADLs, are not only very useful when building a pervasive home monitoring system but ADLs are also important indicators of both cognitive and physical well-being in healthy and ill humans. For example, location-driven ADL (LD-ADL) detection can help to identify if people are following their normal routine or not. It can identify when and where old people or young children are in high-risk locations, such as close to heated surfaces or pools of water. Specific activity detection can remind patients to follow the doctor's advice to take medicine or do some other rehabilitation activities. A major constraint is that the focus is on indoor location determination and related activity detection [1], as increasingly humans tend to spend more and more time, indoors. There are a variety of approaches to recognize human activities. However, each of them has its own limitations. Vision-based methods including video and depth camera [2] are effective yet they face problems of being privacy invasive and computationally intensive, as well as being affected by external environment effects such as the ambient lighting and generally require a line-of-sight. A device-free (off-body) based method

such as Lidar [3] can provide a high localization accuracy, yet it needs a line-of-sight. Recognition approaches via wearable sensors, e.g. accelerometer, are widely applied to fine-grained activities recognition. However, the weakness is that they cannot accurately determine location and hence to detect LD-ADLs. Compared to all of these methods, radio-frequency (RF) signal positioning methods such as WiFi, BLE [4] and UWB, are less affected by the line-of-sight constraint and are far less privacyinvasive. However, they are affected by other RF signal interference sources and physical obstacles that attenuate the signal, reducing the location-determination accuracy. Despite these issues, due to the increasing use of WiFi in smart mobile devices and the availability of densely distributed WiFi access points (APs), especially in urban areas, WiFi-based location determination is now becoming more prevalent in such areas. Further, human activity recognition using WiFi is also receiving more research attention in recent years. There are several methods for WiFi-based activity recognition including WiFi Channel State Information (CSI), Doppler Shift and Received Signal Strength Indicator (RSSI) finger-printing. WiFi CSI can be used for coarse-grained ADL recognition, e.g., E-eyes, CARM, WiKey and WiFinger [5]. The principle of CSI-based sensing is to make use of channel information in the time and frequency domain, e.g. amplitude and phase of each subcarrier at each timestamp, and to leverage how these features change, when caused by human activities between transceivers. However, CSI-based methods rely heavily on a relatively stable RF environment (e.g. single occupant and no furniture movement). Any ambient changes will trigger an activity profile updating procedure, or it may fail to work. There are also some efforts to use detailed physical layer measurements, Doppler Shifts, to detect human activities such as Wi-Vi [5] for coarse movements and WiSee [5] and WiTrack [5] for fine-grained gestures. Because human motion can lead to a pattern of Doppler shifts at the wireless receiver, that can be mapped to human activities. Doppler signal analysis typically involves Doppler shift extraction, segmentation and activity classification. However, special software and hardware are needed to extract Doppler features which results in an extra cost. [6] [7] investigated the feasibility of recognizing activities by increasing the RSSI sampling rate of a device to approximate the continuous changes of the signal affected by human motions. There are two main requirements for using this kind of solution. Firstly, it requires modified WiFi firmware to obtain a sufficient RSSI sampling frequency (i.e. more than 20Hz) which, however, is not possible to access in common mobile WiFi devices. The second requirement is to set WiFi transmitters (or routers) in a location where the WiFi signal can be interfered by human

activities. These requirements limit its application for use in practical scenarios. Currently, little work has investigated using standard WiFifingerprinting (see Fig. 1) for ADL recognition. Hence, our main contributions are: 1) We investigated three methods for LD-ADL recognitoin, using: coordinate points sequence, RSSI feature sequence and a fusion of these two, respectively. 2) We developed a novel multi-source information fusion based method based upon MD-DTW, for LD-ADL recognition that fuses RSSI derived position coordinates with RSSI trajectories. 3) We integrated the use of MD-DTW with the use of Interval Overlap Degree (IOD) to improve AP selection since the number of wireless (Access Points (APs) being used can also affect the positioning accuracy. 4) We designed some common ADLs and evaluated the accuracy of these methods defined in 1) to recognize them. The remainder of this paper is organized as follows: Section II introduces the methodology. Section III presents the evaluation. Section IV gives the conclusions and future work. II. METHODOLOGY A. Multi-source Information Fusion Based ADLs Recognition

extended into scenarios with multi-sensor information. For example, smart phones can collect signals not only from WiFi, but sensors such as magnetic field 𝐹𝑀𝐹 , Bluetooth and so forth The multi-source fusion feature matrix can be denoted by 𝐹𝐹 , see Formula (2). However this is future work. 1 𝐹𝐶1 𝐹𝑅1 𝐹𝑀𝐹 … 2 2 2 𝐹𝐶 𝐹𝑅 𝐹𝑀𝐹 ⋯ 𝐹𝐹 =  ⋮ ⋮ ⋮ 𝑞 𝑞 𝑞 ⋯ [𝐹𝐶 𝐹𝑅 𝐹𝑀𝐹 ] So far, we only fuse 𝐹𝑐 and 𝐹𝑅 which can be viewed as a good test for path matching by using multi-source features. This can be achieved using MD-DTW, an algorithm used to evaluate the comparison between two-time series with different lengths and multi-dimensional information [10]. The algorithm works as follows. Firstly, we need to normalize each type of feature sequence. Formula (3) shows normalizing the estimated coordinates x̃ and ỹ of the path L to max-min (i.e. 0-1) x̃́ and ỹ́ separately. Formula (4) shows the normalization of RSSI vectors. The benchmarks of both methods also need to be ̇ ). Then, in the normalized in the same way (e.g. (ẋ , 𝑦̇ ), 𝑅𝑆𝑆𝐼 phase of path matching, MD-DTW is utilized to find the best match. The distance elements between path L and predefined 𝐿′ can be calculated by formula (5). Formula (6) is used to find the path which minimizes the distance between current path and the trained path (S means the number of types of feature sequences, e.g. 𝐹𝑐 , 𝐹𝑅 : S = 2). The accuracy can be computed by formula (7), where Q is the number of paths matched successfully and N is the sum of paths. (x̃)𝑙𝑖 ∈𝐿 , (ỹ )𝑙𝑖 ∈𝐿 →  (x̃́ ) , (ỹ́ ) 𝑛𝑜𝑟𝑚:max _𝑚𝑖𝑛 (0,1)

𝑙𝑖 ∈𝐿

{{𝑅𝑆𝑆𝐼′1 }, {𝑅𝑆𝑆𝐼′ 2 }, . . . {𝑅𝑆𝑆𝐼′ 𝑛∈𝑁 }}𝑙 ∈𝐿 𝑖 ̃′ ′ ̃ ̃ ′ → {{𝑅𝑆𝑆𝐼 1 }, {𝑅𝑆𝑆𝐼 ′2 }, … {𝑅𝑆𝑆𝐼′ ′𝑛∈𝑁 }} max _𝑚𝑖𝑛 (0,1)

Fig. 1. Process of WiFi fingerprint positioning

The Fig. 1. shows the process of WiFi fingerprint positioning. To realize LD-ADL recognition, path matching methods will be used. Path matching can be implemented by using two different variables. One is to use the RSSI vectors of sequential points (𝑙1 , 𝑙2 , … , 𝑙𝑞 ) to form RSSI feature sequence (i.e. 𝐹𝑅 ) and the other is to use the estimated coordinates (from WiFi fingerprint positioning as shown in Fig. 1) of sequential points (𝑙1 , 𝑙2 , … , 𝑙𝑞 ) to constitute a coordinate feature sequence (i.e. 𝐹𝑐 ). 𝑅𝑆𝑆𝐼𝑙11 𝑅𝑆𝑆𝐼𝑙21 ⋯ 𝑅𝑆𝑆𝐼𝑙𝑁1 𝑥𝑙1 𝑦𝑙1 𝑥𝑙2 𝑥𝑙2 𝑅𝑆𝑆𝐼𝑙12 𝑅𝑆𝑆𝐼𝑙22 ⋯ 𝑅𝑆𝑆𝐼𝑙𝑁2 𝐹𝑐 = [ ⋮ ] 𝐹 =  𝑅 ⋮ ⋮ ⋮ ⋮ ⋮ 𝑥𝑙𝑞 𝑥𝑙𝑞 1 2 𝑁 [𝑅𝑆𝑆𝐼𝑙𝑞 𝑅𝑆𝑆𝐼𝑙𝑞 ⋯ 𝑅𝑆𝑆𝐼𝑙𝑞 ] In existing work, both methods were used separately for path matching[8][9]. However, there is no work fusing the feature sequences for path matching and coordinate estimation together. Our hypothesis is that multiple points and feature sequences fusion can provide more information compared to single type of information only, increasing the accuracy of path matching. What’s more, this fusion method also can be

𝑙𝑖 ∈𝐿

 𝑙𝑖 ∈𝐿

2 2 Dis𝑖∈𝐿,𝑗∈𝐿′ = 𝑠𝑞𝑟𝑡((x́̃ 𝑖 − ẋ 𝑗 ) + ( ̃ý 𝑖 − 𝑦̇𝑗 ) 2 ̃ ̇ + ∑𝑁 𝑛=1(𝑅𝑆𝑆𝐼′𝑛−𝑖 − 𝑅𝑆𝑆𝐼𝑛−𝑗 ) ) 1

𝑆 𝑆 MD − DTW(𝐿𝑆 , 𝐿′ ) = argmin { ∑𝐾 1 W𝑘 } 𝐾

 

accuracy = Q/N III.

EVALUATION

A. Activity Design

Fig. 2. Overview of class 1 activities

In this study, we designed 9 activities which are listed in Table 1. These activities are divided into two classes based

upon the region. The activities among the Class 1 mainly refer to user mobility from the office area to other areas (outside of the office, printing room, meeting room and kitchen) which are shown in Fig. 2. The Class 2 activities reflect the user mobility within the kitchen, which are shown in Fig. 3. The activity recognition accuracy is related to the path length (as shown in Table 1.) and dimension of feature matrix (e.g. 2 in 𝐹𝑐 or N in 𝐹𝑅 ). These two classes activities can constitute a comparative test for the effects of path length and feature dimension on recognition accuracy.

adults walk. The RSSI samples of RPs and TPs are collected using the same mobile device. In the offline stage, the size of RSSI samples at each RP and TP is 120. There are 105 detectable APs in all. We filter out some APs for which the appearance frequency (the recorded times divided by the total sampling times) at each RP is lower than 90%. After filtering, 67 APs are left. We store the mean of RP’s and TP’s samples (from 67 APs) in the radio map, separately. The function of RP’s samples is for localization, but the function of TP’s samples is to benchmark of RSSI feature sequence based path matching. During the online stage, the size of RSSI samples at each TP is 5 (the sampling time is about 2 second). We repeated these activities 20 times so that there were 17*20=340 routes in all. The ground truth of TPs’ and RPs’ coordinates is measured by a laser tape in the offline stage.

Fig. 3. Overview of class 2 activities

TABLE I. Classes

Class 1 (shown in Figure 2.)

Class 2 (shown in Figure 3.)

Activities Leave the office Have a meeting

DETAILS OF ACTIVITIES IN DATABASE Path length/m 7.5;6.5; 12.5 8;9;16

Print

10;11;18

Go to kitchen

13.5;14.5; 21.5

Make tea

1.5

Heat food

2

Drink tea

2.5

Eat food

2

Have some drink

3.5

Path

Sketch

From user desk to exit From user desk to meeting room From user desk to printing room From user desk to the kitchen Take something (e.g. milk) from fridge to Kettle Take food from fridge to microwave oven From kettle to dinner table From microwave oven to dinner table From fridge to dinner table

1 red/ ○ green/blue 2 red/ ○ green/blue 3 red/ ○ green/blue 4 red/ ○ green/blue Red line Blue line Yellow line

Fig. 4. Experiment setup

Since the number ofFig. APs 1. being used can affect the positioning accuracy and the dimension of F𝑅 , and further affect the activity recognition accuracy, we choose different number of APs ranging from 2 to 67 to show the change of accuracy along with AP numbers. The AP selection method we use is Interval Overlap Degree (IOD) [11]. The smaller the IOD of the AP, the better the discrimination of RPs and the better positioning accuracy. During the process, we optimize our positioning result by employing the Kalman Filter. C. Activities Recognition Performance

Pink line Green line

As shown in Fig. 2, the starting point of activities is the user desk (purple pentagram). There are three directions for the user separately going to ○ 1 exit; ○ 2 meeting room; ○ 3 print room; 4 kitchen. There are 12 paths in all for Class 1 activities. ○

Compared to the mobility in Class 1, the mobility in Class 2 (inside a kitchen) is finer. Here, we only choose 1 path for each activity in Class 2 to test the performance of our method for activitites with short paths. The route details are all shown in Table 1. B. Experiment Setup and Data Processing The experiment was conducted in an office room in Queen Mary with size of 16m*7m as shown in Fig. 4. There are 112 Reference Points (RPs) evenly distributed in this room which are 1m apart. The Test Points (TPs) along these predefined paths are 0.5 m apart which is about the typical step length that

Fig. 5. Activities recognition accuracy comparison between coordinate points sequence, RSSI feature sequence and MD-DTW in all (mixed) activities, class 1 (C1) activities, class 2 (C2) activities

Fig. 5 shows the changes in activity recognition accuracy of using coordinate points sequence (blue line), RSSI feature sequence (red line) and fusion feature sequence (green line) as a function of the increasing AP quantity (2~67). As for mixed and C1 activities, when there are less APs, the MD-DTW method performs not very well. When the number of APs surpasses a certain number (about 17), the recognition

performance of the MD-DTW based method shown in green solid line outperformances the methods of using coordinate points sequence and RSSI feature sequence. For C2 activities, the performance of using fusion feature sequence almost stays at the leading position from the start. From Table 2, the accuracy gap between coordinate points sequence and fusion feature sequence based methods is more notable than that of RSSI feature and the fusion based methods. Moreover, the average accuracy for C1 activities is 99.98%, for C2 activities is 99.81% and for mixed activities is 99.95% when their AP quantities are more than 49, 50, 36, respectively. TABLE II. Increase (%) Maximum Average AP Quantity

Mixed Activities C1 Activities Coora RSSIb Coora RSSIb 75.55 33.10 77.16 22.22 66.29 2.80 62.21 1.45 ≥16 ≥17 Average Accuracy of MD-DTW

Average Accuracy AP Quantity a.

RECOGNITION ACCURACY INCREASE FOR MD-DTW

C2 Activities Coora RSSIb 82.00 64.77 60.07 11.96 ≥2 (&≠29)

99.95%

99.98%

99.81%

≥50

≥49

≥36

Coord means the coordinate points sequence method b.

RSSI means the RSSI feature sequence method

When comparing the performance of MD-DTW based method for the recognition of mixed activities (blue line), C1 activities (red line) and C2 activities (green line) respectively (as shown in Fig. 6), the same reference path database (including 17 routes) being used to recognize three types of activities separately, the recognition accuracy of each class gradually increases as the AP quantity increases and finally plateaus. This means increasing the number of APs can effectively increase the recognition accuracy. The recognition accuracy for C1 activities is better than that for mixed activities, and the accuracy for mixed activities is also better than that for C2 activities. The main reason for this is that the length of C1 activities’ corresponding paths (as shown in Table 1.) is longer than the length of C2 activities’ corresponding paths. Therefore, we can conclude that the activities with longer paths achieve higher accuracy than those with short paths.

Fig. 6. MD-DTW-based activities recognition accuracy comparison

IV. CONCLUSION AND FUTURE WORK This paper mainly focused on the indoor, location-driven ADLs recognition using WiFi fingerprint method. We proposed a novel method fusing RSSI coordinate points and RSSI feature

of sequences of points to increase the accuracy for path matching and hence activity recognition. We designed 9 activities with 17 routes and divided them into two classes: Class 1 referring to the office area to other areas and Class 2 referring to inside the kitchen. We validated the path matching method in an office in the Queen Mary Mile End campus. The experiment results showed that the MD-DTW method increases the recognition accuracy of 9 activities. Besides, the accuracy of the MD-DTW method is 99.95% for mixed activities, 99.98% for Class 1 activities and 99.81% for Class 2 activities when the number of APs is more than 49, 50, 36, respectively. In terms of future work, we plan to extend the multi-source information fusion MD-DTW method by fusing more sensor information, e.g. magnetic field measurements to assess if these could further increase the accuracy of location-driven activity recognition and to undertake more experiments to validate this. ACKNOWLEDGEMENTS This research was funded in part by a China Scholarship Council (CSC) and QMUL PhD Grant for two of the researchers. We also gratefully acknowledge the support of NVIDIA Corporation who donated the GPU used for the data analytics. REFERENCES [1]

Z. Liang, I. Barakos, and S. Poslad, “Indoor location and orientation determination for wireless personal area networks,” in Mobile Entity Localization and Tracking in GPS-less Environnments, Springer, 2009, pp. 91–105. [2] R. Poppe, “A survey on vision-based human action recognition,” Image Vis. Comput., vol. 28, no. 6, pp. 976–990, 2010. [3] Z. Ma, J. Bigham, S. Poslad, B. Wu, X. Zhang, E. Bodanese, "DeviceFree Daily Life (ADL) Recognition for Smart Home Healthcare using a low-cost (2D) Lidar," in Proc. IEEE Globecom, 2018. [4] Z. Ma, S. Poslad, J. Bigham, X. Zhang, and L. Men, “A BLE RSSI ranking based indoor positioning system for generic smartphones,” in Wireless Telecommunications Symposium (WTS), 2017, 2017, pp. 1–8. [5] Y. Wang, J. Liu, Y. Chen, M. Gruteser, J. Yang, and H. Liu, “E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures,” in Proceedings of the 20th annual international conference on Mobile computing and networking, 2014, pp. 617–628. [6] S. Sigg, U. Blanke, and G. Troster, “The telepathic phone: Frictionless activity recognition from wifi-rssi,” in Pervasive Computing and Communications (PerCom), 2014 IEEE International Conference on, 2014, pp. 148–155. [7] M. Scholz, T. Riedel, M. Hock, and M. Beigl, “Device-free and devicebound activity recognition using radio signal strength,” in Proceedings of the 4th augmented human international conference, 2013, pp. 100– 107. [8] Y. Li, Y. Zhuang, H. Lan, X. Niu, and N. El-Sheimy, “A profilematching method for wireless positioning,” IEEE Commun. Lett., vol. 20, no. 12, pp. 2514–2517, 2016. [9] A. Ismail and A. Vigneron, “A New Trajectory Similarity Measure for GPS Data,” in Proceedings of the 6th ACM SIGSPATIAL International Workshop on GeoStreaming, New York, NY, USA, 2015, pp. 19–22. [10] G. A. ten Holt, M. J. Reinders, and E. A. Hendriks, “Multi-dimensional dynamic time warping for gesture recognition,” in Thirteenth annual conference of the Advanced School for Computing and Imaging, 2007, vol. 300, p. 1. [11] B. Wu, Z. Ma, S. Poslad, and W. Zhang, “An efficient wireless access point selection algorithm for location determination based on RSSI interval overlap degree determination,” in Wireless Telecommunications Symposium (WTS), 2018, 2018, pp. 1–8.