Development of A Real Time Activity Monitoring Android Application Utilizing SmartStep Nagaraj Hegde, Student Member IEEE, Edward Melanson, and Edward Sazonov, Senior Member, IEEE
Abstract— Footwear based activity monitoring systems are becoming popular in academic research as well as consumer industry segments. In our previous work, we had presented developmental aspects of an insole based activity and gait monitoring system-SmartStep, which is a socially acceptable, fully wireless and versatile insole. The present work describes the development of an Android application that captures the SmartStep data wirelessly over Bluetooth Low energy (BLE), computes features on the received data, runs activity classification algorithms and provides real time feedback. The development of activity classification methods was based on the the data from a human study involving 4 participants. Participants were asked to perform activities of sitting, standing, walking, and cycling while they wore SmartStep insole system. Multinomial Logistic Discrimination (MLD) was utilized in the development of machine learning model for activity prediction. The resulting classification model was implemented in an Android Smartphone. The Android application was benchmarked for power consumption and CPU loading. Leave one out cross validation resulted in average accuracy of 96.9% during model training phase. The Android application for real time activity classification was tested on a human subject wearing SmartStep resulting in testing accuracy of 95.4%.
I. INTRODUCTION rd
More than 2/3 of the United States population is currently overweight or obese [1], and there are multiple health risks that are related with obesity [2]. Obesity results from a chronic imbalance between the energy intake of the individuals and the energy expenditure. Individuals attempting to lose or maintain weight are recommended to modify their diets to reduce energy intake, reduce the sedentary activities in their daily living and increase physical activity to increase daily energy expenditure. Hence, Accurate monitoring of daily living activities is important in the field of obesity control [3]–[5]. Due to the boom in the wearables systems market in the recent years, there are many wearable systems that are used for activity monitoring [6], [7] but most of them struggle with low accuracies for daily living activities classification [8]. Footwear based activity monitoring systems [5], [9] are becoming prominent as they N
. Hegde and E.S. Sazonov are with the Dept. of Electrical and Computer Engineering University of Alabama, Tuscaloosa, AL 35487 USA (
[email protected]) E. Melanson is with Division of Endocrinology, Diabetes, and Metabolism and the Division of Geriatric Medicine at the University of Colorado Anschutz Medical Campus, Denver, CO 80045 USA.
978-1-4577-0220-4/16/$31.00 ©2016 IEEE
can differentiate between weight bearing and non-weight bearing activities such as sitting and standing, which are challenging for accelerometer based devices. Not just for obesity control, activity monitoring is also an important aspect in the treatment of many health conditions, such as stroke rehabilitation programs, rehabilation programs for cerebral palsy and others. Stroke is the third leading cause of death in the United States as more than 140,000 people die each year from stroke in the United States [10]. There are published works, which had utilized wearable systems in activity monitoring of post stroke individuals to help in the rehabilitation programs [11]–[13]. We had presented the developmental aspects of an insole based gait and activity monitor – SmartStep in our previous research [14]–[16], which is user friendly, unobtrusive and versatile wearable system. SmartStep integrated pressure sensors, accelerometer, flash memory, Bluetooth low energy (BLE) connectivity and wireless charging in it, and utilizes approaches to minimize power consumption. Results in [16] showed that SmartStep system is two orders of magnitude less power consuming system, compared to our previous SmartShoe platforms [3]. SmartStep has over the air programming capability, making it a versatile system that can be used in many different application scenarios such as real time streaming or offline data collection [15]. The present work describes the developmental aspects of an Android application, that can help understand the capability of SmartStep system for real time activity monitoring. The following sections describe the human subject experiment that was carried out in a laboratory environment to collect the sensor data from the SmartStep, the data processing methodology to generate machine learning models for automatic detection of activities in smartphone, the implementation of real time Android application for activity classification, testing human subject experiment and results of the benchmarking of the Android application. II. METHODS SmartStep is used in real time streaming mode in this application. The hardware and firmware details for the SmartStep can be found in our previously published work [14], [15]. Figure 1 shows a SmartStep insole. SmartStep has 3 pressure sensors located at the biomechanically important locations of heel, metatarsal head and big toe locations (referred to as P-Heel, P-Meta and P-BT). Along with pressure sensors it features 3-axis accelerometer and 3-axis gyroscope, and only accelerometer data among them
1886
Models for Activity Classification As previously shown [16], the sensor data from SmartStep have very high signal to noise ratio and don’t demand data pre-processing for activity classification purposes. The data processing procedure for activity prediction model development had the below steps: 1. Data from the experiments were labeled according to the activities performed. 2. Features were computed in 2 s interval. The computed features were: mean of total pressure (PHeel + P-Meta + P-BT) over 2 s, standard deviation (std) of total pressure, mean resultant acceleration (sqrt (AX2+AY2+AZ2)) and std resultant acceleration.
Figure 1. SmartStep Insole System
(referred to as AX, AY and AZ) is used in this current application. The SmartStep is configured to sample pressure sensors and accelerometer at 50 Hz, with 16-bit precision and transmit them to the Android client application at 50 Hz in real time over BLE. Android BLE application presented in [14] was used to collect the real time data coming in from the SmartStep at 50 Hz and was written to a csv file in the smartphone. A top level flow chart for development of real time activity classification application in a smartphone using SmartSteps are depicted in Figure 2. A. Human Subject Study Details for Data Collection Four adult subjects (three US M9 size, one US W9 size) were asked to perform the activities of sitting, standing, walking on a treadmill and cycling while they wore SmartStep insole in their shoe. Table I shows the subject demography. The study was approved by the Internal Review Board at The University of Alabama. Each activity was performed for 10 minutes. The sitting and standing activities had variations during the data collection (sitting/standing with and without fidgeting). Walking activity involved walking at 2 mph and 4 mph. Each of the experiment was conducted separately so that the sensor data can be labeled for modeling purpose. To obtain steady state data during experimentation, the first 30 seconds of data were ignored. TABLE I. SUBJECT DEMOGRAPHY
mean ± std Age (years) Height (m) Weight (kg) BMI (kg/m2)
28±0.5 1.7±0.04 69.2 ± 12.7 24.2±3.7
B. Offline Data Processing to Develop Machine Learning
Figure 2. Top level flow chart for development of real time activity classification application in a smartphone
3. Multinomial logistic discrimination (MLD) was used to train the activity prediction models in Matlab. This specific methodology was used because we have previously shown [5] that MLD can result in comparable accuracy with Support Vector Machine (SVM) or Artificial Neural Network (ANN), while consuming less memory and is faster for real time computation. 4. A leave one out cross validation strategy was used for traning of MLD models and average accuracy is reported. The trained MLD activity prediction model (matrix M of 5x3 double precision) was then saved. C. Implementation of the real time Activity Classification Application in Android Google Nexus 5 smartphone running Android v 6.0 was used to develop the real time activity and gait monitoring application using SmartStep. The smartphone has a Qualcomm Snapdragon 800 chipset with quad-core 2.3 GHz Krait 400 CPU. The Android Application starts with BioProfile activity, which captures the user’s data as shown in Figure 3a. The completion of this activity triggers a sensor scanning activity which gives options to select the particular SmartStep device of interest. Once the user selects the particular sensor, the Homescreen Activity assumes the control and starts a background service to interact with the sensor. The Homescreen activity is used as the user interface for the Activity classification. The background service connects to the SmartStep, enables data streaming from the SmartStep and receives the periodic data at 50 Hz. As the data arrives, the same features as above (mean total pressure, std of total pressure, mean resultant acceleration and std resultant acceleration) are computed in real time (every 2 s window), and are then sent for classification. The classification algorithm runs in the background, and doesn’t affect the data reception happening at 50 Hz. MLD prediction model is coded in Java and predicts the current activity. The result of the prediction algorithm is sent to the Homescreen Activity every 2 s. Homescreen activity then displays the current activity and a cumulative amount of time each activity that has been performed over the day as shown in Figure 3b.
1887
TABLE II. CONFUSION MATRIX FOR TESTING OF THE RESULTING ANDROID APPLICATION Sitting
(a)
Standing
Walking
Cycling
283
0
4
30
Standing
2
282
13
0
Walking
0
3
268
0
Cycling
0
0
0
251
Sitting
B. Human Subject Experiment to Test the Real Time Android Application for Activity Classification Classification accuracy is computed from the confusion matrix for the human subject experiment to test the real time android application for activity classification. The confusion matrix for the testing is shown in Table II, resulting in testing accuracy of 95.4%.
(b)
Figure 3. a. Screenshot of the BioProfile Activity b. Screenshot of Homescreen activity showing real time activity classification
D. Human Subject Experiment to Test the Real Time Android Application for Activity Classification To test the implementation of real time activity classification implementation in Android, a human subject study was conducted. One separate adult subject (not used during the training of the model, US M9 size) was asked to perform the activities of sitting, standing, walking and cycling while wearing SmartStep insole. The SmartStep was communicating with the real time activity classification Android application running in the smartphone. Each activity was performed for 10 minutes. Each experiment was conducted separately so that the accuracy of prediction from the real time activity classifier could be computed. The smartphone application stored the classified activity (every 2 s) in a csv file, which was later used to determine the testing accuracy. Figure 4: Screenshot of the Android application for real time
C. Power consumption and CPU load profiling of the Android Application Table III shows the results of the power consumption and CPU loading tests from the Android application for real time activity monitoring. TABLE III. ANDROID APPLICATION PROFILING RESULTS. On a Google Nexus 5 CPU Load, BLE scanning (%) CPU Load, BLE data reception at 50 Hz(%) CPU Load, BLE data streaming at 50 Hz, real time feature computation and classification (%) Average current consumption, BLE scanning (mA) Average current consumption, BLE data reception at 50 Hz (mA) Average current consumption, BLE data streaming at 50 Hz, real time feature computation and classification (mA)
activity classification
E. Power consumption and CPU load profiling of the Android Application Trepn Profiler [17] was used to profile the power consumption and the CPU loading from the Android application for real time activity classification, during periodic scanning, data streaming and real time activity classification. During the profiling tests, no additional applications were running in the background which could potentially affect the results. III. RESULTS A. Leave One Out Cross Validation Classification accuracy is computed from the confusion matrix and a cumulative confusion matrix for the leave one out cross validation is shown in Table I.
0.1 2.0 2.7 14 22 28
IV. DISCUSSION AND CONCLUSION Average leave one out cross validation accuracy of 96.9% and Android application’s testing accuracy of 95.4% suggest that utilizing SmartStep, we can classify common daily living activities of sitting, standing, walking and cycling, with a high degree of accuracy. SmartStep system along with the presented android application has very high accuracy (>99%) in classifying sedentary activities of sitting and standing, which are difficult for many of the commercially available activity trackers to distinguish. Results of leave one out cross validation (Table I) and testing (Table II) show that cycling activities have relatively
TABLE I. CUMULATIVE CONFUSION MATRIX FROM LEAVE ONE OUT CROSS VALIDATION Sitting Sitting
Standing
Walking
Cycling
1253
0
8
98
Standing
3
1250
25
0
Walking
0
5
1167
0
Cycling
0
0
10
1070
Figure 4. Example classification from a leave one out cross validation
1888
less accuracy of ~90%, as some of the cycling activities are confused with sitting activities. This can be explained by the fact that, cycling is performed while sitting on the bike. The results may be improved by utilizing gyroscope data, which will be explored in the future. This is a preliminary developmental report and a more detailed human subject experimentation is under way to validate the approach fully. The work in progress also includes an expanded set of daily living activities such as washing dishes, doing laundry, vacuuming and others. Future work will also be aimed at using the classification algorthims to estimate energy expenditure. The results of the current study suggest that the proposed methodology can be used in computing activitiy transitions during daily living, which can aid in better computation of daily energy expenditure. Results of the profiling tests suggest that the presented Android application is not CPU intensive (2.7% CPU loading at max). The real time activity classification algorithm (with real time data reception, 2 s feature computation and activity classification) consumes 27% higher power relative to that of only real time data reception. On a Google Nexus 5 with 2300 mAh battery, this can mean a ~20 hours reduction in battery life with activity classification algorithms (~run time of 3.5 days), compared to that of only real time data reception (run time of ~4.5 days). Minimizing the wireless connection events from the sensor, that occur at 50 Hz in the present application can help in reducing the power consumption by the Android application. Reducing the BLE connections will also help to save the power in the SmartStep sensor, which is more beneficial as the SmartStep runs on a small 40 mAh battery. This could be achieved by performing some portion of the activity classification on the sensor itself, which will be dealt in our future work. Common with any other form of wearable technology, SmartStep or any footwear based solution will also potentially suffer from missing data problem. Extracting the daily activity information when the footwear is not in use, will be explored in the future work. The methodology can be extended to compute gait parameters of walking activities in real time, implementing the methodology as published before [18]. The temporal gait parameters such as cadence, gait cycle time, swing time, step time, stance time and others can be easily computed utilizing the pressure sensor data from the SmartSteps. The same application can also be installed on a 10 inch Android tablet computers if need arises to offer a better user experience for users.
REFERENCES [1] [2] [3]
[4]
[5]
[6]
[7] [8]
[9] [10] [11] [12]
[13]
[14] [15] [16]
[17] [18]
In this work we presented early developmental aspects of a real time activity monitoring Android application interacting with SmartStep insole monitors. Very high validation accuracy was obtained for common daily living activities of sitting, standing, walking and cycling. The results suggest that SmartStep, which is socially acceptable, low power, versatile insole system, is also accurate in real time activity monitoring applications.
1889
“Adult Obesity Facts | Data | Adult | Obesity | DNPAO | CDC.” [Online]. Available: http://www.cdc.gov/obesity/data/adult.html. [Accessed: 02-Dec-2015]. Flegal KM, Carroll MD, Ogden CL, and Curtin LR, “PRevalence and trends in obesity among us adults, 1999-2008,” JAMA, vol. 303, no. 3, pp. 235–241, Jan. 2010. S. R. Edgar, G. D. Fulk, and E. S. Sazonov, “Recognition of household and athletic activities using SmartShoe,” Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf., vol. 2012, pp. 6382–6385, 2012. M. Saito, K. Nakajima, C. Takano, Y. Ohta, C. Sugimoto, R. Ezoe, K. Sasaki, H. Hosaka, T. Ifukube, S. Ino, and K. Yamashita, “An in-shoe device to measure plantar pressure during daily human activity,” Med. Eng. Phys., vol. 33, no. 5, pp. 638–645, Jun. 2011. E. Sazonov, N. Hegde, R. C. Browning, E. L. Melanson, and N. A. Sazonova, “Posture and activity recognition and energy expenditure estimation in a wearable platform,” IEEE J. Biomed. Health Inform., vol. 19, no. 4, pp. 1339–1346, Jul. 2015. T. Ferguson, A. V. Rowlands, T. Olds, and C. Maher, “The validity of consumer-level, activity monitors in healthy adults worn in free-living conditions: a cross-sectional study,” Int. J. Behav. Nutr. Phys. Act., vol. 12, Mar. 2015. J.-M. Lee, Y. Kim, and G. J. Welk, “Validity of consumer-based physical activity monitors,” Med. Sci. Sports Exerc., vol. 46, no. 9, pp. 1840–1848, Sep. 2014. K. L. Dannecker, N. A. Sazonova, E. L. Melanson, E. S. Sazonov, and R. C. Browning, “A comparison of energy expenditure estimation of several physical activity monitors,” Med. Sci. Sports Exerc., vol. 45, no. 11, pp. 2105–2112, Nov. 2013. B. Najafi, A. Vaziri, and A.-R. Boloori, “Ambulatory system for measuring and monitoring physical activity and risk of falling and for automatic fall detection,” US9005141 B1, 14-Apr-2015. “Stroke Facts | cdc.gov.” [Online]. Available: http://www.cdc.gov/stroke/facts.htm. [Accessed: 07-May-2015]. G. D. Fulk and E. Sazonov, “Using sensors to measure activity in people with stroke,” Top. Stroke Rehabil., vol. 18, no. 6, pp. 746–757, Dec. 2011. G. D. Fulk, S. R. Edgar, R. Bierwirth, P. Hart, P. Lopez-Meyer, and E. Sazonov, “Identifying activity levels and steps of people with stroke using a novel shoe-based sensor,” J. Neurol. Phys. Ther. JNPT, vol. 36, no. 2, pp. 100–107, Jun. 2012. N. Hegde, G. D. Fulk, and E. S. Sazonov, “Development of the RTGAIT, a Real-Time feedback device to improve Gait of individuals with stroke,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015, pp. 5724–5727. N. Hegde and E. Sazonov, “SmartStep: A Fully Integrated, LowPower Insole Monitor,” Electronics, vol. 3, no. 2, pp. 381–397, Jun. 2014. Nagaraj Hegde and Edward Sazonov, “SmartStep 2.0 – A completely wireless, versatile insole monitoring system,” IEEE BIBM. E. S. Sazonov, N. Hegde, and W. Tang, “Development of SmartStep: an insole-based physical activity monitor,” Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf., vol. 2013, pp. 7209–7212, 2013. Trepn Profiler. Qualcomm Innovation Center, Inc., 2015. P. Lopez-Meyer, G. D. Fulk, and E. S. Sazonov, “Automatic detection of temporal gait parameters in poststroke individuals,” IEEE Trans. Inf. Technol. Biomed. Publ. IEEE Eng. Med. Biol. Soc., vol. 15, no. 4, pp. 594–601, Jul. 2011.
