A Simple Algorithm to Monitor HR for Real Time Treatment Applications Banitsas K., Pelegris P., Orbach T., Cavouras D., Sidiropoulos K. and Kostopoulos S. Abstract— As the demand for effective and reliable telecare systems increases rapidly over the last years, novel ideas applied on existing consumer products enables the development of innovative solutions that could enhance the user’s wellbeing. In this research, we are going to demonstrate the potential of a system that enables users to monitor their own heart beat rate in real time and use specialised software for personal health coaching. In this paper we will explain and demonstrate how to extract heart beat rate information from a user using the camera of a commercially available mobile phone which will enable us to supply the users of the system with vital information and utilize interactive tools useful for personal health coaching. Our industrial partner Health Smart Limited have filed a patent [1] for this application, they retain the full intellectual property of this project. Index Terms— HBR, camera, mobile, cbt, health coaching I. INTRODUCTION
T
HE rapid evolution of technology has given us with the opportunity to gradually provide access of cost effective sophisticated electronic devices to the public, currently most people can buy a smart phone with a camera for a reasonable cost. From an engineering point of view this creates additional possibilities of application deployment and service delivery: a modern mobile phone gives you enough processing power in conjunction with mobility while it can also be used as a media platform for increased user interaction [3],[8]. Our industrial collaborator, Health-Smart Limited has substantial experience in self-care ICT for prevention and controlling Long Term Conditions (LTC) including cardiovascular and psychological conditions, one of their purpose is to empower patients to assess their state of body and mind and train them to improve their health and prevent Long Term Conditions (LTC), by using inexpensive friendly consumer ICT and sensors [2]. We have developed a demonstration based on an invention - patent they had filed
Manuscript submitted 21th August 2009. K.Banitsas ,P. Pelegris and K. Sidiropoulos are with the Electrical and Computer Engineering Department, Brunel University, West London, Uxbridge, Middlesex, UB8 3PH, UK (phone: +44(0)1895266886;(email:{konstantinos.banitsas,panagiotis.pelegris,konstantinos.sidiropoulos} @brunel.ac.uk). D. Cavouras and S. Kostopoulos are with the Department of Medical Instruments Technology, Technological Educational Institute of Athens, Ag. Spyridonos Street, Egaleo, 122 10, Greece. (email : {cavouras,skostopoulos}@teiath.gr).
[1] to extract heart rate (HR) information from a user, using mobile phone with a camera (without any other sensor). The importance of this invention is that it enables anyone to use a standard mobile phone with a camera as a heart rate monitor and health coach. In this research the delivered service is actually pre-emptive in character and aims in enhancing the well-being of the users by providing them with simple tools, methods, information regarding the state of their heart and coach them on how to improve their physical and emotional state [9]. In the proposed system we are examining the usage of an embedded camera in a mobile phone or PDA as means of measuring the heart rate (HR). This will help us provide near real time feedback to the users and will assist them in using self coaching software to enhance their well-being. Although there are numerous researchers active on working with mobile phone and PDA's implementing healthcare systems the vast majority of them use the mobile phone and PDA as a processing and media interface unit (or communication device) and never as a sensor by itself. In each and every case the user has to wear some kind of sensor unit that enables a device to monitor the user's physiological signs. The less intrusive a solution is the more chances it has to be accepted by people [8]. A touch screen mobile phone / PDA with a simple interface that has no need for extra wearable sensors might prove to be more successful. The proposed system is compatible with most of the commercially available modern mobile phones with a video camera. Basically the camera is used to capture a video from the users’ finger and also as a media interface that will generate the feedback and provide them with useful self coaching advice that can improve their well-being [5]. In order to do that they just need to press their fingertip on the camera while the PDA device captures a short length video. When this is over the device will analyze their heart beat information and provide feedback for the actual heart beat rate. Following that personal health coaching software takes over to utilize this information for an actual pre-emptive health service. For example software can teach the users how to breathe properly and help them regulate their heart rate. The process can be repeated over a period of time so T. Orbach is with Health-Smart Limited, 77b Fleet Road, Hampstead, London, NW3 2QU, UK (e-mail:
[email protected], www.healthsmart.co.uk).
that users can check their progress and see immediate results on heart beat self regulation. In the future it will be possible to analyze the HR continuously and provide feedback and coaching to the users in near real-time. The key challenge here is to engage the user in a process that will enhance his well-being providing him with personalised health coaching, delivering effective preemptive health services. In order to succeed in that you need an easy to use interface, a system that requires no extra effort like wearing sensor units and a way to provide quick feedback for the users enabling them to check the effects of their “treatment” right away [4]. The user friendliness of self monitoring equipment is crucial for the adoption and success of it. Familiarising users with such solutions will help them adapt to more complex systems in the near future. The ongoing standardisation on telecare and Telehealth will soon start producing practical health monitoring systems based on multiple sensor readings. This research was designed to be carried out in two stages: firstly take the input from a PDA and do the calculations on a PC and finally implement the whole system in real time in a mobile device. In this paper we demonstrate the process in a desktop environment using video files taken from a PDA Mobile phone (Eten Glofish X800) as input. II. METHODOLOGY The invention is based on two interesting phenomena: 1) Every heart beat creates a wave of blood that reach the capillaries in the tip of the finger; when the capillaries are full of blood less light can pass through them. So the changes in the amount and the colours of light which is passing through the finger can represent the changes in the shape of the pulse and its timing (the HR). 2) Normally to take a photo or a video we need focus and some distance between the lens and the object. However we are interested only in the quantity of light, therefore there is no need of focus, and it is possible to cover the object lens of the camera with the finger tip as shown in Figure 1 . As long as there is a source of light (which can be natural ambient light or artificial light) which can pass through the finger, it is possible to view the pulsation of the blood as changes in the amount of light in the video. The major advantage of this invention is that the user does not need any sensors, does not require any external hardware whatsoever, it does not require even to focus; one just touches and cover the lens of the camera with their finger tip and with the right software they can view their HR and learn to improve their health and fitness. It was worth to examine if one can discern any information about the pulse wave when placing the finger directly on top of a camera with strong flash light right next to it.
Figure 1: Pressing the finger on top of the camera while the flash is turned on
The initial findings where very encouraging, as it is illustrated in Figure 2 the red portion of the finger is clearly visible and easily distinguishable from the rest of the picture, unfortunately in this paper we cannot visually demonstrate the fluctuation from the pulse wave as it only appears on video but it is important to note that the resolution provides enough information to estimate the heart rate as long as the finger is kept right on top of the camera.
Figure 2: Sample frames from finger images on PDA camera arrows denote the pulsating regions.
In order to develop the system we used both desktop and mobile environment applications for testing. The PDA Mobile phone used was an Eten Glofish X800. The resolution used was 320 x 240 pixels with an effective frame capture rate of 25 frames / sec. Although these are not the current highest available characteristics for a mobile phone with a camera they proved to be adequate for our experiments [6].
Figure 3: Frame Profile with virtual vertical axis along the middle
Figure 4: Crude pulse signal, illustrating unexpected dive
The system's behaviour was initially simulated on Matlab using videos captured from the camera on the PDA as input. For each video the frames were extracted and a profile was created for every frame. We deliberately scanned the profile of a virtual vertical axis right in the middle of the frame calculating light variation along this axis. The same procedure was repeated for every frame. However this is only one method to calculate the changes in amount of light.
Figure 4 represents a crude preview of the pulse variation, notice how the signal dives at approximately frame 210 indicating a possible sudden movement of the finger or an external source of noise, however this unforeseen development does not considerably affect the outcome of the calculation as the information we are looking for is basically contained in the frequency of the signal rather than on its amplitude.
Our technique is based on determining red channel variation along a given area of the image. This represents a good estimation on how much light is getting through. The process is followed for all frames and amplitude is normalised giving the crude pulse signal as shown in Figure 4. An alternative technique would be to process data from all channels in RGB mode, or take multiple measurements from different areas of the image to reinforce our result accuracy. However in this work we mostly aim in demonstrating the efficiency of the algorithm in this application as the fine tuning is a process that will be explored in the development of the final product.
The next step is to normalise the crude signal into something more meaningful, which is easily achievable using smooth differentiation. From this plot it is simple to get the number of peaks, which divided by the running time of the video will give us the estimation for the heart beat rate. In the given example the result was 61.3 beats / min as shown in Figure 5.
Moreover since our target platform is a mobile device with limited resources both in terms of processing power and memory, it is reasonable to get the input algorithm fairly simple as long as the results appear to be highly accurate. Another issue is that we can directly scan the red channel without any other type of conversion ensuring that our input data goes through a minimum of transformation and compression stages giving us the maximum possible efficient information.
Figure 5: Normalized signal
III. RESULTS After the simulation stage, we needed to evaluate accuracy of the algorithm using real examples and think of ways to improve performance. A custom software application was built to process the frames from the video, scan for colour variations, locate peaks and calculate beats per minute. It appears that videos of 20 seconds produce results with around 91% average accuracy, with minimum 85% and maximum 99% compared to the measured HBR values. Samples of 14 – 16 and 18 seconds have an average accuracy of 89%, the minimum was 81% at 14 second samples and the maximum was 100% at 16 second samples. The results are summarised on Table.1. The samples of 12 and 10 seconds have only 87% and 86% accuracy respectively, that is more or less anticipated since when calculations are based on data from 10 seconds, any error would appear to be six times higher when you are estimating beats per minute. A simple optimisation that we will incorporate in future versions is the sharp definition of start and end time based on the actual beginning or ending of a pulse. This would directly result in an improvement of up to 12 pulses for a 10 second video in some cases, this change might make it possible for shorter streams to give one enough data for reliable analysis. The flexibility of the system lets the user decide on the required resolution, for example a sample of 20 seconds would produce results with a resolution of 3 beats. If a user for example has 25 beats in 20 seconds based on that sample he has 75 beats in 60 seconds. If the same user had 26 beats in 20 seconds then the software would calculate 78 beats in 60 seconds. Similarly samples of 10 seconds would have a resolution of 6 beats, being far less reliable since only the integer portion of the beats is used in the calculations. The sampling time effectively gives you the flexibility to define the desired resolution based on your needs. Figure 6 illustrates the demonstrating application in action while calculating results based on a 16 second sample; this process doesn’t take more than 3 seconds on a 2.4 GHz processor. The frames are extracted from video files into PNG images using FFMPEG and are later processed by the application to produce the actual results. During testing we found out that PNG images give much finer detail when it comes colour compression compared to the standard JPG images. The sources files from the X800 are in 3GP format and frames are extracted directly in order to avoid the detrimental effects of subsequent recompression stages into different compression schemes.
Figure 6: Sample desktop application to calculate pulses
In the above example the actual beats per minute where 76 and the application calculated 75 in a 16 second sample. In this case the 16 second result was more accurate compared to the 20 second result which was 78 but in both cases the results are within the given resolution for 20 second samples which are 3 beats per minute. Table.1 Measured and Calculated HBR for different sampling times Sample/HBR Measured Calc.10sec Calc.12sec Calc.14sec Calc.16sec Calc.18sec Calc.20sec 1 69 60 65 68 67 63 63 2 78 60 60 64 63 66 66 65 68 67 70 72 3 78 66 4 75 72 70 68 67 70 69 5 74 60 60 60 60 63 69 6 79 72 75 77 78 76 78 7 72 60 60 60 63 63 63 8 75 66 65 64 63 63 66 9 75 66 70 77 75 73 72 10 78 66 65 68 67 66 69
Table.1 summarizes the heart beat rate calculation results for 10 random video samples. This is still an early stage on the development of the algorithm and it appears that a few minor adjustments will have significant impact. Table.2 Relative HBR accuracy percentages Sample/Accuracy 1 2 3 4 5 6 7 8 9 10
10sec 0.87 0.77 0.85 0.96 0.81 0.91 0.83 0.88 0.88 0.85
12sec 0.94 0.77 0.83 0.93 0.81 0.95 0.83 0.87 0.93 0.83
14sec 0.99 0.82 0.87 0.91 0.81 0.97 0.83 0.85 0.97 0.87
16sec 0.97 0.81 0.86 0.89 0.81 0.99 0.88 0.84 1.00 0.86
18sec 0.91 0.85 0.90 0.93 0.85 0.96 0.88 0.84 0.97 0.85
20sec 0.91 0.85 0.92 0.92 0.93 0.99 0.88 0.88 0.96 0.88
IV. DISCUSSION As life expectancy increases over the last decades so does the burden on the health system supporting people with chronic conditions, the only way out of this is implementing telecare solutions that will manage to increase the quality of delivered health care while maintaining low installation and running costs [7]. It is not only that health services delivery will change, but also that the nature of the service itself will change shifting from re-active treatment of conditions to pre-emptive health care. Avoiding health risks would be more efficient than sustaining patient's with chronic conditions that could have been avoided [2]. This is where health monitoring and cognitive therapy comes into play, providing the users with information on how to avoid getting a health condition rather than focusing on how to treat it.
greatly improve his general feeling about using new modern technology in order to maintain well-being or improve health through specialised self coaching. The invention will be used in the near future as a subsystem for a self help application that aims to help users reduce their stress levels and improve their well-being.
ACKNOWLEDGMENT We would like to express our appreciation to HealthSmart Limited for its important contribution to this paper and the general research that resulted in a patent application [1].
REFERENCES [1]
Biomedical data acquisition will also play a major role on applications where monitoring physical condition and alertness will be critical in respect to safety minimizing associated risks in sensitive and high risk areas like aerospace and aviation. Cross disciplinary synergies in the past have produced important advancements in the area of unobtrusive multi sensorial data acquisition systems [10]. Such platforms can provide invaluable input to self coaching health systems that will aim in enhancing well being. The challenge remains both for researchers and the industry to agree on standards, engineer innovative solutions that will cover the needs while minimising the cost and take telecare to the next level. This application is a subsystem of a telecare platform that is being developed in collaboration with our industrial partner Health-Smart Limited, as a standalone system though it may serve as a media platform to deliver cognitive therapy treatment and positive psychology advice and practice. Future work includes migrating the application and algorithm on mobile devices and including it in a larger software package that will deliver health-smart solutions. This includes further optimisation of the code to improve accuracy and increase tolerance to noise sources and combine the end result with interactive “game-style” applications that will run on a PDA in the context of personal health coaching. V. CONCLUSION In this work demonstrated the efficiency of an innovative algorithm to detect heart beats per minute using inexpensive widely available hardware. The objective of this research is to provide a simple and fairly accurate tool for pre-emptive health services to be delivered on patients using mobile devices. Furthermore real time feedback information for a user will
PCT patent application No. PCT/GB2009/050989 Blood Analysis – Health-Smart Ltd. [2] T. Orbach and J. Vasquez, “Self-care and the need for interactive ICT”, Journal of holistic healthcare, vol. 6, pp. 35-39, August 2009. [3] R. Gururajan, S. Murugesan and J. Soar, “Bringing Mobile Technologies in Support of Healthcare Recommendations for a Healthy Beginning and Growth,” Cutter IT Journal Article, Aug. 2005. [4] A. Duckworth, T. Steen and M. Seligman, “Positive Psychology in Clinical Practice,” Annual Review of Clinical Psychology, vol. 1, pp. 629 – 651, April 2005. [5] M. Seligman, T. Steen, N. Park and C. Peterson, “Positive Psychology Progress : Empirical Validation of Interventions”, American Psychologist, July 2005. [6] M. Fischer, Y. Yang Lim, E. Lawrence and L. Ganguli, “ReMoteCare: Health Monitoring with Streaming Video,” in 7th International Conference on Mobile Business, July 2008. [7] V. Shnayder, B. Chen, K. Lorincz, T. Fulford-Jones and M.Welsh, “Sensor Networks for Medical Care,” Technical Report TR-08-05, Division of Engineering and Applied Sciences, Harvard University, 2005. [8] A. Pantelopoulos and N. Bourbakis, “A Survey on Wearable Biosensor Systems for Health Monitoring,” 30th Annual International IEEE EMBS Conference, Vancouver, British Columbia, Canada, August 20-24, 2008. [9] H. Jang, S. Kim and C. Bae, “Personalized Healthcare through Intelligent Gadgets,” 30th Annual International IEEE EMBS Conference, Vancouver, British Columbia, Canada, August 20-24, 2008. [10] A. Astaras, PD. Bamidis, C. Kourtidou-Papadeli and N. Maglaveras, “Biomedical real-time monitoring in restricted and safety-critical environments”, Hippocratia, vol. 12, suppl. 1, pp 10 – 14, August 2008 .
