Application of infrared thermal imaging in ...

TIC-STH 2009

Application of infrared thermal imaging in rehabilitation engineering: preliminary results Negar Memarian, Tom Chau

Anastasios N. Venetsanopoulos

Institute of Biomaterials and Biomedical Engineering University of Toronto and Bloorview Kids Rehab Toronto, Canada [email protected], [email protected]

Department of Electrical and Computer Engineering Ryerson University and University of Toronto Toronto, Canada [email protected] In this paper we present an excerpt summary of our research on infrared thermography based access technology for people with severe motor impairments [9]. Before discussing our proposed system, we briefly describe infrared thermal imaging and review some of its recent applications particularly in the medical domain. We will also discuss why we believe infrared thermal imaging can be an appealing access pathway for individuals with physical disabilities. The interested reader is encouraged to see our recent publication [9] for more details.

Abstract— People with severe motor impairments often require an alternative access solution to communicate. We introduce a novel, non-invasive access solution based on infrared thermal imaging. Thermographic videos of ten participants (two with severe disabilities) were recorded with an infrared thermal camera during repeatedly cued mouth opening and closing. An algorithm based on adaptive intensity thresholding, motion tracking and morphological analysis detected mouth opening with high sensitivity and specificity (average sensitivity: 88.5%, average specificity: 99.4%). Our findings suggest that further research on the infrared thermographic access solution is warranted. Infrared thermography can be used as a non-contact and non-invasive access pathway for individuals who retain voluntary mouth opening and closing. Flexible camera location, convenience of use and robustness to ambient lighting levels, changes in background scene and extraneous body movements make this a potential new access modality that can be used night or day in unconstrained environments. Keywords- physical disability; rehabilitation engineering; access solution; computer vision

I.

II.

Infrared thermography refers to the measurement of the radiation emitted by the surface of an object in the infrared range of the electromagnetic spectrum, i.e., between wavelengths of 0.8 µm and 1.0 mm [10]. Infrared cameras use specialized lenses manufactured from materials such as germanium to focus thermal radiation onto a focal plane array of infrared detectors [11]. Cameras with the capability of producing digital output yield an image that is a spatial twodimensional (2-D) map of the 3-D temperature distribution of the object [12].

INTRODUCTION

Humans are constantly using bodily resources to communicate with other humans or to perform various daily activities. We have the ability to talk through the intricate interplay of various articulators (e.g., jaw, tongue, velum, lips, and mouth) within the oral, nasal and pharyngeal cavities. Often times we use body language to complement what we speak. We use our fingers to write or type, use our feet to walk, run or push on bike or car pedals. These are only a few examples of how we express ourselves or translate our intention to function. Individuals with severe motor impairments who are unable to communicate through speech or gestures require an alternative means to convey their intentions or perform functions. In the rehabilitation engineering context, these alternative channels are called access pathways and they constitute the critical front end of an access solution [1]. Some of the physiological signals that have been harnessed to develop noninvasive access pathways are electrical [2], [3] and hemodynamic activity [4], [5], [6] of the brain, or the electrodermal response of the skin [7], [8]. The aim of physiological signal based access solutions is to translate the sensed physiological signal into functional communication. Emerging access technologies are thoroughly discussed in [1].

A. Applications of Infrared Thermal Imaging Infrared thermal imaging has been used extensively in military, industry, and security applications [10], remote sensing [13], agriculture, plant physiology, and ecophysiology [14], geology [15] and medicine [12], [10]. With the advent of superior hardware technology, infrared thermal cameras with spatial resolution as high as 76800 pixels per image and thermal sensitivity to 0.05°C have been manufactured [16]. Advancements in signal and image processing software have allowed quick and convenient processing and exchange of thermal imaging data. Infrared thermography has been widely deployed in health research, including, for example, breast cancer detection [17], [18], brain surgery [19], [20], heart surgery [21], diagnosis of vascular disorders [22], arthritis [23], pain assessment [24] and post-surgical follow-up in ophthalmology [25]. Recently, Murthy and Pavlidis non-invasively measured human breathing using infrared imaging and a statistical methodology based on multinormal distributions, the method of moments, and Jeffreys divergence measure [26].

978-1-4244-3878-5/09/$25.00 ©2009 IEEE Funding support for this research is provided by Natural Sciences and Engineering Research Council of Canada (NSERC)

978-1-4244-3878-5/09/$26.00 ©2009 IEEE

INFRARED THERMAL IMAGING

1

recorded temperatures [10]. Each participant was cued to open his or her mouth and to hold it ajar for one second before closing the mouth. Participants were given an auditory prompt upon every open and close action. The end of each mouth closing was followed by a 3 second rest before the onset of the next mouth opening. The participants were instructed to maintain a constant head position, so that their mouth movement stayed within the camera’s field of view.

B. Potential Access Technologies based on Infrared Thermal Imaging We believe that infrared thermography can potentially be used in the development of new access technologies. Recently, Nhan and Chau examined the baseline characteristics of facial skin temperature, as measured by dynamic infrared thermal imaging, to gauge its potential as an emotion detector for nonverbal individuals with severe motor impairments [27]. That study was a first step toward detecting emotional thermoregulation as detectable changes in facial skin temperature that result from some emotional or mental intervention [27]. While a skin temperature based emotion detector can be a valuable access pathway for the population who do not have any voluntary motor ability (e.g., locked-in syndrome patients), infrared thermal imaging can also be utilized to develop access technologies that can be voluntarily controlled by the patients. In the present paper, we propose an access technology that can be activated by user’s voluntary mouth opening. By proposing a binary switch that can be activated by voluntary mouth opening, we attempt to exploit the fact that the temperature inside the human mouth is generally higher than that of the surroundings. A relatively constant temperature is always maintained within the human body [10], and therefore localized temperature changes due to mouth opening and closing can serve as a switch. A simple binary switch can open many doors to a person with severe motor disabilities. For example, he/she can use it to select characters from a scanning keyboard (typing), or make his/her wheelchair move (mobility).

B. Video Processing Algorithm Fig. 1 shows a schematic of our algorithm for detecting mouth openings from the thermal video data. The system consisted of three main components, namely face segmentation, motion and intensity analyses, and filtering non mouth objects. Each component will be discussed below. To begin, the boundary pixels of each video frame (the first and last pixels of every column and every row) were set to zero to detach objects that may be connected to the borders. 1) Face segmentation In addition to the participant’s head and facial region, other body parts such as the participant’s neck, thorax and upper limbs also appeared in the videos. For the participants with disability, parts of their wheelchairs were also captured on thermal video. Objects in the background, and in a couple of instances people moving around the participant were also recorded. It was thus essential to segment the participant’s face region from all other non-target body parts and objects. Each frame of the video was binarized. Given that facial temperature distributions vary within and among individuals [29], we adopted Otsu’s method to determine an adaptive rather than fixed intensity threshold which minimized, on a frame by frame basis, the intra-class variance of the grayscale values of the pixels to be binarized [30]. The binarized frames were then morphologically opened with a disk structuring element of radius 5 pixels to remove small objects, break thin connections, remove thin protrusions, and smooth object contours [31]. In the resulting image, the object with maximum area (presumably the face region) was retained and the object’s

If one can voluntary control mouth open and close action, sip and puff technology, EMG based switches, and computer vision based switches can also be used. The advantage of the proposed thermography based access technology over sip and puff and EMG based switches is that it is non-invasive and non-contact, i.e., does not require attachment of any sensor or external object to the user. Hence it is more hygienic and safe, as the risk of choking is also eliminated. Its advantage over visible light computer vision based access pathways is that it is independent of lighting/color and can thus be used both night and day, indoor and outdoor. III.

METHODS

A. Instrumentation and setup A THERMAL-EYE 2000B thermal video camera by L-3 Communications with thermal sensitivity ≤ 100 mK [28] was connected via an NTSC to USB TV convertor (Dazzle Multimedia). Videos were recorded as 240×320 AVI files (30 frames per second) and processed offline in MATLAB & Simulink (version R2007b). Participants were comfortably seated within a laboratory environment. Those with disability remained in their wheelchairs. The thermal camera was positioned anterior and lateral to the participant at a 45° angle. This camera location was chosen over the often-used frontal view, keeping in mind the eventual application as an access switch where the user’s field of view ought to be unobstructed. In the 45° angle condition, infrared thermograms only exhibit a small error in

Figure 1. Components of the mouth detection algorithm for the proposed access solution.

2

interior holes were filled by morphological closing with a disk structuring element of radius 20 pixels. The camera-user distance and the user’s head size affect the dimension of the above mentioned structuring elements. In a real life application, the camera will be mounted on the user’s wheelchair at a fixed distance from the user’s face. Hence, once the appropriate parameters are selected in the initial calibration, they do not need to be changed for subsequent use. An example of a segmented face region is depicted in Fig. 2(c).

and moving and were therefore retained subsequent to the motion and thermal intensity filters. An example is the forehead, which according to the literature, is the warmest part of the human body with a temperature (34.5 °C) close to that inside the mouth [33]. Therefore motion of the forehead may result in a false positive. To deal with these false positives, we deployed a series of additional filters based on morphology, size variation between frames, and facial anthropometry. Objects that did not meet the following morphological conditions were deemed as false positives and removed.

2) Motion and Intensity Analyses All subsequent processing was applied to the intensity image and confined to the identified face region. The region of interest (ROI) was the participant’s mouth and the task of interest was mouth opening. A combination of motion tracking and temperature thresholding was used to perceive mouth opening.

1. 2. 3.

The first condition rejects objects which are either too small or too large to be candidate mouth openings. Likewise, the second condition removes regions that are too elongated to qualify as mouth regions while the third condition eliminates hollow regions as the mouth is expected to be solid. The constants in these morphological filters were selected to resemble the shape of the open mouth and were empirically defined. In addition, objects whose size varied less than 25% between the current frame and the frame occurring ten frames earlier were considered static warm facial regions (e.g., forehead, chin, around the eyes, neck) and were also discarded. Finally we exploited the fact that facial anatomy is static (i.e., unlikely to change over time). Based on human face anthropometry, the mouth is located in the lower half of the menton-sellion length [34], [35]. When we partitioned the facial ROI along its major axis into four strips, we noticed that indeed the mouth was usually located in the second strip from the bottom. With this anthropometric filter, we dismissed candidate ROIs outside of the second facial quarter. Fig. 2(b)(f) demonstrate the action of the different video processing modules.

Since mouth opening involves motion, optical flow was utilized to estimate the direction and speed of motion from one video frame to the next using the Horn- Schunck method [32]. Motion vectors in each frame of the video sequence were computed by solving the optical flow constraint equation (1)

I xu + I yv + It = 0 where I x ,

I y and I t are the spatiotemporal image brightness

derivatives, u is the horizontal optical flow and v is the vertical optical flow. By assuming that the optical flow is smooth over the entire image, the Horn-Schunck method T computes an estimate of the velocity field, [u , v] , that minimizes this equation: 2

2

2

2 ⎧⎛ ∂u ⎞ ⎛ ∂u ⎞ ⎛ ∂v ⎞ ⎛ ∂v ⎞ E = ∫∫ (I x u + I y v + I t ) dxdy + α ∫∫ ⎨⎜ ⎟ + ⎜⎜ ⎟⎟ + ⎜ ⎟ + ⎜⎜ ⎟⎟ ⎩⎝ ∂x ⎠ ⎝ ∂y ⎠ ⎝ ∂x ⎠ ⎝ ∂y ⎠

⎛ ∂u ⎞ ⎜ ⎟

2

30 pixels < Area < 150 pixels Eccentricity ≤ 0.9. Area of object > 0 .5 Area of bounding box

⎫⎪ ⎬dxdy ⎪⎭ (2)

⎛ ∂u ⎞ ⎜⎜ ⎟⎟

In this equation ⎝ ∂x ⎠ and ⎝ ∂y ⎠ are the spatial derivatives of the optical velocity component u , and α scales the global smoothness term [32]. Motion vectors with velocity magnitude exceeding the mean velocity (i.e., the average of velocity magnitudes across the most recent five frames) per frame across time were retained, yielding a motion mask.

C. Deriving truth sets for algorithm evaluation To facilitate algorithm evaluation, a truth set was prepared manually for each recorded thermal video. The truth set contained the frame numbers corresponding to the beginning and ending of each mouth opening, the end points of the line maximally spanning the width of the mouth at the onset of opening and the end points of the line maximally spanning the height of the mouth when fully ajar. This truth set served as the gold standard for automatic algorithm evaluation, i.e., calculating sensitivity and specificity. A true positive was defined as the detection of a ROI temporally within the range of frames corresponding to a gold standard mouth opening, and spatially situated within the bounding box defined by the endpoints extracted above. All other detected objects were considered false positives. A mouth opening that was missed by the algorithm was counted as a false negative. A true negative occurred when there was no mouth opening and the algorithm concluded the same. Table 1 summarizes the truth set characteristics for each participant.

Warm zones inside the facial region were extracted by thresholding the segmented face with a scaled version of Otsu’s threshold [30] to favour higher intensity (i.e., warmer) pixels. The scale factor was empirically derived as: Scale factor = 3 – (mean intensity in face region – 150)/50 (3) and typically ranged from 2.5 to 3. This segmentation yielded a warm zone mask which served to detect instances of mouth opening. The intersection of this motion mask and the warm zone mask, introduced above, yielded all the regions of the face that were both warm and moving. 3) Filtering non-Mouth Objects Despite the combination of motion and thermal cues, the processed frames occasionally contained non-mouth objects (false positives) such as parts of the chin, forehead and the periorbital regions. These non-mouth objects were also warm

3

(a)

(b)

(c)

Figure 3. Sesitivity and specificity of the infrared thermography based mouth open-close detection system for ten particiants.

(d)

(e)

All participants (or their caregiver relative) provided written consent. The experimental protocol was approved by the research ethics board of the university and affiliated hospital. Detection of mouth opening is generally achieved with very high sensitivity and specificity (mean sensitivity of all participants: 88.5% ± 11.3; mean specificity of all participants: 99.4% ± 0.7). The exception is the poorer result for participant 10, which is mainly due to participant’s posture, frequent involuntary head rotation away from the camera, and suboptimal camera placement.

(f)

Figure 2. The action of different modules of the mouth opening detection algorithm, (a) Input thermal video frame, (b) Result of intial thresholding to segment the face region, (c) Segmented face, (d) Moving facial zones (shaded blocks prepresent parts of the frame that involved acceptable motion), (e) Intersection of motion mask and warm zone mask (before morphological, size variation, and anthropometric filtering), (f) Final output; detected mouth open is highlighted on the original video with a hollow box.

TABLE I.

V.

Infrared thermography has been successfully applied in many domains, such as security, remote sensing and medicine. We have proposed a novel application of this technology in the field of rehabilitation. Thermography can be used as a noncontact and non-invasive access pathway for individuals with motor impairments. In this paper we showed the potential of voluntary mouth opening as an access switch by reporting the results of automated thermal video analysis of eight healthy individuals and two persons with motor disability. Our results show the robustness of this potential access technology against user’s ethnicity, posture, motion, and background variations. This access technology can be modified to be triggered by the user’s most reliable access site, for example eye blinking or deep breathing.

TRUTH SET CHARACTERISTICS FOR EACH PARTICIPANT

Participant

Video length (sec)

Total Video frames

1 2 3 4 5 6 7 8 9* 10*

256 252 254 252 244 243 245 243 153 272

7662 7546 7621 7481 7424 7594 7664 7613 4592 8160

Actual # of mouth openings 50 50 50 50 50 50 50 50 30 15

ACKNOWLEDGMENT The authors would like to acknowledge Bloorview Kids Rehab and Ministry of Health and Long Term Care, Canada. The authors would also like to thank Mr. Russel Rasquinha and Ms. Denise Darmawikarta for their assistance in thermal video recording and preparation of the truth sets, respectively.

*Participant with severe disability.

IV.

CONCLUSION

RESULTS

REFERENCES

The performance of the proposed algorithm on the thermal video of ten participants is shown in Fig. 3. Eight able-bodied participants and two individuals with quadriplegia (one with a C1-C2 incomplete spinal cord injury and the other with severe spastic quadriplegic cerebral palsy) participated in this study.

[1]

[2]

4

K. Tai, S. Blain, and T. Chau, “A review of emerging access technologies for individuals with severe motor impairments,” Assistive Technology, vol. 20, pp. 204-219, 2008. E.W. Sellers, A. Kubler, and E. Donchin, “Brain–computer interface research at the university of South Florida cognitive psychophysiology

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13] [14]

[15] [16]

[17]

[18] [19]

[20]

[21]

[22]

[23] S.J. Spalding et al., “Three-dimensional and thermal surface imaging produces reliable measures of joint shape and temperature: a potential tool for quantifying arthritis,” Arthritis Research & Therapy, vol. 10, pp. R10, 2008. [24] C.L. Herry and M. Frize, “Quantitative assessment of pain-related thermal dysfunction through clinical digital infrared thermal imaging,” BioMedical Engineering OnLine, vol. 3, no. 19, pp. 1-14, 2004. [25] A.D. Rose and V. Kanade, “Thermal imaging study comparing phacoemulsification with the sovereign with WhiteStar System to the legacy with AdvanTec and NeoSoniX System,” American Journal of Ophthalmology, vol. 141, no. 2, pp. 322-326, 2006. [26] R. Murthy and I. Pavlidis, “Non-contact measurement of breathing function,” IEEE Engineering in Medicine and Biology Magazine, vol. 25, no. 3, pp. 57-67, 2006. [27] B.R. Nhan and T. Chau, “Infrared thermal imaging as a physiological access pathway: a study of the baseline characteristics of facial skin temperatures,” Physiol. Meas., vol. 30, pp. N23-N35, 2009. [28] GotoInfrared, “L-3 Communications: Thermal-Eye 2000B/300D,” GotoInfrared, 2007. [Online]. Available: http://gotoinfrared.com/L3.htm [Accessed: April 23, 2009]. [29] N. Zaproudina, V. Varmavuo, O. Airaksinen, M. Narhi, “Reproducibility of infrared thermography measurements in healthy individuals,” Physiological Measurement, vol. 29, no. 4, pp. 515-524, 2008. [30] N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, pp. 62-66, 1979. [31] R.C. Gonzalez, R.E. Woods, and S.L. Eddins, Digital Image Processing Using MATLAB. New Jersey: Pearson Prentice Hall, 2004. [32] J.L. Barron, D.J. Fleet, S.S. Beauchemin, T.A. Burkitt, “Performance of optical flow techniques,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Patter Recognition (CVPR), 1992, pp. 236-242. [33] S. Uematsu, “Symmetry of skin temperature comparing one side of the body to the other,” Thermology, vol. 1, pp. 4–7, 1986. [34] J.C. Bailar, E.A. Meyer, and R. Pool, Eds., Assessment of the NIOSH head-and face anthropometric survey of U.S. respirator users. Washington D.C.: The National Academic Press, 2007. [35] D. DeCarlo, D. Metaxas, M. Stone, “An anthropometric face model using variational techniques,” in Proceedings SIGGRAPH, 1998, pp. 6774.

laboratory: the P300 speller,” IEEE Transactions on Neurological Systems and Rehabilitation Engineering (Special Issue on BrainComputer Inerfaces), vol. 14, no.2, pp. 221-224, 2006. F. Piccione et al., “P300-based brain computer interface: Reliability and performance in healthy and paralyzed participants,” Clinical Neurophysiology, vol. 117, no. 3, pp. 531-537, 2006. S.M. Coyle, T.E. Ward, and C.M. Markham, “Brain-computer interface using a simplified functional near-infrared spectroscopy system,” J. Neural Eng. vol. 4, pp. 219-226, 2007. R. Sitaram et al., “Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface,” NeuroImage. vol. 34, pp. 1416-1427, 2007. M. Naito, Y. Michioka, K. Ozawa, Y. Ito, M. Kiguchi, T. Kanazawa, “A communication means for totally locked-in ALS patients based on changes in cerebral blood volume measured with near-infrared light,” IEICE Transactions on Information and Systems, E90D(7), pp. 10281037, 2007. S. Blain, A. Mihailidis, and T. Chau, “Assessing the potential of electrodermal activity as an alternative access pathway,” Medical Engineering & Physics, vol. 30, no. 4, pp. 498-505, 2008. R. Tsukahara and H. Aoki, “Skin potential response in letter recognition task as an alternative communication channel for individuals with severe motor disability,” Clinical Neurophysiology, vol. 113, pp. 1723-1733, 2002. N. Memarian, A.N. Venetsanopoulos, and T. Chau, “Infrared thermography as an access pathway for individuals with severe motor impairments,” Journal of NeuroEngineering and Rehabilitation, 2009, vol. 6, no. 11, 2009. B.F. Jones, “A reappraisal of the use of infrared thermal image analysis in medicine,” IEEE Transactions on Medical Imaging, vol. 17, no. 6, pp. 1019-1027, 1998. J. Lupo and R. Balcerak, “The physical basis of thermal imaging,” in Proceedings of the 22nd Annual IEEE Engineering in Medicine and Biology Society Conference, 2000. B.F. Jones and P. Plassmann, “Digital infrared thermal imaging of human skin,” IEEE Engineering in Medicine and Biology, vol. 21, pp. 41-48, 2002. S.G. Burnay, T.L. Williams, and C.H. Jones, Applications of Thermal Imaging, Taylor & Francis, 1988. H.G. Jones, “Application of thermal imaging and infrared sensing in plant physiology and ecophysiology,” Advances in Botanical Research, vol. 41, pp. 107-163, 2004. K. Watson, “Geologic applications of thermal infrared images,” Proceedings of the IEEE, vol. 63, no. 1, pp. 128-137, 1975. FLIR Systems, “NEW! ThermaCAM® P65HSV”, FLIR Systems, 2005. [online]. Available: http://www.flir.ca/cameras/infrared_camera_copmplete/1044/ [Accessed: June, 2008]. H. Qi and N.A. Diakides, “Thermal Infrared Imaging in Early Breast Cancer Detection - A Survey of Recent Research,” in Proceedings of the 25th Annual IEEE Engineering in Medicine and Biology Society Conference, 2003, pp. 17-21. M. Gautherie, Atlas of breast themogmphy with specific guidelines for eramination and interpretation. Milan, Italy: PAPUSA, 1989. Y. Okada, T. Kawamata, A. Kawashima, T. Hori, “Intraoperative application of thermography in extracranial-intracranial bypass surgery,” Neurosurgery, vol. 60(Suppl 2), pp. 362-365, 2007. J.C. Watson, A.M. Gorbach, R.M. Pluta, R. Rak, J.D. Heiss, E.H. Oldfield, “Realtime detection of vascular occlusion and reperfusion of the brain during surgery by using infrared imaging,” J Neurosurg. vol. 96, pp. 918-923, 2002. M. Madjid, J.T. Willerson, and S.W. Casscells, “Intracoronary thermography for detection of high-risk vulnerable plaques,” J Am Coll Cardiol. Vol. 47(Suppl 8), pp. C80-C85, 2006. F. Ring and R. Harding, “Infrared thermal imaging in peripheral vascular diseases,” in World Congress on Medical Physics and Biomedical Engineering, 2000.

5