VibroCV: A Computer Vision-Based Vibroarthrography Platform with Possible Application to Juvenile Idiopathic Arthritis* Andrew D. Wiens, Student Member, IEEE, Sampath Prahalad, and Omer T. Inan, Senior Member, IEEE
Abstract— Vibroarthrography, a method for interpreting the sounds emitted by a knee during movement, has been studied for several joint disorders since 1902. However, to our knowledge, the usefulness of this method for management of Juvenile Idiopathic Arthritis (JIA) has not been investigated. To study joint sounds as a possible new biomarker for pediatric cases of JIA we designed and built VibroCV, a platform to capture vibroarthrograms from four accelerometers; electromyograms (EMG) and inertial measurements from four wireless EMG modules; and joint angles from two Sony Eye cameras and six light-emitting diodes with commercially-available off-the-shelf parts and computer vision via OpenCV. This article explains the design of this turn-key platform in detail, and provides a sample recording captured from a pediatric subject.
I. INTRODUCTION First studied by Blodgett in 1902 [1], the diagnostic potential of knee-joint sounds has been demonstrated in several different studies. For example, in 1987, McCoy et al showed that the amplitude of knee joint sounds may decrease after meniscal resection when 86% of a population of 170 subjects with meniscal tears exhibited changes in the vibroarthrographic signal [2]–[4]. Since then, several research groups have developed vibroarthrographic signal processing techniques [3], [5], [6]. Others have focused on systems approaches. Our group has applied vibroarthrography to assess progress in joint rehabilitation post-injury with sit-to-stand and flexion/extension exercises of the knee [7], and recently we developed a wearable system to measure knee joint sounds in the context of different physical activities [8]. Diseases studied with vibroarthrography have involved cartilage degeneration such as osteoarthritis [3]. However, to our knowledge, the effect of juvenile idiopathic arthritis (JIA) on the vibroarthrographic signal has not been studied even though the disease tends to degrade synovial joints like the knee when untreated. Known as juvenile rheumatoid arthritis until relatively recently, JIA is any form of arthritis of unknown cause that lasts more than 6 weeks and has an onset before age 16 [9]. It is the most common type of arthritis in children [9], and affects 7-21 out of every 100,000 children in the US and Northern Europe. The disease can persist into adulthood, has adverse long-term effects on joints, and does not respond to standard therapy in 30% of patients [10]. Furthermore, while the main goal of treatment is to achieve disease remission, approximately 30-50% of patients relapse *This work was supported by the Center for Pediatric Innovation, a research partnership between Children’s Healthcare of Atlanta and the Georgia Institute of Technology. A. D. Wiens and Omer T. Inan are with Electrical and Computer Engineering at Georgia Institute of Technology, Atlanta, GA 30308 USA (314-610-9194;
[email protected]).
978-1-4577-0220-4/16/$31.00 ©2016 IEEE
after discontinuation of methotrexate, the most common standard therapy [11]. In response to the recent introduction of several highlysuccessful anti-TNF (tumor necrosis factor) biologic drugs such as infliximab, adalimumab, etanercept, and golimumab, US guidelines now recommend switching to biologic therapy after four months of unsuccessful treatment with standard therapy [10]. Still, this is a long time for most juvenile patients to wait. It is exacerbated by the fact that the process of iterating toward an effective treatment plan typically takes many months and requires several costly return visits to the rheumatologist for subjective evaluation. In severe cases, such as a 2-year-old girl who exhibited JIA-induced growth retardation with a height 3 standard deviations below the mean for that age [10], several months of ineffective treatment is entirely unacceptable.
SIDE
FRONT
WIRELESS EMG & IMU ACCELEROMETER
120 FPS CAMERA KNEE ANGLE
LED MARKER Figure 1. Diagram of the experiment and instruments with a pediatric subject. Four ultra-low noise accelerometers are configured as contact microphones on the knees. A data acquisition unit (not pictured) records the knee sounds while four wireless modules capture electromyograms (EMG) and 3-axis acceleration on the quads and calves. Simultaneously, two high-speed cameras operating at 120 frames per second (FPS) record the angles of the left and right knees via six 10mm bright green light-emitting diodes (LED) powered by lithium coin cells. Custom software running on a laptop (not pictured) determines the location of the LEDs in each video frame, syncs the information from all sensors, and streams data to the file system. Sampath Prahalad is with the Department of Pediatrics at Emory University School of Medicine, Atlanta, GA 30322 USA.
4431
B)
A) 4x EMG/IMU EMG/IMU EMG/IMU EMG/IMU 4x EMG/IMU ACCEL. EMG/IMU EMG/IMU
NI DAQ
OpenCV
TRIGNO BASE
TRIGNO DLL
USB 2.0
NI DAQ DLL
VibroCV
CL-EYE DLL
SSD
2x EMG/IMU CAMERA
Figure 2. A) Block diagram of the necessary hardware and software for VibroCV. Data from all sensors are sent to VibroCV via dynamically-loaded libraries (DLLs) and streamed to a fast solid-state disk (SSD) for later analysis. OpenCV provides the libraries for graphics and computer vision capabilities. B) Photo of the hardware loaded onto a cart for a human-subjects research (HSR) study of juvenile idiopathic arthritis (JIA) in the clinic.
To reduce the amount of time it takes to arrive at an effective treatment plan after initial diagnosis, there have been efforts to uncover personalized biomarkers of JIA. Biomarkers in blood, urine, and saliva such as S100 proteins and MRP8/14 have been identified that predict remission and response to treatment with varying degrees of success [11]. However, all require sample collection and lab tests. Vibroarthrography could provide a noninvasive biomarker of JIA disease that can be easily measured outside of the clinic and without trained personnel. If vibroarthography can provide information about JIA disease state, it would constitute the first quantitative metric that can provide low cost day-to-day feedback between visits to the clinic. To study the usefulness of vibroarthography for JIA assessment we developed VibroCV, a fully-integrated system for capturing time-synchronized vibroarthrographic and related physiologic signals with a standard PC. This paper describes the design, implementation, and testing of VibroCV and briefly describes its future use in our pediatric humansubjects research study of JIA. II. METHODS This preliminary human-subjects research study was approved by the Institutional Review Boards of Georgia Institute of Technology and Emory University. A. Equipment An overview of the technique is shown in Figure 1, and a block diagram of VibroCV appears in Figure 2a. All hardware was connected to a laptop PC (Latitude E6000 series, Dell Inc., Round Rock, TX) running Windows 7 (Microsoft, Redmond, WA). Four wireless electromyogram (EMG) sensors with built-in 3-axis linear inertial measurement units (IMUs) were used to capture muscle activity and lower body movements (Trigno Lab, Delsys Inc., Natick, MA). In particular, the IMUs are useful for joint angle estimation when marker based joint angles are not available, i.e. when walking. The EMG receiver provided a USB connection to a PC. Four single axis analog miniature low-noise piezoelectric accelerometers were used for capturing vibrations of the joints and recording the vibroarthrogram signal (3225F7, Dytran Instruments Inc., Chatsworth, CA). These accelerometers were chosen because
they are Integrated Electronic Piezoelectric (IEPE) sensors with low mass (0.85 grams), wide bandwidth (2 Hz—10 kHz, ±10%), and low noise (700 µgrms) properties. The accelerometers were attached to the lateral sides of each patella with elastic kinesiology tape (Kinesio Tex Gold, Kinesio, Albuquerque, NM) and recorded on a PC with a USB data acquisition unit (USB-4432, National Instruments Corporation, Austin, TX). This unit was chosen for its high resolution (24 bits), high sample rate (102.4 kS/s), ANSI C API (NI-DAQmx), and built-in IEPE capability, which allowed us to eliminate the need for a separate 4-channel analog IEPE signal conditioner. Two USB cameras were used to capture joint angles with a marker-based approach (PlayStation Eye, Sony Computer Entertainment, Tokyo, Japan). This camera was chosen because its light sensitivity is optimal for LED marker-based capture (i.e. PS Move) and it has a high framerate (120 frames per second at 320 x 240 pixels resolution), fixed-focus lens, wide field of view (75° for wide zoom setting), a third-party high-framerate driver with a dynamically-loaded library (DLL) and example code (CLEye, Code Laboratories, Henderson, NV), and low cost. An example image from the left camera is shown in Figure 3c. Each camera was mounted on a tripod with cyanoacrylate glue (LOC1365882, Loctite, Düsseldorf, Germany) and a ¼-20 nut (08424, The Home Depot, Atlanta, Georgia). All hardware was installed on a cart for quick clinical setup as shown in Figure 2b (FR1006C-COM, Mainstays, Bentonville, AR). A generic stool with adjustable height was also used for sit-tostand exercises. B. LED Markers with Snap Connectors To capture joint angles, LED markers were made as shown in Figure 3a and placed laterally on the femur, tibia, and at the knee joint as shown in Figure 1. The markers were made from available commercial off-the-shelf electronics components. Each marker had a bright green 10mm diffused LED (844, Adafruit Industries, New York, NY) connected to a switch (2750409, RadioShack, Fort Worth, TX) and a lithium coin cell battery (CR2032, Dantona Industries, Wantagh, NY) directly without a resistor. Wires were glued to the positive and negative terminals of the coin cell battery with conductive graphite glue (6400146, RadioShack, Fort Worth, TX). The coin cell was then wrapped in 2:1 heat-shrink tubing (21-8765,
4432
A) CYANOACRYLATE GLUE
ON/OFF SWITCH ECG SNAP
1000 MCD 10 mm DIFFUSED GREEN LED CR2032 COIN CELL & HEATSHRINK
The data samples received from each API were timesynchronized using QueryPerformanceCounter to obtain high-resolution time stamps (< 1 μs) from the Windows API. All data were then streamed to disk with corresponding time stamps in raw binary files for later processing. A solid-state hard drive allowed continuous writing to the file system at high bandwidth, and the total CPU usage by VibroCV was less than 25% across two hyper-threaded cores (Ivy Bridge, Intel Corporation, Santa Clara, California). A second Visual C++ application was written to convert these binary files to MATLAB format (MathWorks, Natick, MA) and to compress the raw camera frames with H.264 (Windows 7 built-in codec) to save disk space for data storage after the experiments were completed.
ECG ELECTRODE FOR SKIN ADHESION B)
C)
and IMU time series were obtained from the Trigno Lab sensors with EMGworks and the Trigno Digital SDK (Delsys Inc., Natick, MA). OpenCV (Open Source Computer Vision, Itseez, San Francisco, CA) was used to find the location of the three LED markers in each video frame. The locations of the markers were then used to calculate the joint angle.
D)
Figure 3. A) Schematic of the LED markers. Inexpensive off-the-shelf components are soldered into a simple on-off circuit and glued together to form a compact package. An ECG snap connector and electrode are used to provide convenient and safe mechanical attachment to the subject’s skin. B) Photo of completed LED markers. C) Raw image from the camera during one recording in a bright room. D) One joint angle as displayed on-screen in VibroCV. The software locates the three brightest spots in the image using OpenCV and computes the inner angle in degrees. The three LEDs are bright and distinct even with overhead fluorescent lights turned on.
MCM Electronics, Springboro, OH), and a snap connector from electrocardiogram (ECG) test leads (ECG-PRO-3WAY-CABLE, Olimex Ltd, Plovdiv, Bulgaria) was attached with cyanoacrylate glue (LOC1365882, Loctite, Düsseldorf, Germany) to one side. The snap connectors fit to disposable ECG electrodes (2660, 3M Company, Maplewood, MN) to provide easy mounting to the body. The electrodes were not electrically connected to a circuit. The switch and LED were also attached with cyanoacrylate glue to the other side of the coin cell. Cyanoacrylate glue is very strong and results in a very durable light marker when applied liberally and allowed to dry overnight. Two examples of finished LED markers are shown in Figure 3b, and an example of their use in practice is shown in Figure 3c. C. Software VibroCV was implemented as a multithreaded C++ application and compiled for Win32 (Visual C++, Microsoft Corporation, Redmond, WA). Raw video frames were captured directly from each camera at a nominal rate of 120 Hz with CL-Eye (Code Laboratories, Henderson, NV). Accelerometer waveforms from the data acquisition unit were streamed to VibroCV via the NI-DAQmx ANSI C library (National Instruments Corporation, Austin, TX), which was configured to sample each accelerometer at 102.4 kHz. EMG
Finally, OpenCV was used to create separate GUI windows showing live previews of the cameras and scrolling plots of the EMG, IMU, and accelerometer time series. Green circles around the three LED markers, two magenta lines connecting the markers, and white text of the computed angle in degrees were drawn on each camera preview with OpenCV as shown in Figure 3d. This helped the study coordinator check each camera’s field of view and that OpenCV was registering each marker when the cameras were set up in the clinic. III. RESULTS AND DISCUSSION A test vibroarthrogram of a healthy pediatric subject’s knee as captured with VibroCV appears in the bottom of Figure 4b, and the knee joint angles obtained with the markerbased computer vision method appear in the top. The high framerate of the cameras used in this work provided the benefit of true 120 Hz measurements of joint angles. The low resolution of the cameras at that framerate (320 x 240 pixels) resulted in some jitter because single-pixel changes in the estimated location of the markers caused relatively larger changes in the estimated joint angle than higher resolution 170
90 +4 m/s2
-4 m/s2
16.7 s
Figure 4. Knee-joint vibrations from the four accelerometers (bottom) and the joint angles (top) of the left knee (red) and right knee (blue) during a trial sit-to-stand recording of a male juvenile subject without JIA. These waveforms were recorded with VibroCV and show the capability of the system. Clicks and pops are not present in these time series presumably because the subject was healthy. Future recordings will be made on male and female children with and without JIA.
4433
cameras would produce. However, low-pass filtering of the 120 Hz joint angle measurements would be appropriate given the low frequency of sit-to-stand exercise (< 1 Hz) and would adequately suppress such high-frequency jitters. VibroCV was also very convenient to use with pediatric patients, and the bright LEDs engaged young subjects much more than the wireless IMU system in our tests. One of our initial test subjects was very excited to wear the LED markers and watch the output of the camera preview as shown in Figure 3d while he performed the experiment. Joint angles with LED markers could thus allow easier data collection with young subjects, which could improve the quality of the collected data if the subjects are more cooperative. The computer vision approach could also provide an excellent opportunity for children to become excited about STEM at an early age by seeing it in practice during their appointment, thus broadening the potential educational impacts of the work. Two healthy pediatric subjects were recorded during initial testing of VibroCV, and knee joint angles and sounds of one healthy male pediatric subject (9 years old) are shown in Figure 4. Because the patient was healthy, the vibroarthrogram does not reveal distinct peaks that would be expected of an inflamed joint. The joint angles extracted from frames of the two cameras (top) corresponded to the joint sounds from the four accelerometers (bottom) as evidenced by low-frequency oscillations in the accelerometer signals. Joint auscultation for flexion/extension of the wrist of an adult male with JIA was also recorded and appears in Figure 5 in the time domain (top) and time-frequency domain (bottom). Computer-vision joint angles were not available for this recording because it was taken prior to the completion of VibroCV. It also cannot be compared directly to pediatric vibroarthrograms due to the use of a different joint (wrist vs. knee) and the subject’s age. Nonetheless, the diseased wrist joint produced large clicks during each flexion/extension cycle, which has been seen with other types of joint problems in vibroarthrography literature [4]. IV. CONCLUSION The design, implementation, and a preliminary recording obtained with a new vibroarthrography platform based on computer vision and accelerometers called VibroCV were presented. Trial experiments with two subjects subjectively validated the performance of the system. Optical markerbased joint angles appear to offer significant benefits over IMU-based joint angles, namely, more convenient placement on pediatric subjects. Quantitative comparisons of the optical system to traditional goniometers or IMU systems could be a topic of a future investigation. Next steps include characterization of the vibration signal characteristics of noise from sensor rubbing and kinesiology tape adhesive from our hardware platform as described in the vibroarthrography literature where researchers obtained superior results with accelerometers compared to air microphones by filtering mechanical noise out of the signals after characterizing the shape of unwanted noise in the time-frequency domain [3].
+10 m/s2 -10 m/s2 25 kHz 0 Hz
23 s Figure 5. Time series (top) and short-time Fourier transform (bottom) of joint sounds of the wrist of an adult male subject with JIA during flexion-extension movements. This figure is shown for reference as an example of the clicks that can occur in a diseased joint, although it cannot be compared directly to the knee example since it is a different joint and the subject was an adult.
Finally, VibroCV will be used in the clinic to study the feasibility of vibroarthrography for assessment of patients with JIA. Clicks identified in a similar wrist auscultation signal from an adult with JIA suggest that pediatric JIA patients may exhibit similar changes at the knee, especially because the knee is a larger joint than the wrist. Vibroarthrography could potentially improve treatment for patients with juvenile idiopathic arthritis by providing feedback between doctor visits to allow the patient’s physician to respond to disease changes and modify the treatment plan more quickly. This upcoming pilot study should be relevant to any rheumatologist, biomedical engineer, or patient suffering from JIA or related inflammatory diseases. REFERENCES [1]
W. E. Blodgett, “Auscultation of the Knee Joint,” Bost. Med. Surg. J., vol. 146, no. 3, pp. 63–66, 1902. [2] G. F. McCoy, J. D. McCrea, D. E. Beverland, W. G. Kernohan, and R. a Mollan, “Vibration arthrography as a diagnostic aid in diseases of the knee. A preliminary report.,” J. Bone Joint Surg. Br., vol. 69, no. 2, pp. 288–293, 1987. [3] Y. F. Wu, S. Krishnan, and R. M. Rangayyan, “Computer-aided diagnosis of knee-joint disorders via vibroarthrographic signal analysis: A review,” Crit. Rev. Biomed. Eng., vol. 38, no. 2, pp. 201–224, 2010. [4] S. C. Abbott and M. D. Cole, “Vibration arthrometry: a critical review.,” Crit. Rev. Biomed. Eng., vol. 41, no. 3, pp. 223–42, 2013. [5] Y. Shen, R. M. Rangayyan, G. D. Bell, C. B. Frank, Y. T. Zhang, and K. O. Ladly, “Localization of knee joint cartilage patholog by multichannel vibroarthrography,” Med. Eng. …, vol. 17, no. 8, pp. 583– 594, 1995. [6] S. Krishnan, R. M. R. Rangayyan, G. G. D. Bell, C. B. Frank, and K. O. Ladly, “Adaptive filtering, modelling and classification of knee joint vibroarthrographic signals for non-invasive diagnosis of articular cartilage pathology,” Med. Biol. …, vol. 35, no. 6, pp. 677–684, 1997. [7] C. Teague, S. Hersek, H. Töreyin, M. L. Millard-Stafford, M. L. Jones, G. F. Kogler, M. N. Sawka, and O. T. Inan, “Novel Methods for Sensing Acoustical Emissions from the Knee for Wearable Joint Health Assessment,” IEEE Trans. Biomed. Eng., In Press, 2016. [8] H. Toreyin, S. Hersek, C. Teague, and O. Inan, “A Proof-of-Concept System to Analyze Joint Sounds in Real Time for Knee Health Assessment in Uncontrolled Settings,” IEEE Sens. J., In Press, 2016. [9] S. Rangaraj, “Diagnosing juvenile idiopathic arthritis,” Paediatr. Child Health (Oxford)., vol. 1, no. 12, pp. 552–557, 2007. [10] F. La Torre, M. Cattalini, B. Teruzzi, A. Meini, F. Moramarco, and F. Iannone, “Efficacy of adalimumab in young children with juvenile idiopathic arthritis and chronic uveitis: a case series.,” BMC Res. Notes, vol. 7, no. 1, p. 316, 2014. [11] C. L. Duurland and L. R. Wedderburn, “Current developments in the use of biomarkers for juvenile idiopathic arthritis.,” Curr. Rheumatol. Rep., vol. 16, no. 3, p. 406, 2014.
4434
