Available online at www.sciencedirect.com
ScienceDirect Procedia Computer Science 90 (2016) 151 – 156
International Conference On Medical Imaging Understanding and Analysis 2016, MIUA 2016, 68 July 2016, Loughborough, UK
Detection, reconstruction and segmentation of chronic wounds using Kinect v2 sensor Damir Filko*, Robert Cupec, Emmanuel Karlo Nyarko Faculty of Electrical Engineering, Kneza Trpimira 2B, 31000 Osijek, Croatia
Abstract The advent of inexpensive RGB-D sensors pioneered by the original Kinect sensor, has paved the way for a lot of innovations in computer and robot vision applications. In this article, we propose a system which uses the new Kinect v2 sensor in a medical application for the purpose of detection, 3D reconstruction and segmentation of chronic wounds. Wound detection is based on a per block classification of wound tissue using colour histograms and nearest neighbour approach. The 3D reconstruction is similar to KinectFusion where ICP is used for determining rigid body transformation. Colour enhanced TSDF is applied for scene fusion, while the Marching cubes algorithm is used for creating the surface mesh. The wound contour is extracted by a segmentation procedure which is driven by geometrical and visual properties of the surface. Apart from the segmentation procedure, the entire system is implemented in CUDA which enables real-time operation. The end result of the developed system is a precise 3D coloured model of the segmented wound, and its measurable properties including perimeter, area and volume, which can be used for determining a correct therapy and treatment of chronic wounds. All experiments were conducted on a medical wound care model. © 2016 2016The TheAuthors. Authors. Published by Elsevier © Published by Elsevier B.V.B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of MIUA 2016. Peer-review under responsibility of the Organizing Committee of MIUA 2016 Keywords: chronic wound; detection; reconstruction; segmentation; measurement; kinect v2
1. Introduction Chronic wounds heal very slowly and the healing process may further be prolonged if an ineffective treatment is used. Clinicians need an objective wound characterization method to decide if the current treatment is adequate or
* Corresponding author. Tel.: +385-(0)31-495-404; fax: +385-(0)31-224-605. E-mail address:
[email protected]
1877-0509 © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of MIUA 2016 doi:10.1016/j.procs.2016.07.022
152
Damir Filko et al. / Procedia Computer Science 90 (2016) 151 – 156
requires adjustments. Accurate wound measurement is an important task in chronic wound treatment because changes in the physical parameters of the wound are indicators of the healing progress. Wound measurement methods currently in use are mostly based on simple approaches for area measurement, using rulers and transparency tracing, or photography based non-contact systems. Such methods give a rough estimate of the total wound area and the evaluation depends on human experience; it is not objective and not eligible for validation of the wound healing process. The first step in creating a precise non-contact measurement system is having the ability to reconstruct the wound as a 3D model. This would facilitate complex analysis or measurement of wounds which is otherwise awkward and painful for the patient or inconvenient for clinicians. Furthermore, the reconstructed 3D model could be used for collaboration between clinicians and telemedicine. Wound measurement is typically divided into two main approaches: contact and non-contact. Contact based methods require measurements to be taken directly with wound contact1, 2. Non-contact methods require a recording device, whereby a standard camera is typically used. It is generally accepted that image based methods are a viable alternative to contact based methods, and have a maximum error of area measurement of about 10% depending on the wound location and the camera angle3. Other non-contact wound assessment systems use 3D reconstruction to enhance image based methods and gain more precise measurements. For example, two wound images taken from different angles can be used to generate a 3D mesh model4. Eykona, a system similar in principle and commercially available, generates more dense 3D representation5. Besides being very expensive, costing about $18000, Eykona has limitations with respect to the size and position of the wound it can reconstruct since the whole wound needs to be visible in a single frame. Lasers are also commonly used for 3D reconstruction in medical research6, 7, 8. Even though they have proven to be very precise, they are not easy to use and are very expensive. Furthermore, they also share the limitation as stereo vision systems: the whole wound must be visible in one frame. InSight by eKare uses a technology similar to the original Microsoft Kinect and is used in the study9 for comparison in measurement performance with SilhouetteStar. The new InSight system has the limitation regarding the visibility of the whole wound in a single frame. As can be seen, most of the systems have limitations regarding precision, ease of use and price, the latter of which limits the widespread use of most systems. However, the biggest limitation of all presented systems is that the whole wound must be visible from a single view. This constraint limits the wound’s size and location. The 3D wound reconstruction system proposed in this paper does not have such a constraint. Since it is based on scene fusion, the wound can be located on a larger area and on curved surfaces. Furthermore, an inexpensive Microsoft Kinect v2 sensor is used which costs about $150, making the system available to a much wider user base. In this paper, we propose a system for the detection, 3D reconstruction and segmentation of chronic wounds using a Kinect v2 sensor. The Kinect v2 sensor consists of a 1920 x 1080 resolution RGB camera and a 512 x 424 resolution depth camera, both at 30FPS. The developed system automatically detects wounds by analysing image blocks and utilizing colour histogram similarity and nearest neighbour approach. The 3D reconstruction is similar to the KinectFusion approach10 where Iterative Closest Point (ICP) algorithm is used for determining rigid body transformation. Colour enhanced Truncated Signed Distance Function (TSDF) is applied for scene fusion and the Marching cubes algorithm is used for creating a surface mesh. The wound segmentation algorithm is driven by changes in surface geometry and colour with the main goal of separating the wound tissue from the surrounding healthy tissue (skin). Furthermore, except for segmentation, the entire system is implemented in CUDA, which enables real-time operation. The proposed system provides a high resolution 3D coloured model of the wound and its surrounding tissue suitable for further analysis. Analysis implemented in the system includes measurement of wound perimeter, area and volume, which clinicians could use to develop more responsive therapy and treatment for chronic wounds. The system proposed herein was fully developed in C++ and CUDA v7.0. Furthermore, all experiments were run on a laptop computer with Intel Core i7 4710HQ CPU, Nvidia 860M GPU and Windows 8.1 Pro x64 operating system. In addition to the Kinect v2 sensor and the PC computer, the materials used also included Saymour II wound care model by VATA Inc. which was used for experimental evaluation of the developed system. The rest of the paper is structured as follows. In Section 2 an overview of the wound detection and 3D reconstruction subsystems is provided, while Section 3 elaborates on the segmentation subsystem. Section 4 explains the methods used for measurement of wound perimeter, area and volume. Finally, this paper is concluded with Section 5, which comments on the results.
Damir Filko et al. / Procedia Computer Science 90 (2016) 151 – 156
2. Wound detection and reconstruction The main objective of the wound detection subsystem is to find the centre of the wound the user wishes to reconstruct. The system is based on the concatenated (1D) histograms of each colour channel in HSV colour space and k-Nearest Neighbour algorithm (kNN). To this end, an initial database of classified regions is created and used in the classification process. The image is divided into a uniform grid of 16 x 16 square cells or blocks. A colour histogram is generated for each block and, with the aid of the kNN algorithm (k = 7) and the reference database, each image block is classified. For a given test histogram (or image block), the kNN search algorithm returns the k vectors in the reference database that are closest to the test histogram. The Bhattacharyya distance11 is used to calculate the similarity between the test and reference histograms. The probability of the test histogram being classified as a wound can be estimated as the ratio of the number, n, of k-nearest vectors classified as wound divided by k. As a result, each image block (or subimage) of the original image is assigned a probability of being a wound. At the end of the classification process, a threshold is applied and only those regions with a probability greater than the threshold are marked as wounds. A threshold of 0.89 was used in all the experiments. Circles are then drawn around the regions marked as wounds in order to ease the selection of a region. The choice of the threshold value (0.89) was, at this stage, obtained experimentally. Extensive experimental and statistical analysis still remains to be performed. The CUDA implementation of the wound detection subsystem has an average performance of 8.68ms per frame. Fig. 1. shows an example test image and the results obtained using HSV color histograms.
a) b) c) Fig. 1. (a) Test image with regions marked as wounds; (b) result of per block classification obtained using kNN and reference database, where the block's intensity is proportional to the probability of a block being a wound; (c) result of image thresholding.
The 3D reconstruction subsystem is divided into two phases: scene fusion and mesh generation. Scene fusion is the process where each captured RGB-D frame is integrated into a reconstructed volume. This process is further divided into two sub-phases: registration and integration. Registration determines the rigid body transformation between the model and the current frame and is based on the ICP algorithm. In order to use Kinect v2 data properly, a slightly modified ICPCUDA implementation12 is used for model-to-frame registration. The registration sub-phase performance was on average 8.73ms per frame. The integration sub-phase is based on the Truncated Sign Distance Function (TSDF)13. It is based on discretizing the volume into voxels of the same size. Each voxel is assigned an SDF value. Voxels with a positive SDF value are considered to be outside the object surface, while those with negative values are considered to be inside the object. Therefore, the object surface is the contour with the iso-value of 0. By truncating the SDF value to the interval [-1, 1], the truncated SDF (TSDF) is obtained. The integration sub-phase performance was on average 8.39ms per frame for a 256x256x256 volume of voxels. With colour integration enabled, the performance was on average 17.86ms. Given the TSDF volume gained by the scene fusion phase, Marching cubes algorithm14 is applied in order to obtain a 3D model of a wound and the surrounding skin in the form of triangular mesh. The performance of the CUDA implementation of the Marching cubes algorithm was on average 125ms for a 256x256x256 TSDF volume, which typically resulted in 3D models with about 325k triangles. Visualization of the state of wound reconstruction during recording is provided by ray casting. Ray casting generates a rendered image from a given camera position by casting rays directly onto the volume i.e. calculating the depth values of the zero crossing of the iso-surface of TSDF for rays originating from the camera position for each pixel of the rendered image. Depth images rendered by ray casting are also used in our system in a similar way as in KinectFusion, as a model for current pose estimation in the registration sub-phase.
153
154
Damir Filko et al. / Procedia Computer Science 90 (2016) 151 – 156
Our system uses two TSDF volumes in parallel, one high resolution model with voxel size of 1mm3 resulting in a total volume of 0.25m3, and a second lower resolution model with 4mm3 voxel size and 1m3 total volume. The low resolution TSDF volume is used for model depth image rendering which is used for pose estimation, while the high resolution TSDF volume is used for 3D mesh generation of the wound model which is later used for segmentation and measurement. The performance of TSDF ray cast rendering was on average 3.47ms per frame. Fig. 2 displays the reconstructed high resolution colored 3D models of all the wounds on the Saymour II wound care model.
Fig. 2. High resolution coloured models with an average of 325k triangles
3. Wound segmentation The approach applied in our system to segment out wounds is based on the assumption that a wound can be detected as a region consisting of bumps and dents, characterized by high local surface curvature, surrounded by a smooth skin surface. Furthermore, if a wound is very shallow and cannot be distinguished from the surrounding skin using its geometric properties, the colour of the wound can be used to segment it out from the uniformly coloured surrounding skin. The proposed approach consists of three stages: (i) oversegmentation to surfels, (ii) grouping of surfels into smooth surface segments and (iii) heuristic selection of segments which represent wounds. As a preprocessing step, our segmentation approach performs oversegmentation of the mesh obtained by the 3D reconstruction subsystem described in Section 2 into approximately planar surface patches of approximately uniform colour referred to in this paper as surfels. The method used for segmentation into surfels is based on the region growing method15, but, in contrast to that approach, we take into consideration the colour information and require each surfel to be approximately uniformly coloured. An example of oversegmentation into surfels is shown in Fig. 3(b). The detected surfels are grouped into smooth surfaces by a region growing process which starts by selecting a surfel, and proceeds by attaching recursively neighbouring surfels satisfying certain smoothness and colour similarity conditions. The result of the surfel grouping procedure is a set of smooth uniformly coloured surface segments. An example is shown in Fig. 3(c). Due to its irregular shape and non-uniform colour, the wound region is expected to consist of multiple smaller regions surrounded by a large smooth surface segment representing the healthy skin. The heuristic wound segmentation approach proposed in this paper is based on two assumptions. The first assumption is that the skin region surrounding the wound region is the largest smooth surface segment in the mesh. The second assumption is that the segment closest to the wound centre, determined by the wound detection approach proposed in Section 2, belongs to the wound region. Considering these two assumptions, the wound region is determined by a region growing procedure, which starts by creating the wound region, consisting initially of the single segment which is closest to the wound centre, and grows this region recursively by adding neighbouring segments until reaching the largest smooth mesh surface segment. The performance of segmentation procedure was on average 220ms. In order to isolate only the wound as a separate 3D model from the whole 3D scene generated, the Visualization Toolkit (VTK)16 is used. The selection filter of the VTK library uses the spline interpolated boundary to generate scalar data (Fig.3(d)) needed by the extraction filter, from the same library, to isolate the wound mesh into a separate 3D object.
Damir Filko et al. / Procedia Computer Science 90 (2016) 151 – 156
a) b) c) d) Fig. 3. (a) Reconstructed wound; (b) Generated surfels; (c) Final regions from region growing; (d) Wound region after edge spline interpolation
4. Wound measurement The developed system, using the extracted 3D wound model, provides the perimeter, area and wound volume as end results. The perimeter is easily calculated as the length of the connected points representing the boundary of the extracted 3D wound model. In order to calculate the wound area and volume, a cover or a virtual skin needs to be generated. When filming the wound using the Kinect sensor, the wound recording (reconstruction phase) usually begins at an angle perpendicular to the wound surface, and thus, the wound border is well defined on the 2D x-y plane. This fact is exploited when generating the virtual skin. An iterative 2D Delaunay triangulation algorithm is used in virtual skin generation. The initial iteration of the Delaunay algorithm starts with an initial point set containing the points from the spline interpolated boundary. Each consecutive iteration expands this point set by including the centres of the Delaunay triangles of the previous iteration. Experiments indicate that three iterations are sufficient to generate a reasonable approximation of the virtual skin. Since the Delaunay algorithm creates a convex hull which our border shape doesn’t always have, the virtual skin is isolated from the 3D mesh model in a similar manner as the wound is isolated from its neighbouring mesh. Fig. 4. shows the reconstructed and isolated wounds and corresponding virtual skins of the Saymour II wound care model.
Fig. 4. Isolated 3D wound models with virtual skin generated
The area and volume are calculated by appropriate algorithms implemented in the VTK library. The area is calculated as the sum of areas of individual triangles in the virtual skin mesh, while the individual triangle area is calculated using Heron’s formula. The volume is calculated using a morphometric method called MUNC (Maximum Unit Normal Component) and a discrete form of the divergence theorem (DTA). The accuracy and precision of these methods usually used for area and volume calculation are described17. Measurement calculation was on average 17ms. Table 1. shows the results of the repeatability experiments by means of the average and coefficient of variability of the perimeter, area and volume measurements for the three largest wounds found on the Saymour II wound care model. The low variability can be related to increased robustness of the system. The sequences used for reconstruction in these experiments had between 78 and 351 RGB-D images. The results show low variability of perimeter and area measurements for all wounds, while the volume has a slightly higher variability. This may be due to the fact that each recorded experiment had a different image sequence, which possibly did not include all the views necessary to determine the actual wound depth at every point.
155
156
Damir Filko et al. / Procedia Computer Science 90 (2016) 151 – 156 Table 1. Measurement repeatability experiments Wound type
Perimeter
Area
Volume
avg (mm)
variability (%)
avg (mm2)
variability (%)
avg (mm3)
Stage IV Pressure Ulcer
354.2
2.608
7763.6
2.033
139074.3
4,885
Stage III Pressure Ulcer
192.1
2.707
2712.1
2.175
28511.9
12,875
5 ½” long dehisced surgical wound
288.1
1.233
2898.0
1.294
39241.2
8,268
variability (%)
5. Conclusion In this paper, a system for automatic measurement of wound parameters is presented, which enables detailed wound analysis and, indirectly, improved treatment for patients. The system consists of four subsystems: wound detection, reconstruction, segmentation and measurement. The proposed system is experimentally evaluated on Saymour II wound care model, for which adequate accuracy with respect to clinicians’ requirements is achieved. However, in order to get an insight into its robustness, a more thorough testing on real wounds and different lighting conditions should be performed. Acknowledgements This work was supported by the Josip Juraj Strossmayer University of Osijek, under grant No. IZIP-2014-70. References 1. Gethin G, Cowman S. Wound measurement comparing the use of acetate tracings and Visitrak digital planimetry. Jornal of Clinical Nursing; vol. 15, no. 4, pp. 422-427, 2005. 2. Gilman T. Wounds Outcomes: The Utility of surface Measures. Lower Extremity Wounds; vol. 3, no. 3, pp. 125-132, 2004. 3. Chang AC, Dearman B, Greenwood JE. A comparison of wound area measurement techniques: visitrak versus photography. Eplasty; vol. 11, 2011. 4. Treuillet S, Albouy B, Lucas Y. Three-dimensional assesment of skin wounds using a standard digital camera. IEEE Transactions in Medical Imaging; vol. 28, no. 5, pp. 752-762, May 2009. 5. Bowling FL, King L, Paterson JA, Hu J, Lipsky BA, Matthews DR, Boulton AJM. Remote assessment of diabetic foot ulcers using a novel wound imaging system: Remote foot ulcer assessment using a wound imaging system. Wound Repair and Regeneration; vol. 19, no. 1, pp. 25–30, Jan. 2011. 6. Callieri M, Cignoni P, Pingi P, Scopigno R, Coluccia M, Gaggio G, Romanelli MN. Derma: Monitoring the Evolution of Skin Lesions with a 3D System; VMV, pp. 167–174., 2003. 7. Zvietcovich F, Castañeda B, Valencia B, Llanos-Cuentas A. A 3D assessment tool for accurate volume measurement for monitoring the evolution of cutaneous leishmaniasis wounds. Engineering in Medicine and Biology Society (EMBC) 2012 Annual International Conference of the IEEE; pp. 2025–2028, 2012. 8. 3DYORYþLþ U, Diaci J, Možina J, Jezeršek M. Wound perimeter, area, and volume measurement based on laser 3D and color acquisition. BioMedical Engineering OnLine; vol. 14, no. 1, Dec. 2015. 9. Bills JD, Berriman SJ, Noble DL, Lavery LA, Davis KE. Pilot study to evaluate a novel three-dimensional wound measurement device: Threedimensional wound assessment tool. International Wound Journal; Nov. 2015., In press 10. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohi P, Shotton J, Hodges S, Fitzgibbon A. KinectFusion: Real-time dense surface mapping and tracking. Mixed and augmented reality (ISMAR), 10th IEEE international symposium on; pp. 127–136., 2011. 11. Bhattacharyya A. On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society; Vol. 35, No. 1, pp. 99–109, 1943. 12. Whelan T, https://github.com/mp3guy/ICPCUDA, downloaded on 28. April 2015. 13. Curless B, Levoy M. A volumetric method for building complex models from range images. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques; pp. 303-312., 1996. 14. Lorensen WE, Cline HE. Marching Cubes: A high resolution 3D surface construction algorithm. SIGGRAPH Computer Graphics; Vol. 21, No. 4, July 1987. 15. Holz D, Behnke S. Approximate Triangulation and Region Growing for Efficient Segmentation and Smoothing of Range Images. Robotics and Autonomous Systems; Vol. 62, No. 9, pp. 1282–1293, 2014. 16. The Visualization Toolkit, www.vtk.org, version 6.3 17. Alyassin AM, Lancaster JL, Downs JH3, Fox PT. Evaluation of new algorithms for the interactive measurement of surface area and volume. Medical Physics; Vol. 21, No. 6, pp. 741-752, June 1994.
