Using the Kinect for Search and Rescue Robotics Jesus Suarez
Robin R. Murphy
Texas A&M University College Station, Texas 77843
[email protected]
Texas A&M University College Station, Texas 77843
[email protected]
ABSTRACT The Microsoft Kinect has been used prolifically in robotics applications including rescue robotics since its introduction in November 2010, but its limitations have seldom been addressed. Although the low cost depth sensor is an attractive option for use in robot navigation, mapping, and human robot interaction, its poor performance in bright sunlight makes it generally unsuitable for outdoor work. This paper briefly surveys the Kinect’s use in rescue robotics and similar applications, and highlights the associated challenges.
Keywords Kinect, Search and Rescue Robotics
1.
INTRODUCTION
The Kinect is a low cost ($110 at the time of writing) sensor that includes a video camera, microphone array, and an infrared-based depth camera. Though it was designed and marketed as a body-tracking peripheral for the popular Microsoft Xbox 360 video game console, the Kinect was quickly adopted by researchers as a means to cheaply acquire 3D range data (called “2.5D” by some researchers) for use in generating world maps and searching for open paths and obstacles in a navigation task. Roboticists had previously relied on expensive time-of-flight (ToF) depth cameras or 2D laser range finders on custom rigs to get the same data. Moreover, two widely available APIs for the Kinect (Microsoft’s Kinect for Windows SDK and the OpenNI framework) have made the Kinect a valuable sensor for human computer and human robot interaction by providing body- and hand-tracking solutions. The principal challenge of using the Kinect for robotics applications is the fact that as an infrared-based sensor, it is susceptible to image saturation in bright sunlight, making it challenging to use outdoors. The Kinect’s depth camera produces a depthmap image by projecting a dense, nonuniform array of infrared light dots onto a scene and capturing them with an infrared camera. Pixel distances/depths are determined according to distortions in the dot map, and the outcome is a 320x240 16-bit image for which each pixel’s value is the real-world distance to the sensor (the image is scaled up to 640x480 in software). The resulting depthmap can be used directly or transformed into a 3D point cloud
Figure 1. An example of Kinect depth image degradation in bright light for a hand-detection problem. The top image was captured indoors, the lower left is a similar scene captured outdoors in shade, and the lower right image is a similar scene captured in direct sunlight. The green squares are candidate hand locations and the blue area is a segmented hand. containing the same information. Since the infrared dots used to produce the depth images are dim, they are indiscernible by the IR camera if there is too much ambient infrared light in the scene, as is the case outdoors in direct sunlight. Figure 1 shows an example of this image saturation for a hand detection problem (the hand detector is that described in [1]). With more sunlight in the scene, fewer pixels are correctly resolved, and image quality quickly deteriorates. Though search and rescue scenarios vary in settings and environmental conditions, they certainly do not all occur indoors or in the dark, so the effects of bright sunlight on sensors cannot be overlooked. This work surveys the use of the Kinect in rescue robotics and in general robotics problems that pertain to search and rescue robotics. The emphasis is on the challenges associated with using the sensor, and in determining the viability of the Kinect for rescue robotics.
2.
KINECT USAGE FINDINGS
The Kinect has been used extensively in robotics, where its most popular functions are to aid in robot navigation ([2-7]), environment mapping ([8]) and object manipulation ([9, 10]). However, with one exception (Abbas and Muhammad [7]), the cited examples are all set indoors. Abbas and Muhammad highlight the difficulty of using the Kinect for outdoor tasks in their paper that evaluates the performance of the Kinect for SLAM under different lighting conditions and ground types. Although the authors concede that the Kinect is not well suited for outdoor work, they suggest that their application (slow mine detection) is a special exception and that the Kinect is a viable option for them since there is no time limit and the detection process can be repeated until successful. The Kinect’s limitations with respect to bright light are also well documented in several performance evaluations of the sensor ([11-13]), all of which recommend against using the sensor outdoors. These evaluation papers also point out that the sensor tends to produce image noise when non-diffuse objects (i.e. reflective and specular objects) are in a scene. Furthermore, the sensor’s operation is limited to a range of 1.2m to 3.5m (though objects as close as 0.5m can be discerned in appropriate lighting). To the authors’ best knowledge, the Kinect has not been used on a deployed rescue robot, though it has been used in simulated disaster settings – namely the RoboCup Rescue competition. One undergraduate team from the University of Warwick (lead by Dr. Emma Rushforth and Dr. Peter Jones) included a Kinect in their robot entry[14], which they use to perform 3D SLAM. Again though, the environment is indoors, and moreover the highly structured RoboCup course (made largely of highly diffuse plywood) lends itself easily to the task. What this means for rescue robotics is that unless the environment is known to be indoors or otherwise out of direct sunlight, the Kinect is generally not a viable sensor to use for mapping or navigation. The findings from the brief literature survey confirm that while the Kinect can be used successfully indoors, direct sunlight degrades image quality too severely to permit use outdoors.
4.
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12] [13]
REFERENCES
[1] J. Suarez and R. R. Murphy, "Haar Cascade Hand Detection for Gesture Recognition with a Depth Camera under Different Lighting Conditions," In review at the Journal of Visual Communication and Image Representation, 2012. [2] P. Benavidez and M. Jamshidi, "Mobile robot navigation and target tracking system," in System of Systems Engineering (SoSE), 2011 6th International Conference on, pp. 299-304, 2011. [3] D. S. O. Correa, D. F. Sciotti, M. G. Prado, D. O. Sales, D. F. Wolf, and F. S. Osorio, "Mobile Robots Navigation in Indoor Environments Using Kinect
[14]
Sensor," in Critical Embedded Systems (CBSEC), 2012 Second Brazilian Conference on, pp. 36-41, 2012. R. A. El-laithy, H. Jidong, and M. Yeh, "Study on the use of Microsoft Kinect for robotics applications," in Position Location and Navigation Symposium (PLANS), 2012 IEEE/ION, pp. 1280-1288, 2012. J. Biswas and M. Veloso, "Depth camera based indoor mobile robot localization and navigation," in Robotics and Automation (ICRA), 2012 IEEE International Conference on, pp. 1697-1702, 2012. N. Ganganath and H. Leung, "Mobile robot localization using odometry and kinect sensor," in Emerging Signal Processing Applications (ESPA), 2012 IEEE International Conference on, pp. 91-94, 2012. S. M. Abbas and A. Muhammad, "Outdoor RGB-D SLAM Performance in Slow Mine Detection," Robotics; Proceedings of ROBOTIK 2012; 7th German Conference on, pp. 1-6, 2012. W. J. Woodall and D. Bevly, "Using the microsoft kinect for 3D map building and teleoperation," in Position Location and Navigation Symposium (PLANS), 2012 IEEE/ION, pp. 1054-1061, 2012. J. Stuckler, D. Holz, and S. Behnke, "RoboCup@Home: Demonstrating Everyday Manipulation Skills in RoboCup@Home," Robotics & Automation Magazine, IEEE, vol. 19, pp. 34-42, 2012. J. Figueroa, L. Contreras, A. Pacheco, and J. Savage, "Development of an Object Recognition and Location System Using the Microsoft Kinect Sensor." vol. 7416, T. Röfer, N. Mayer, J. Savage, and U. Saranli, Eds., ed: Springer Berlin / Heidelberg, pp. 440-449, 2012. S. Milani and G. Calvagno, "Joint denoising and interpolation of depth maps for MS Kinect sensors," in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pp. 797-800, 2012. K. Khoshelham, "Accuracy Analysis of Kinect Depth Data," presented at the ISPRS Calgary 2011 Workshop, Calgary, Canada, 2011. J. Smisek, M. Jancosek, and T. Pajdla, "3D with Kinect," in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pp. 1154-1160, 2011. P. Jones, E. Rushforth, et al., "Warwick Mobile Robotics: Urban Search and Rescue Robot (Technical Report)," University of Warwick, 2012.
