Investigation of Vision-based Underwater Object Detection with ...

International Journal of Advanced Robotic Systems

ARTICLE

Investigation of Vision-based Underwater Object Detection with Multiple Datasets Regular Paper

Dario Lodi Rizzini1*, Fabjan Kallasi1, Fabio Oleari1 and Stefano Caselli1 1 Universita degli Studi di Parma, Parma, Italy *Corresponding author(s) E-mail: [email protected] Received 02 December 2014; Accepted 17 March 2015 DOI: 10.5772/60526 © 2015 Author(s). Licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

1. Introduction

In this paper, we investigate the potential of vision-based object detection algorithms in underwater environments using several datasets to highlight the issues arising in different scenarios. Underwater computer vision has to cope with distortion and attenuation due to light propaga‐ tion in water, and with challenging operating conditions. Scene segmentation and shape recognition in a single image must be carefully designed to achieve robust object detection and to facilitate object pose estimation. We describe a novel multi-feature object detection algorithm conceived to find human-made artefacts lying on the seabed. The proposed method searches for a target object according to a few general criteria that are robust to the underwater context, such as salient colour uniformity and sharp contours. We assess the performance of the proposed algorithm across different underwater datasets. The datasets have been obtained using stereo cameras of different quality, and diverge for the target object type and colour, acquisition depth and conditions. The effectiveness of the proposed approach has been experimentally dem‐ onstrated. Finally, object detection is discussed in connec‐ tion with the simple colour-based segmentation and with the difficulty of tri-dimensional processing on noisy data.

In recent years, the interest of the scientific community in underwater computer vision has increased, taking advant‐ age of the evolution of sensor technology and image processing algorithms. The main challenges of underwater perception are due to the higher device costs, their complex setup, and the distortion in signals and light propagation introduced by the water medium. In particular, light propagation in underwater environments suffers from phenomena such as absorption and scattering which strongly affect visual perception. While underwater computer vision has been applied to inspection, environ‐ ment monitoring, mapping and terrain segmentation, underwater object detection has not been thoroughly investigated. In particular, accurate and robust object detection and pose estimation are essential requirements for the execution of manipulation tasks. A robust detection method is able to correctly identify different target objects in different experimental conditions. Once the region of the image corresponding to an object is found, its pose with regard to the observer frame can be estimated in a single image, if the geometry and the size of the object is known, or else by using stereo-processing.

Keywords Underwater Computer Vision, Object Detec‐ tion, Image Segmentation

In this paper, we investigate the potential of vision-based object detection algorithms in underwater environments using different datasets, including their contribution to underwater stereo vision processing. Our first contribution is a novel multi-feature object detection algorithm to find Int J Adv Robot Syst, 2015, 12:77 | doi: 10.5772/60526

1

human-made artefacts lying on the seabed and which improves and generalizes the work in [1]. The proposed method searches a target object according to a few general criteria, such as salient colour uniformity and sharp contours. The images are initially processed to enhance the contrast in the underwater images. First, the image is partitioned into clusters according to the feature values extracted from each pixel. The current feature vector includes the values of the hue saturation value (HSV) space and its response to the gradient, but it can be easily extended to include pattern-related (e.g., the response to Gabor filters or Eigen transform [2]) and other features. Second, the connected regions of each cluster are classified according to the corresponding angular histogram. The angular histogram of the artefacts is concentrated on one or a few peaks, while the histograms corresponding to the blobs extracted from the natural seabed are usually more distributed. The proposed algorithm relies on these salient properties of common human-made objects, but it is robust to the moderate violation of such hypotheses (e.g., the presence of stripes on general colour uniformity). The second contribution of this paper is the assessment of the proposed algorithm across different underwater datasets. There are few available datasets focused on objects and acquired using stereo-vision systems in underwater environments. Our experiments were per‐ formed on two datasets obtained during the MARIS project (Marine Autonomous Robotics for InterventionS) [3] and a dataset from the Trident project [4]. The differences among the datasets concern the camera quality, the experimental conditions (depth, light conditions, background, sensor guidance, etc.) and the target objects. Despite these differ‐ ences, the proposed algorithm achieves precise and reliable detection, while elementary colour-based segmentation approaches cannot reliably find a region of interest (ROI). Finally, the positive results achieved in mono-camera detection are discussed in connection with the problems occurring in 3D object recognition. In underwater environ‐ ments, stereo processing is usually not able to provide a reliable 3D representation of the scene enabling object recognition from shapes. In our evaluation, the target object's pose can be reliably estimated only by exploiting the output of mono-camera processing. The paper is organized as follows. Section 2 reviews the state of the art in vision-based object detection for under‐ water environments. Section 3 describes the image proc‐ essing pipeline and, in particular, the proposed object detection algorithm. Section 4 illustrates the results on object detection and pose estimation in underwater environments. Section 5 discusses problematic issues arising in the underwater context. Section 6 provides some final remarks and observations.

to the problems arising with light transmission in water. Instead, sonar sensing is largely used as a robust perception modality for localization and scene reconstruction in underwater environments. In [5], Yu et al. describe a 3D sonar imaging system used for object recognition based on sonar array cameras and multi-frequency acoustic signals emissions. An extensive survey of ultrasonic underwater technologies and artificial vision is presented in [6]. However, object detection using sonar imagery is difficult, although some techniques have been proposed. Williams and Groen [7] propose an integral image method to find items differing from the homogeneous background. Sawas et al. [8] use a boosted classifier on Haar features. Under‐ water laser scanners guarantee accurate acquisition [9]; however, they are very expensive and are also affected by problems with light transmission in water. Computer vision provides information at lower cost and with a higher acquisition rate compared to acoustic perception. Artificial vision applications in underwater environments include the detection and tracking of submerged artefacts [10], seabed mapping with image mosaicing [11], and underwater SLAM [12]. Garcia et al. [13] compare popular feature descriptors extracted from underwater images with high turbidity, but not for object detection. Aulinas et al. [14] search salient colour regions of interest in order to select stable SURF features such as landmarks in SLAM applications. Without colour segmen‐ tation, the data association is unreliable, even for scene description purposes. Stereo vision systems have only been recently introduced in underwater applications due to the difficulty of calibration and the computational perform‐ ance required by stereo processing. To improve homolo‐ gous point matching performance, Queiroz-Neto et al. [15] introduce a stereo matching system specific to underwater environments. The disparity of stereo images can be exploited to generate 3D models, as shown in [16, 17]. Leone et al. [18] present a 3D reconstruction method for an asynchronous stereo vision system. Although the 3D reconstruction achieved by underwater stereo vision may be satisfactory to represent a scene, its accuracy is not generally sufficient for the detailed perception required in object detection and recognition.

INPUT IMAGE

PREPROCESSING

ROI IDENTIFICATION

OBJECT DETECTION

STEREO POSE ESTIMATION

MONO POSE ESTIMATION

2. Related Work Computer vision is a major perception modality in robotics and, in particular, for object detection tasks. In underwater environments, however, vision is not as widely used due 2

Int J Adv Robot Syst, 2015, 12:77 | doi: 10.5772/60526

Figure 1. Schema of the object detection and pose estimation algorithm. The algorithm's operations can be combined variously in a pipeline.

Underwater object recognition using computer vision is the contour of which accurately delimits the object's edges somewhat difficult due to the lighting conditions of such in the image. The point cloud is used only to estimate the environments. To cope with the challenging environment, pose of an already-detected object (Stereo Pose Estimation underwater object detection always exploits the peculiar block in Figure 1). Alternatively, if the geometric model and characteristics of the object to be detected, such as shape the dimensions of the object are known, the pose can be [19, 20, 10] or colour [21, 22]. Several of these works address estimated from the shape of the object's projection in a INPUT IMAGE pipeline detection problems, since pipelines recur in many single in frame. Both pose estimation require block Figure 1). the Alternatively, if themethods geometric modela underwater applications. Olmos et al. [23] addressed the geometric description target object. and the dimensions of of thethe object are known, the pose can be estimated from the shape of the object’s projection in a problem of detecting whether there is a man-made object PREPROCESSING single frame. Both the pose estimation methods require a using contour and texture criteria. However, the proposed 3.1 Image Pre-processing geometric description of the target object. method does not return a region in the image containing Underwater object detection requires the vision system to the object, but simply a binary decision about the object's copeImage withPre-processing difficult underwater lighting conditions. In presence. Several object detection algorithms [22, 21, 24] 3.1. ROI IDENTIFICATION OBJECT DETECTION particular, light attenuation in water produces blurred exploit colour segmentation to find one or more regions of Underwater object detection requires the vision system images with limited contrast, and light back-scattering interest and perform a more accurate assessment on the to cope with difficult underwater lighting conditions. In results in artefacts in the acquired images. Object detection found ROI. Kim et al. [25, 22] present a vision-based object STEREO POSE ESTIMATION MONO POSE ESTIMATION particular, light attenuation in water produces blurred detection method based on template matching and tracking becomes evenlimited more difficult in the of suspended images with contrast, andpresence light back-scattering for underwater robots using artificial objects, but all the particles or with an irregular and variable background. results in artefacts in the acquired images. Object detection Figure 1. Schema of the object detection and pose estimation tests are performed in a swimming pool. Bazeille et al. [21] Hence, for underwater perception, special attention must algorithm. The algorithm's operations can be combined variously becomes even more difficult in the presence of suspended in a pipeline. discuss the colour modification occurring in underwater be paid tooralgorithmic solutionsand improving quality. particles with an irregular variableimage background. environments and experimentally assess the performance Hence, for underwater perception, special attention must method based on template matching and tracking for The first phase of the algorithmic pipelines in Figure 1 is of object detection based on colour. Since underwater be paid to algorithmic solutions improving image quality. underwater robots using artificial objects, but all the tests conceived to compensate the colour distortion incurred by imaging suffers from a short range, low contrast and nonare performed in a swimming pool. Bazeille et al. [21] The first phase of the algorithmic pipelines in Figure 1 is the light propagating in water through image enhance‐ uniformthe illumination, simple colour segmentation is one of discuss colour modification occurring in underwater conceived to compensate the colour distortion incurred ment. No information about the object is used in this phase, the few viableand approaches. In [24],assess the underwater stereo environments experimentally the performance by light propagating water image sincethe the processing is applied in to the wholethrough image. Popular vision system usedbased in theonEuropean projectunderwater Trident is of object detection colour. Since enhancement. No information about the object is used in techniques for image enhancement are based on colour described. Objectfrom detection is performed by constructing imaging suffers a short range, low contrast anda this phase, since the processing is applied to the whole restoration [26]. The approach adopted in this paper colour histogram in the HSV space of the target object. In non-uniform illumination, simple colour segmentation is image. Popular techniques for image enhancement are the performed experiments, there is an intermediate step focuses on strengthening contrast recoveradopted blurry one of the few viable approaches. In [24], the underwater based on colour restoration [26]. The to approach between inspection intervention whereproject the realTrident images stereo vision systemand used in the European underwater images. A contrast mask method is first applied in this paper focuses on strengthening contrast to recover ofdescribed. the site to Object manipulate are available and by used for acquir‐ is detection is performed constructing of the CIELAB colour of the to the component blurry underwater Limages. A contrast maskspace method is aing colour histogram in appearance the HSV space the target object's [4]. of the target object. inputapplied image. In the component of each pixel first toparticular, the component L of the LCIELAB colour in,i In the performed experiments, there is an intermediate space of the input image. i is extracted, a median filterInisparticular, applied to the the component L -channel step between inspection and intervention where the real L of each pixel i is extracted, a median filter 3. Algorithms in,i of the image to obtain a new blurred value L bluris,i , applied and the images of the site to manipulate are available and used for to the L-channel of the image to obtain a new blurred new value is computed as . The L 1.5L − 0.5L acquiring the target appearance [4]. addressed by out ,i blur ,ias L effect Vision-based objectobject’s detection may be value Lblur,i , and the new value is in,i computed out,i = of the is a sharpened image with increased different approaches according to the input data: through 1.5 Lin,icontrast − 0.5 Lmask . The effect of the contrast mask is a blur,i contrast. image with increased contrast. image processing of an image acquired by a single camera, sharpened 3. Algorithms or through more complex shape matching algorithms Next, in in order order to to re-distribute re-distribute luminance, luminance, contrast-limited contrast-limited Next, Vision-based object detection may be addressed by based on stereo processing. The set of algorithms for adaptive histogram equalization equalization(CLAHE) (CLAHE) is different approaches according to the input data: through adaptive histogram [27] [27] is per‐ underwater object detection proposed in this paper consists performed. The combined application of themask contrast image processing of an image acquired by a single camera, formed. The combined application of the contrast and of several phases operating at decreasing levels of abstrac‐ mask and CLAHE compensates the lightand attenuation or through more complex shape matching algorithms CLAHE compensates the light attenuation removes tion and increasing knowledge object to be and removes some of the artefacts in the image. Figure based onwith stereo processing. Theabout set the of algorithms some of the artefacts in the image. Figure 2 shows an detected. Figureobject 1 shows how proposed the different algorithm 2 shows an example of the effect of pre-processing for for underwater detection in this paper example of the effect of pre-processing for an underwater operations can be combined togetheratinto a pipeline. The an underwater image. In our experiments, the image consists of several phases operating decreasing levels image. In our experiments, the image enhanced by CLAHE chosen processing could depend upon the amount enhanced by CLAHE alone is not discernible from one of abstraction and pipeline with increasing knowledge about the alone is not discernible obtained of reliable features offered by thehow object, well as obtained after applying from both one filters. Hence,after the applying contrast object to bevisual detected. Figure 1 shows the as different both filters. Hence, the contrast mask computation may be upon the specific underwater conditions. The initial step mask computation may be skipped, thereby reducing algorithm operations can be combined together into a skipped, thereby reducing processing time. aims to detect salient regions with regard to the back‐ processing time. pipeline. The chosen processing pipeline could depend ground objects, possibly with by no upon therepresenting amount of candidate reliable visual features offered priorobject, knowledge aboutastheupon object.the Thespecific output of such a step the as well underwater conditions. Thecandidate initial step aims to detect regions may be a broad ROI to narrow thesalient searching area with to thephases, background representing candidate in theregard successive with the classification of the objects, with no prior about the object. regionspossibly as a target object or notknowledge (respectively the blocks ROI The output of such a step may be a broad candidate Identification and Object Detection in Figure 1). In the ROI first to narrow the searching in the successive phases, case, no decision about thearea target's presence is taken from with theframe. classification of the regions as a target or a single Such a decision is delegated either object to a more not (respectively the blocks ROI Identification and Object accurate monocular processing or is jointly performed with Detection in Figure firstcloud. case, In nothe decision Figure 2.2.AnAn underwater image before (left) and after (right) the(right) application pose estimation on 1). theIn 3Dthe point secondabout case, Figure underwater image before (left) and after the the target’s presence is taken from a single frame. Such a of a contrast mask and CLAHE application of a contrast mask and CLAHE. the output of the Object Detection module consists of an ROI, decision is delegated either to a more accurate monocular processing or is jointly performed with pose estimation Dario Lodi Rizzini, Fabjan Kallasi, Fabio Oleari and Stefano Caselli: 3 on the 3D point cloud. In the second case, the output Investigation of Vision-based Underwater Object Detection with Multiple Datasets 3.2. Mono Camera Processing of the Object Detection module consists of an ROI, the contour of which accurately delimits the object’s edges in The goals of single image processing may include the the image. The point cloud is used only to estimate the identification of an ROI containing target objects, the pose of an already-detected object (Stereo Pose Estimation

3.2 Mono Camera Processing The goals of single image processing may include the identification of an ROI containing target objects, the detection of the specific target object and, if the target object's geometry is known, the estimation of its pose. The careful use of monocular processing can significantly improve the effectiveness of the application. Even in an underwater environment, with proper illuminators, shallow and clean waters, and quality cameras, the con‐ tours of objects are distinguishable in the image. The target object thus can be detected in a single image or - at least its detection can be facilitated by restricting the search region to be analysed in later, more expensive steps. Object recognition on a 3D point cloud is computationally expensive and depends upon the quality of the input data obtained by stereo processing. If the object has already been detected based on appearance in mono camera images, the 3D data corresponding to the segmented ROI are used only to estimate the pose of the object. In the following, two rough but fast techniques for the identification of an ROI possibly containing the target object (RO I area and RO I colour ) and a more accurate technique for target object detection

(RO I shape ) are illustrated. Furthermore, a method for the

pose estimation of cylindrical objects in a single images is proposed. ROI Identification The algorithms described in this section aim to detect an ROI that may represent - or at least contain - an object. The ROI may be searched according to different criteria based on specific features of the object to be found. We have developed two approaches that exploit different assump‐ tions about the properties of the target. The HSV colour space is used to improve colour segmentation [28] since it better represents human colour perception. In particular, to reduce the effect of noise and light distortion, a colour reduction is performed on the H channel of the input image. The algorithms described in this paper use 16 levels of quantized colour. The input image is partitioned into subsets of - possibly not connected - pixels with the same hue level according to the value of the reduced channel H . The rough level quantization is not affected by the patterns generated by light back-scattering. The first segmentation method, hereafter denoted ROIarea,

is based on the assumption that the unknown object never occupies more than a given portion of the image pixels and that it has a uniform colour. The region corresponding to a given hue level is estimated as the convex hull of the pixels. Only regions the area of which is less than 50% of the image are selected as part of the ROIarea. This heuristic rule rests

on the hypothesis that the object is observed from a distance, such that only the background occupies a large portion of the image. The second approach exploits information about the colour of the target. When the object's colour is known, a more

4

Int J Adv Robot Syst, 2015, 12:77 | doi: 10.5772/60526

specific colour mask(ROIcolour ) can be applied to detect the

object with an accurate estimation of the object's contour. Hence, ROIcolour is obtained by composing the regions

where the colour is close (up to a threshold) to the expected target colour. The region computed either by ROIarea or ROIcolour is made

available for further processing. Both these ROI estimation techniques exploit only the relative colour uniformity of a texture-less object, but they do not identify a specific object. Furthermore, they tend to overestimate the area that potentially contains the object. Object Detection The proposed RO I shape algorithm performs object detection

in two steps: image segmentation and contour shape valida‐ tion. The goal is the identification of a connected region with straight and sharp contours like typical human-made artificial objects. RO I shape relies on the relative colour

uniformity of the object for segmentation and on its regular contour shape for validation. The detection exploits these two salient and relatively general features of artefacts in an underwater environment. In contrast with the previously described coarse segmentation approaches (ROIarea and ROIcolour ), RO I shape is able to detect whether the target object

belongs to the image before performing pose estimation. The current implementation of the algorithm finds a single object in the image, although it could be adapted to return multiple regions in the image satisfying the above criteria. The image segmentation step classifies each pixel of the image according to its corresponding vector of local features. The input image can be rescaled to a lower size to remove unnecessary details and to reduce the computa‐ tional complexity of the object detection. The scaling operation acts as a low-pass filter in the image. The initial classification of each pixel pi is independent of the classifi‐ cation of other pixels. In particular, the feature vector computed for pi consists of the colour channels of the HSV space, respectively hue h i , saturation si and value vi , and of

the gradient response to a Sobel filter gi . The feature space

adopted in this paper is larger than that used in [29] and could be further expanded. Next, the item vectors fi = h i ,si ,vi ,gi Τ are clustered according to a k-means algo‐

rithm [30]. The number of clusters used in the experiments described in Section 4 is k = 3 and is independent of the number of objects in the scene. Figure 3(a)-(b) illustrates a typical output of k-means clustering for one of the datasets investigated in this paper: the artificial objects are classified as belonging to the same cluster, whereas the background is split into a light region and a dark region. The image partition achieved by k-means is further refined by computing the connected components of the cluster. Before searching the connected components, a morphological dilation followed by an erosion is performed on the image to avoid over-segmentation. The result is shown in Figure 3(c). The achieved segmentation is correct when there is no

(a)

(b)

(c)

normalized histogram

1

mean

0.8 0.6 0.4 0.2 0 0

10

20

30

40

50

60

70

80

90

angle [deg]

(d)

(e)

(f)

Figure3.3.Steps Steps of ROI theshapeROI (a) image; the input image; the clusters obtained from k-means; (c) the connected components shape algorithm: Figure of the algorithm: (a) the input (b) the clusters(b) obtained from k-means; (c) the connected components extracted from one of the extracted one ofimage the ofclusters; (d) the(within contour image ofbox); one(e)component (withinangle its histogram; bounding (f)box); (e) the corresponding angle clusters; (d) from the contour one component its bounding the corresponding the output mask histogram; (f) the output mask.

contact the objects; otherwise, a more sophisticated region. among The contours of human-made regularly shaped and computationally technique must applied. objects - in particularcomplex when their projection inbe the image planeproposed is approximately a rectangle - often consists The approach has been successfully applied of to parallel edges. Under datasets this assumption, the target region different underwater corresponding to typical is recognized by detecting parallel line-segments from the scenarios of robotic underwater experiments. contour, e.g., using the Hough transform. Hence, a set of The shape validation stepfrom is applied to each cluster segments is extracted the contour. Each obtained segment from image segmentation. algorithm computes line the j is described by its length The l j and by its supporting with equation cos α j image + y sincorresponding α j = r j (coordinates are contour of each xbinary to the cluster, expressed with regard to Each the image detectiona as shown in Figure 3(d). closedcentre). contourThe represents of line direction is allowed by an angular histogram H cluster-region, and shape matching between the contours with bin counters hs ∈ N and the intervals [s∆θ, (s + 1)∆θ [, and the target shape allows the identification of the target s = 0, . . . , nh − 1 and ∆θ = π/(2nh ). In particular, the region. contours the of corresponding human-made regularly segmentThe j increments angle bin hshaped k with objects in particular when their projection the image a contribution proportional to the square of itsinlength l j as plane is approximately a rectangle - often consists of parallel edges. Under this the target region is  assumption, (α j + π ) mod π2 recognized by detecting parallel line-segments from the s= (1) ∆θ contour, e.g., using the Hough transform. Hence, a set of  2 lj segments is extracted is hs ←from hs + the contour. Each segment j(2) maxi li described by its length lj and by its supporting line with equation xcosαj + ysinαj = rj (coordinates are expressed with regard to the centre). The detection of line direction The square ofimage the normalized length reduces the influence of allowed the smaller resulting ℋ from is by an segments angular histogram withthe bin potential counters of the contours. Finally, a cluster is ( ) sΔθ, s + 1 Δθ hover-segmentation ∈ ℕ s = 0, … ,n and the intervals , h − 1 and s classified anInobject with a the regular shapej ifincrements the histogram Δθ = π / (2nas . particular, segment the ) h is “peaked”, i.e., if it is distributed along a few principal corresponding angle bin h k with a contribution propor‐ directions. In particular, the validation condition of the tional to the square of its length lj as clusters is n −1

1 h h¯ =ê ∑ hs < σthp ú max hs ê (nahj +s=p0 ) mod ú06s