EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS EVALUACIÓN DE ALTERNATIVAS BASADAS EN LA TRANSFORMADA DE DISTANCIA PARA LA SEGMENTACIÓN DE IMÁGENES DE OBJETOS QUE SE SOLAPAN Chinea Valdés, Lyanett1, Lorenzo Ginori, Juan Valentín1 1 Centro de Estudios de Electrónica y Tecnologías de la Información, Facultad de Ingeniería Eléctrica, Universidad Central “Marta Abreu” de Las Villas, Cuba,
[email protected]
ABTRACT In the field of image segmentation, the case of objects that are touching or overlapping is frequently found. These require the use of appropriate techniques to achieve a good segmentation. Examples of this are found in digital image processing for cellular imaging, in the microscopy analysis of rocks and others. One way to address this problem is by combining the use of the distance transform and the watershed transform. In this case, it is particularly important to build adequate markers for the watershed transform. The ways of obtaining and using the markers influence the quality of segmentation, as well as the level of human participation in the process. The latter is determinant in regard of time consumption, and is particularly important when a large number of images are to be processed. In this work, various alternatives for building the markers are evaluated, which are oriented to avoid the human participation in the process. To evaluate the quality of the segmentation results, these were compared through calculation of the Jaccard index, with those obtained by means of manual segmentation of the selected images. Tables with the quantitative results are exhibited and graphical examples are shown as well. Application of the classical distance transform using as markers the centroids of the extended minima obtained from the application of the H-minima transform, exhibited the best results. Keywords: Segmentation, distance overlapped images, watersheds.
transform,
RESUMEN En el campo de la segmentación de imágenes, es frecuente encontrar casos de objetos que se tocan o se solapan. Estos requieren del uso de técnicas apropiadas para lograr una buena segmentación. Ejemplos de esto se encuentran en el procesamento digital de imágenes para imaginología celular, en el análisis microscópico de rocas y otros. Una forma de abordar este problema es combinando el uso de la transformada de distancia y la transformada watershed. En este caso, es particularmente importante construir marcadores adecuados para la transformada watershed. La forma de obtener y utilizar los marcadores, influye sobre la calidad de la segmentación y sobre el grado de participación humana en el proceso. Esto último es determinante en cuanto al consumo de tiempo, y tiene particular importancia cuando se requiere procesar grandes cantidades de imágenes. En el presente trabajo se evalúan varias alternativas para la construcciòn de los marcadores, orientadas a prescindir de la participación humana en el proceso. Para evaluar la calidad de los resultados de la segmentación, estos fueron comparados, mediante el cálculo del índice de Jaccard, tomando como referencia los obtenidos mediante la segmentaciòn manual de las imágenes seleccionadas. Se exhiben tablas con los resultados cuantitativos y se muestran también ejemplos gráficos de los mismos. La aplicación de la transformada de distancia clàsica, usando como marcadores los centroides de los minimos regionales obtenidos mediante la transformada Hminima, exhibió los mejores resultados.
ISBN: 978-959-7213-01-7
1
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
Palabras clave: Segmentación, transformada de distancia, imágenes solapadas, watersheds.
1. INTRODUCTION The segmentation of touching or overlapped objects (usually some class of particles) is a challenging problem for the automated image analysis in various fields, as occur in cellular image processing, where separating overlapped objects through segmentation is critical in applications like identification and classification of cells [1]–[3], in rock microscopy and blast fragmentation analysis [4], which is used in geology and mining, and others. This paper deals with the process of segmentation of the overlapping objects, which is subsequent to the initial segmentation of the image and the detection of these aggregates. The latter can be performed through various techniques, as morphological granulometry analysis and others. Once the aggregates are found, the task is to separate them into their constituent elements. This is performed by means of a second segmentation process, in such a way that the features of these elements can be analyzed for purposes like classification and differential counting. A well known method for segmentation of aggregates in images, is based in the calculation of the distance transform [5] of a binary image associated to the aggregate, and the application of the watershed transform [6] to it, after having introduced adequate markers. The distance transform of a binary image is defined as follows: for every pixel x in set A, DT(A) is its distance from x to the complement of A,
{
}
DT ( A)( x ) = min d ( x, y ), y ∈ A c .
grayscale image is shown in Fig. 1(c), where the points in the black region in Fig. 1(b) that are farthest from the background (white) will appear as maxima. However, notice that due to characteristics of the figure morphology, several local maxima can appear. Finally the grayscale image of Fig. 1(c) is complemented and the image in Fig. 1(d) is obtained, where the background is white and the former maxima appear now as minima. In this work, this procedure to apply the distance transform is to be referred as “inner” distance transform. It is to this last image to which the watershed transform is applied in order to separate the overlapping objects. The region outside the aggregate to be segmented can be complemented to obtain an appropriate external marker. The spurious minima are a source of over-segmentation, a major drawback that can appear unless adequate markers are built.
(1)
The distance transform of a binary image is usually calculated considering that Ac is the set of 1valued pixels. It results in a grayscale image, whose complement is usually segmented through the watershed transform. However, it is a well known phenomenon that watersheds can lead to a severe over-segmentation unless a good selection and application of markers be performed [5], [7]. Typical results of the distance transform calculation are illustrated in Figure 1. A binary image associated to the two overlapping round objects shown in Fig. 1(a), is obtained from a first “coarse” segmentation, which is not capable of separating them. This first segmentation step can be performed through any of the standard methods according to the characteristics of the addressed problem [8]. The image in Fig. 1(a) is firstly negated to obtain the image in Fig. 1(b). The distance transform is applied to it, and the obtained
Fig. 1: (a) Binary image from overlapping objects, (b) complemented binary image, (c) distance transform of the image in (b), and (d) complemented distance transform. Notice the maxima and minima indicated by arrows in (c) and (d).
There are various grayscale morphological functions [5], [7], that can be used in the process of setting the inner markers, in order to control the negative effects of spurious minima, before applying the watershed transform. Some of these are: • Minima imposition in specific points: this is a morphological function that inserts a − ∞ value in a selected place and eliminates all local minima that could exist in other points of the grayscale image. This can help to create markers, when placed in appropriate points.
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
• Application of the H-minima transform to the result of the inner distance transform. The Hminima transform eliminates all the minima with depth less or equal than a certain positive threshold, and reduces the depth of the remaining minima in the threshold’s magnitude. When an appropriate threshold is used, it can eliminate local minima and create regional minima that can be used effectively as markers. This is implemented through a morphological sup-geodesic reconstruction ∇ D of the image intensity surface f , from the surface increased by the threshold h, with D as the structuring element defining the connectivity, which is denoted as
HMIN h , D ( f ) = f ∇ D ( f + h ) .
(2)
However, finding the adequate place to impose a minimum or determining the appropriate thresholds for the H-minima transform are not trivial tasks: either the markers can be misplaced, spurious minima can remain, or merging of regional minima can occur when they should give rise to isolated markers. Referring to Fig. 1(d), the task is eliminating the minimum in the “bridge” connecting the blobs as well as other spurious minima, while maintaining two isolated minima, approximately at the geometrical centre of each one of the overlapping blobs to be segmented. Jierong and Rajapakse [3] developed an adaptive H-minima transform that proved to yield good results, but still one parameter is to be defined by the analyst, which limits the level of automation. In the adaptive H-minima transform, the threshold h is incremented at steps from a minimum value. This eliminates the shallow local minima and creates extended minima (that are flat connected zones inside the blobs) which can be used as markers. Considering the scheme depicted in Fig. 1, the threshold is increased until the minimum in the “bridge” is eliminated and two regions corresponding to regional minima are found, which are used as markers. The algorithm runs until these two extended minima just merge and then the threshold is reduced in a number Δ of steps of fixed magnitude. This parameter is to be determined by the user, leading to the markers shown in Figure 2.
Fig. 2: Typical markers obtained by means of the adaptive H-minima transform.
Notice that these are the internal markers, that lie inside the original image. The outer external marker can be a regional minimum occupying the background region obtained as the result of the first segmentation step. The adaptive H-minima transform algorithm determines adaptively the threshold h and represents it by hadp through a process that is described in [3] as follows. Let gI be the inner distance transform, and H(gI, h) its H-minima transform at threshold value h. S is the set of all the connected regions obtained in the coarse segmentation and Nj(h) the number of minima within the connected region j ∈ S after applying the transform. The value of h is increased until the number of minima decreases (which is associated with merging of regional minima). The algorithm is summarized below. Considering that the image magnitude is a discrete variable, h is incremented in unitary steps. begin: Adaptive H-minima Transform hadp = 1; Find H(gI , hadp) for connected region j ∈ S do h = hadp; if Nj (hadp) > 1 then repeat h = h + 1; Find H(gI , h) until Nj (h) < Nj (hadp) hadp = h − Δ; else hadp = h; end if Find H(gI , hadp) end for This procedure showed good results, however it has been observed that the selection of the parameters tends to be critical when dealing with more challenging overlapped figures, having more blobs, more irregularities that result in a non-perfect convexity of the overlapping forms and
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
consequently more spurious minima. The initial value of h should be high enough to avoid merging of spurious minima that would make the program to stop generating false markers. On the other hand, the value of Δ (usually a small multiple of h) is to be determined by the user for a particular case, which means an undesirable human intervention that should be avoided. The authors of [3] also introduced the calculation of markers based on the “outer” distance transform, which is a map with the distances from every pixel in the binary image to the resulting regional minima, after application of the H-minima transform. They showed that this allows obtaining a new set of markers from the markers obtained through the previously described process, which for the cases analyzed led to smoother segmentation lines. Another improvement introduced in [3] was weighting the distances when calculating the outer distance transform, with the values of the original regional minima obtained. In this article, new alternatives to obtain the markers based in the previously described method were devised, implemented and tested. The idea behind these alternatives is to avoid the human intervention in the segmentation process, which means that there is no need to define externally the value of any parameter. The modifications realized were:
Fig. 3: Markers, (a) extended minima from the Hminima transform and (b) their centroids, marked with a cross.
2. METHODOLOGY 2.1 Experimental data The methodology used implied selecting firstly a source of images containing overlapping objects, forming aggregates of different sizes and forms. In order to approximate real-life problems, erythrocyte images from blood smear microscopy were drawn from an atlas of hematology available in the Internet [9]. These images are represented in the RGB color space and their size is 745 X 508 pixels. Typical images present aggregates of various sizes as can be seen in Figure 4.
• Substitution of the extended minima markers by their centroids, both with the inner distance transform and with the outer distance transform as well. This allows fixing Δ and eliminates the need of determining this parameter, and the resulting marker will be a point where a minimum can be imposed. The positions of the markers when using the extended minima and their centroids are shown in Figure 3 for an experimental example. • Imposing minima in the positions of the centroids. Minima imposition guarantees that no other spurious local minima will be present when applying the watershed transform. • Imposing minima in the whole extended minima obtained from the H-minima transforms, with Δ fixed to only one step backwards. The objective of this work has been to evaluate the quality of segmentation of overlapped objects using the previously described alternatives, and assess their effectiveness quantitatively by means of the Jaccard coefficient.
Fig. 4: An image of peripheral blood smear in which several aggregates of erythrocytes can be observed.
A conventional segmentation method was applied to the original images in order to obtain binary masks containing the aggregates. Notice that evaluating the quality of this first segmentation process is not included among the objectives of this work. In the binary images obtained in this way, there will be objects corresponding to isolated cells as well as to cell aggregates. The latter were isolated and drawn manually, and then labeled as connected components, until having a representative set of aggregate forms. These
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
aggregates were segmented using the techniques under evaluation. Notice that the development or evaluation of automated methods to identify the aggregates is also out of the scope of this work.
2.2 Marker alternatives for the watershed transform implemented and tested A set of specific combinations of the morphological techniques described in section 1 to obtain the markers, prior to using the watershed transform, were implemented and tested in this work. These are described below, where in all cases the inner and outer distance transforms employ the Euclidean distance. A. Imposing minima ( − ∞ ) in the regional minima obtained through application of the inner distance transform, with step size Δ for the adaptive H-minima transform. B. Calculating the centroids of the regional minima obtained in (A) and imposing minima on them in the inner distance transform map. C. Applying the outer distance transform, with respect to the centroids obtained after having used the inner distance transform. The watershed transform was applied here directly to the outer distance transform map. D. Same as in (C), but imposing minima in the positions of the centroids. E. Use of the weighted outer distance transform with respect to the regional minima. Here the distances from any point to these minima were weighted with the regional minima magnitudes, obtained using the inner and H-minima transforms. Minima imposition was not used. F. Outer distance transform, calculated with respect to the centroids of the regional markers, and weighting the distance values as in (E), without minima imposition. G. Same as in (F), but imposing minima in the centroids. In the following, the letters A to G identifying the previous items will designate the corresponding complete segmentation process of the aggregates.
in Figure 5. Ten aggregates were used in total: 6 with 2 elements, 1 with 3 and 3 with 4, with a variety of forms and spatial orientation. Once the aggregates were segmented, their constituent parts were labeled and used for a quantitative evaluation, in order to determine the error between the ground truth and the results obtained using the different methods under study. There are various forms to evaluate the quality of segmentation algorithms, and in this case calculation of the Jaccard coefficient between the ground truth and the computer-segmented regions was chosen. The Jaccard coefficient, a widely used tool to evaluate the results of image segmentation in different applications [10], is a similarity measure between sets defined as
J ( A, B) =
A∩ B A∪ B
, 0 ≤ J ( A, B) ≤ 1.
(3)
In terms of binary images, the Jaccard coefficient is the ratio between the numbers of pixels in the intersection and in the union, respectively, of the segmented image objects to be compared. Here a value of 1 indicates a perfect coincidence between the two objects, while a value equal to zero means total absence of coincidence, practical cases being in between. The binary masks that resulted from the coarse segmentation process of the previously described set of images, containing aggregates of 2, 3 and 4 cells, were segmented both manually to obtain the ground truth and by methods (A)-(G). Then the Jaccard coefficient was calculated for the different alternatives. In cases where small spurious segments appeared, a situation that will be described in the Results section, these were eliminated through morphological area opening of the images, with a structuring element of an appropriate size, whose area was calculated as half the mean area of the connected sets in the image.
2.3 Evaluation of the segmentation results Evaluating the results of a segmentation process requires an adequate ground truth. In this case, the aggregates selected for testing the algorithms were manually segmented, following the criterion of drawing a straight line manually between the vertices of the concavities that appear in the region of overlapping. Examples of aggregates segmented in this way and used as the ground truth are shown ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
Fig. 5: Examples of binary images resulting from manually segmented aggregates of various sizes.
3. RESULTS AND DISCUSSION The results of the experiments performed will be shown both graphically, through representative images, and numerically by tabulating the Jaccard coefficients obtained for the different cases. Figure 6 illustrates the differences in the reliefs maps that are obtained from the binary image, for four of the cases previously listed. In Figure 6(a) it is shown the relief of the complemented inner distance transform, with minima imposed in the regional minima resulting from the H-minima transform (case A) and 6(b) corresponds also to the complemented inner distance transform, but after minima imposition in the centroids of these regional minima (case B). Figure 6(c) shows the outer distance transform calculated from the centroids of the regional minima, and imposing minima in them (case D). Finally, Fig. 6(d) illustrates the weighted outer distance map that corresponds to the method described in [3] (case E, for a particular value of Δ). The process of segmentation using the inner and the outer distance transforms produced binary images that differ. It was observed that the waterdshed lines that separate the constituent objects, take a form that depends on the alternative used. Here in all cases the same algorithmic implementation of the watershed transform was employed, as otherwise these results could depend upon the specific algorithm used, including the type of connectivity employed, which in this case was 8.
Fig. 6: Examples of relief maps: (a) and (b), inner distance transform, corresponding to cases A and B respectively; (c) and (d) correspond to the outer distance transform, as used in cases D and E respectively.
Figure 7 shows the difference between the segmented objects using the complemented inner and the outer distance transforms, in regard to the form of the watershed lines. In Fig. 7(a) the presence of broken lines can be seen, while this problem was solved in this case through the use of the outer distance transform, as shown in Fig. 7(b). However it was observed during the experimental work that for some images, with lower symmetry and not so regular morphology, the use of the outer distance transform led to the presence of small spurious regions in the segmentation result. This is shown in figure 8 for alternatives E and F, both using the outer distance transform, with the spurious areas marked by arrows. This phenomenon can be attributed to the way in which the specific watershed algorithm employed determines the watershed lines from the distance transform map. If attention is paid to the region marked with arrows in Fig. 8, and look to the same region in Fig. 6(c) and (d), the proximity of the ridge line (that can be observed as a bright line traversing the whole upper blob near the ideal line of separation) to the vertex of the concavity in the points of union of the two blobs, makes difficult to the watershed algorithm to trace the appropriate path. It is worth noting at this point that the Euclidean distance coincides with the Geodesic distance only within convex regions, which is not the case for the aggregates being analyzed, and
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
this could be another influencing factor in these results.
Fig. 9: Elimination of small spurious areas through morphological area opening. The arrow indicates the zone of interest. Fig. 7: Illustration of the broken watershed lines that can appear when using the distance transform, (a) using the inner distance transform, (b) using the outer distance transform.
Fig. 8: Example of segmentation result using the outer distance transform: (a) corresponds to alternative F and (b) to alternative E.
To the effects of eliminating the presence of these small spurious areas, a morphological area opening was used. This operation deletes objects having size in pixels less than a certain threshold. This threshold was calculated in this case as onehalf of the mean area of the constituent elements, of the binary image resulting from the segmentation process. This is a necessary step for evaluation, to make equal the number of connected components in the binary images to be compared, making them compatible when evaluating the quality of the segmentation results. Figure 9 illustrates the effects of area opening on the figure shown in Fig. 8(b).
The results of the segmentation evaluation through the Jaccard coefficient are shown in Table I, for the alternatives A, B, E and F. This constitutes a representative sample because the alternatives C, D, F and G produced, as expected, quite similar results. The similarity in the results of these alternatives occur because minima imposition in the outer distance maps has little or no effect, given the absence of other local minima. The mean values and variances of the Jaccard coefficient values are also shown in Table I, and they reveal that there is not a significant difference between the different methods, with slight tendency to superiority of alternatives A and B (inner distance transform), despite the probable presence of broken watershed lines. This can be attributed partially to the undesirable effect of the spurious segmented areas, which appeared in certain cases when using the outer distance transform. Something that deserves attention was the use, as inner markers for the watershed transform, of the centroids of the regional minima produced by the adaptive Hminima transform, with a fixed value of Δ. This strategy makes unnecessary the human intervention to define the value of Δ, allowing a higher degree of automation of the segmentation process. Notice also that minima imposition helped to eliminate spurious minima, that provoke the appearance of erroneous watershed lines.
4. CONCLUSION In this work, a set of alternatives were proposed Table I: Results of the segmentation of aggregates in terms of the Jaccard coefficients
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS”
minima transform, by their centroids, was found to produce almost the same results that if these regional minima were used as markers, but suppressing the need of human intervention to define a parameter for the segmentation process. Another important issue was the evaluation of using of the outer distance transform, as a means to reduce the presence of broken watershed lines. It was found that in a significant number of cases this certainly improves the smoothness of the watershed lines. However, for certain irregular forms of the aggregates, this method led to the appearance of relatively small spurious segmented areas. Although it was found that these can be eliminated in a simple way by morphological area opening, their presence constitutes a drawback of the method. In summary, the results of the experiments performed suggest that the best strategy to establish the inner markers for watershed segmentation of aggregates is using the inner distance transform, find the extended minima that results from the application of the H-minima transform and then calculate their centroids and use them as inner markers. Future work would consider the morphological calculation of the geodesic distance and use it in association with the outer distance transform, to build the relief map and obtain the markers for the watershed transform, and compare the results against those of the inner distance transform as it was done in this work.
5. ACKNOWLEDGMENT This research was partially funded by the Canadian International Development Agency Project Tier II-394-TT02-00 and by the Flemish VLIR-UOS Programme for Institutional University Co-operation (IUC).
to build the inner markers to be used, when applying the watershed transform to the problem of segmenting aggregates of overlapping objects. The quality of the segmentation results when using these alternatives was experimentally tested, by means of the evaluation of the Jaccard coefficient. To obtain the images to be compared through the Jaccard coefficient, the aggregates used in the evaluation were manually segmented, and then segmented by means of the watershed transform, using the markers built by means of the methods under evaluation. The alternative based on the substitution of the regional minima, obtained from the adaptive H-
6. REFERENCE 1.
2.
3.
4.
Mao, K. Z. et al, “Overlapping/touching cell nuclei segmentation based on analysis of perpendicular distance curve,” in IEEE EMBS Asian-Pacific Conference on Biomedical Engineering 2003. Kyoto-Osaka-Nara, Japan 2003 Nasr-Isfahani, S. et al, “A new approach for touching cells segmentation,” in International Conference on Biomedical Engineering and Informatics BMEI 2008. Hainan, China, 2008. Jierong, C. and Rajapakse, J. C., “Segmentation of clustered nuclei with shape markers and marking function,” IEEE Transactions on Biomedical Engineering, vol. 56, (3), pp. 741-748, 2009. Wang, W., “Rock particle image segmentation and systems,” in Pattern Recognitiion Techniques, Technology and Applications, Peng-Yen Yin, Vienna: I-Tech, 2008, chap. VIII, pp. 197-226.
ISBN: 978-959-7213-01-7
Chinea Valdés, L.; Lorenzo Ginori, J. V.| “EVALUATION OF DISTANCE TRANSFORM BASED ALTERNATIVES FOR IMAGE SEGMENTATION OF OVERLAPPING OBJECTS” 5.
Dougherty, E. R. and Lotufo, R. A., Hands-on morphological image processing. Washington: Spie Press, 2003, p 69. 6. Vincent, L. and Soille, P., “Watersheds in digital spaces: an efficient algorithm based on immersion simulations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, ( 6), pp. 583-598, 1991. 7. Soille, P., Morphological image analysis. Berlin Heidelberg: Springer Verlag, 2003, p. 267. 8. González, R. C. and Woods, R. E., Digital image processing. New Jersey: Pearson Prentice Hall, 2007, 3ra. ed, p. 413. 9. Ichihashi T. et al (1996). Atlas of Hematology, Available: http://pathy.met.nagoyau.ac.jp/atlas/doc/, Accessed: 24/11/ 2009. 10. Ge, F., Wang, S. and Liu, T., “A new benchmark for image segmentation evaluation,” Journal of Electronic Imaging, vol. 16, (3), pp. , 033011-1-16, 2007.
ISBN: 978-959-7213-01-7
