Multiresolution texture segmentation with application to diagnostic

108

IEEE TRANSACTIONS ON MEDICAL IMAGING. VOL. 12, NO. 1, MARCH 1993

Multiresolution Texture Segmentation with Application to Diagnostic Ultrasound Images Russell Muzzolini, Yee-Hong Yang, Senior Member, IEEE, and Roger Pierson

Abstruct- A commonly taken first step in a computer vision system is to reduce the large amount of information present in an image to a point where automated processes can recognize the components in the image. Image segmentation provides a means for determining these basic components. Different components can be characterized by different texture features. These features can then be incorporated into a segmentation algorithm to partition the image into regions, each of which has a homogeneous texture. The accuracy in which the regions are located is determined by how well the textures in the image are characterized as well as the level of detail (resolution) in which the objects are to be recognized. The time it takes to achieve the segmentation is also an important consideration. A new multiresolution texture segmentation (MTS) approach is presented which addresses the issues of texture characterization, image resolution, and time to complete the segmentation. This approach generalizes the conventional simulated annealing method to a multiresolution framework and minimizes an energy function which is dependent on the resolution of the size of the texture blocks in an image. Based on a priori information, the MTS algorithm selects the best feature to determine if a texture block is homogeneous or not. If it is homogeneous, then no splitting is required; otherwise splitting of the texture block into four children is performed. Based on a similar argument, merging is performed by examining the similarity of a texture block with its neighbors. A rigorous experimental procedure is also proposed to demonstrate the advantages of the proposed MTS approach on 1) the accuracy of the segmentation, 2) the efficiency of the algorithm, and 3) the use of varying features at different resolution. Semireal images, created by sampling a series of diagnostic ultrasound images of an ovary in vitro , were tested to produce statistical measures on the performance of the approach. The ultrasound images themselves were then segmented to determine if the approach can achieve accurate results for the intended ultrasound application. Experimental results suggest that the MTS approach converges faster than and produces better segmentation results as compared with the single level approach.

seeks to combine these two endeavors to achieve an approach which can efficiently use a priori information of texture characteristics to segment a class of images. The procedure used to test the MTS approach is as important as the approach itself. A major drawback of much of the research on segmentation algorithms is inadequate testing and evaluation of the algorithms. The use of synthetic images is widespread. Although there is nothing inherently wrong with this, many claims to predict how well a segmentation algorithm will perform on any image are based on the results obtained from synthesized images. Further, these synthesized images are often chosen with little justification or relevance to the images of interest. It is of utmost importance that a rigorous experimental procedure is developed along with appropriate choices for test images: most certainly involving the use of real images. The proposed multiresolution texture segmentation algorithm uses the simulated annealing approach to segment an image. The constrained optimization image segmentation (COIS) algorithm [ I ] uses a similar approach in segmenting an image. The differences are the following. First, the MTS algorithm uses a multiresolution approach to segment an image whereas the COIS algorithm segments an image at a single fixed level. Second, the MTS algorithm uses the best texture feature to determine the type of texture of an image block. The COIS algorithm uses a fixed texture feature or set of features because it is a single level approach. There are no other published multiresolution texture segmentation algorithms that use varying texture features depending on the size of the image block. Third, the MTS algorithm varies the temperature whereas the COIS algorithm fixes the temperature, a parameter used in simulated annealing to control the rate of convergence of the algorithm. In addition to the proposcc! MTS algorithm this paper also proposes a rigorous experimental procedure I. INTRODUCTION to demonstrate the properties of the MTS algorithm and to EXTURE segmentation and texture classification have compare the MTS algorithm to the COIS algorithm. This traditionally progressed along independent paths. The experimental procedure can also be applied to study other new multiresolution texture segmentation (MTS) approach segmentation algorithms. The organization of this paper is as follows. Section I1 deManuscript received December 5 , 1991; revised June 15, 1992. This work scribes the simulated annealing and the constraint optimization was supported by the Natural Sciences and Engineering Research Council approaches. The COIS algorithm is presented in Section 111 of Canada (A0370), the Medical Research Council of Canada and the Saskatchewan Health Research Board. This paper was presented in part at and the proposed MTS algorithm in Section IV. The proposed the IEEE Symposium on Medical Imaging, Sante Fe, NM, 1991. experimental procedure is given in Section V. Experimental R. Muzzolini and Y.-H. Yang are with the Computer Vision Laboratory, results are shown in Section VI and the comparison of the the Department of Computational Science, University of Saskatchewan, MTS and the COIS algorithms is given in Section VII. The Saskatoon, Sask., Canada. R. Pierson is with the Department of Obstetrics and Gynecology, University paper concludes in Section VIII.

T

of Saskatchewan, Saskatoon, Sask., Canada. IEEE Log Number 9206095.

0278-0062/93$03.00 0 1993 IEEE

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

11. SIMULATED ANNEALING AND CONSTRAINT OPTIMIZATION Simulated annealing [2]-[5]is a technique used for large scale optimization problems. Typically, these problems involve combinatorial minimization. There is an objective function to minimize and the space in which the solution is obtained is a large, discrete configuration space such as all the possible allocations of labels to pixel blocks in an image. The space is so large that it is combinatorially impossible to exhaust every possible configuration. Simulated annealing utilizes a much smaller set of configurations from the space and continually modifies the configuration until the objective function is minimized. Simulated annealing draws its analogy from thermodynamics. At high temperatures, molecules of a liquid can move around freely. As the liquid cools the molecules begin to lose their mobility. If the liquid is cooled sufficiently, the molecules form a crystalline structure. This is the minimum energy state for the system. When a liquid cools rapidly, there is not enough time for the molecules to organize themselves in an orderly fashion before they lose their mobility. Thus, the system ends up in a polycrystalline state with higher energy. So the essence of simulated annealing is slow cooling allowing a system to settle into a state of low energy. In 1953, Metropolis et al. [ 2 ] simulated a thermodynamic system. They assumed that the system would change its state with probability

Each time the system moves into a new state with energy the energy E l , from the old configuration is compared to E2 using (1) to provide a measure of how probable it is for the new configuration to occur given the old configuration. If E2 < El the new configuration will always occur ( p is greater than unity). When the state change increases the energy in the system (i.e., E2 > E l ) the new configuration will occur with probability p . As kT decreases (cools) the probability of going into a higher energy configuration decreases. When kT reaches its lowest temperature (i.e., T = 0) the system will have its lowest energy and the current configuration will provide the minimum solution. In order to apply simulated annealing to a problem, the problem must be described in terms of the following components: 1) an objective function to minimize, 2 ) a description of possible configurations, 3) a generator of random changes in configuration, and 4) the control parameter T (temperature) and a cooling schedule telling how to lower T from high to low. Image segmentation can be viewed as a combinatorial problem in which pixels or blocks of pixels are to be grouped into one of possibly many regions. This grouping is minimized when each region contains pixels having the same texture. Minimization is achieved initially by allowing any configuration for the optimal solution. In the case of image segmentation, minimization will result in regions containing pixels having more than one texture. By slowly reducing the rate (temperature) at which random configurations occur, and thus reducing the chance of a region containing pixels with E2,

109

more than one texture, an optimal solution is obtained when the temperature reaches a minimum. At this point the image will have been segmented into homogeneous texture regions. Constrained optimization is similar to simulated annealing [l]. It seeks to optimize an objective function by refining the system configuration. Instead of cooling a system to arrive at a low energy state, constrained optimization fixes the temperature and applies constraints to the objective function. The goal is to minimize the objective function while not violating any of the constraints. The probability of a state change from a configuration with energy El to a configuration with energy E2 is

This equation is the same as (1) with the addition of the constraint conditions A V and the removal of the constant IC. V represents the constraints imposed on the system and X determines the rate at which the constraints are increased. A low energy configuration occurs when V = 0 (no constraints violated) because as X + E. p -+ 0 which rules out the chance of any state changes producing a higher energy configuration. State changes in constrained optimization may be controlled by stochastic relaxation which enumerates the possible state changes from one configuration to the next and then uses the probabilities to choose the appropriate state change. After each state change, the probabilities are updated for the next state change. The objective function is minimized by performing the relaxation process until the probabilities controlling the state changes reach unity (the ideal condition). The effect of V in ( 2 ) is dependent upon the penalty schedule A. As stochastic relaxation proceeds, X is increased, thus increasing the effects of the constraints on the objective function. Consequently, illegal configurations that may have been allowed early in the relaxation process are gradually being eliminated from the solution. Depending on the design, the effect of the constraints can be scheduled to occur early or in a gradual fashion. It is important to note that the constraints should not be applied too early in the relaxation process as the configuration can get stuck in a local minimum. On the other hand, too gradual a schedule should be avoided because it slows down the convergence of the algorithm. 111. THE CONSTRAINED OPTIMIZATION IMAGESEGMENTATION ALGORITHM

Simulated annealing has been applied to the detection of boundaries in ultrasound images [ 5 ] . However, the characteristics of ultrasound images suggest that a texture-based segmentation algorithm may produce a more accurate segmentation [4]. In [4] the goal was to find an appropriate algorithm which could both identify textures and segment an image into regions based on the texture features in the image. In addition, a long term goal of [4] and the present paper is to provide a basis for which a real time solution for 3-D reconstruction of the ultrasound images can be achieved. An algorithm which incorporates these elements (with perhaps an exception for real time computation) is the constrained optimization image segmentation (COIS) algorithm proposed and developed by

I10

IEEE TRANSACTIONS ON MEDlCAL IMAGING. VOL. 12, NO. 1, MARCH 1993

Geman et al. [l]. This algorithm uses a stochastic relaxation process to label similar texture blocks in an image as belonging to the same region. A global measure of the segmentation process is continually updated and when the measure reaches its minimum, the segmentation process is complete. The components presented in the previous section on constrained optimization are the foundation of the COIS algorithm. Recall that in constrained optimization, the objective function to be minimized is subject to some given constraints. In the COIS algorithm, the objective function is a measure of matching labels with a block of pixels, U(labels,pixels), which encourages the placement of boundaries between different texture regions. Minimizing U results in an optimal labeling of the image into its appropriate regions. The constraints imposed on U inhibit the formation of illegal label configurations. For example, very thin regions or very small regions are not allowed to exist. These regions have to be merged with the neighboring regions. Two region models are employed in the COIS algorithm, namely, the region partition and the boundary detection models. The region partition model assigns labels to blocks of pixels in the image. Each label determines to which region a block of pixels belongs. The boundary detection model uses labels to classify boundaiy label sites as being either on or off the boundary between different blocks of pixels. The resulting segmentation is an edge map separating differently textured regions in the image. The boundary detection model is more effective for complex, multitextured images while the region partition model is appropriate when a small number of homogeneous regions are present. Since ultrasound images fall into the latter category, the region partition model is employed in this study. The goal of the region partition model is to segment an image so that all regions in the image having the same texture are determined to belong to the same region. A lattice of label sites is superimposed with the image. Each label site S , in the lattice, is located at pixel (ai 1,a j 1) where 0 5 i . j < ( N - l ) / a and N is the size of the image (assuming the image is N x N pixels). Each block of pixels is centered around site S.The region to which each block belongs is determined by the label at S . Site S is assigned the label which minimizes the energy between itself and its neighboring sites. The optimization function U(labels, pixels) measures this interaction energy and is expressed as

Similarity of pixel blocks is determined through the use of texture measures such as those used in [6]. In (3) @ measures the amount of disparity between pixel blocks. The disparity measures between two blocks of pixels are obtained by evaluating different texture features with the Kolmogorov-Smirnov (KS) statistic [ 11. The KS statistic measures the difference between two probability distributions. In this case, the probability distributions are normalized cumulative distributions of the histograms for each pixel block. The KS statistic is defined to be the maximum distance between the two cumulative distributions of the data sets VI and V,

+

+

where

6{5-t1 =

{

if label, = labelt otherwise

@(texture at s, texture att) -1 1

if texture of pixels at .Y and t are similar if texture of pixels at s and t are distinct.

riiax

IFl(t) -

-m r ) - s d - 1)6(h! < 0.5) (8) sample a local area of texture in the image. If the block size is small then a large amount of unnecessary computation will be where performed when identifying large regions. When a large block s d = b ( d c ( P )> 1) and is used, detail is lost as differences in neighboring texture E-t sr = e m . regions are averaged out. It is more appropriate to use varying block sizes to sample texture in an image. This varying block Merging occurs when d, 2 0 and is expressed as size approach is used in the proposed MTS algorithm. In the MTS algorithm, the image is stored in quadtree d,, = ((2b(mr > T ) - V L , ~ ) S ~-( 1)S(R P ) 2 0.5) (9) structure with the root node representing the entire image and the leaf nodes representing the individual pixels in the image where r a d = G(dc(P)5 t) [4], [7]-[161. Level 5 in the quadtree has a finer resolution t-L than at level :E - 1. Conceptually, this is the same as applying vi,.= e m a lattice on top of the image as in Geman et al.’s approach except that there is a corresponding lattice for each level in if any child, C,, of P is a parent node 0 the quadtree. The problem now is to devise a method for b c ( p ) = 1 otherwise measuring texture between blocks of varying sizes at different and P is the parent of C. levels of the quadtree. Also at issue is the procedure for In (8) and (9) R and r are random numbers in the range [0, determining which blocks within the quadtree should be used to measure the texture within the image. For example, a block I] and t is a predetermined threshold. In the MTS algorithm, at level 2 is used to measure texture for a block of pixels in t = 0. In (9), & ( P ) discourages merging when at least one the image. However, if the region with that texture is large, of the child nodes Ct has children of its own. A node with it may be more appropriate to use the block’s parent at level its own children indicates that there are at least two distinct types of textures within its block. Merging these types of :c - 1. Similarly, if the region is smaller than the block size, the children at level .E 1 will provide a more accurate measure nodes results in averaging distinct textures into one: the exact of the texture in the region. This leads to the use of a split- problem occurring in the COIS algorithm when the block size nnd-merge approach in selecting which blocks are to be used is too large. Both S d and r n d in (8) and (9) are based on the following in measuring texture in an image. A block is split if its four children produce a stronger measure than its own. A block equation: is merged with its three neighbors if its parent produces a stronger measure than its own and its three neighbors. The best d c ( P ) = M ~ ~ i < k r;

(1 1)

otherwise R 2 0.5 and random merging will occur if t-E

m, = e w > r

(12)

where E = d c ( P ) ,the threshold t = 0 and T is a uniformly distributed random number in the range [0, 11. As the temperture kT decreases the probability of a random state change becomes much less likely. When both d, = 0 and d, = 0 in (7), the node C is either split or merged and the energy in U remains the same since @ c ( P )= 0. When either d, = 1 or d, = 1 the energy in U is increased since @ c ( P )= 1. This occurs due to a random state change. When either d, = -1 or d, = -1 node C is not split or merged and the energy in U is decreased since @ c ( P )= -1. The minimum energy in U occurs when no splitting or merging takes place for every leaf node in S,. Using (7) and applying it to the simulated annealing framework the MTS algorithm performs a segmentation of an image by iterating until no more splitting and merging takes place or some maximum number of iterations is reached. In the first case the minimum energy state is reached. In the second case an approximation to the solution results. It is possible that this approximation does not provide a reliable estimate but the approximation can decrease the time it takes to produce an initial segmentation. Since blocks are labeled as they are examined, when iteration completes, all blocks should be labeled. In some cases (terminating the segmentation process at a maximum number of solutions, for example) it is possible that some unknown blocks (blocks having no texture class label) may exist. The identity of these blocks is determined by assigning each block to the texture class with the closest characteristics to the block. A. Pseudocode

This section presents the pseudocode outlining the steps involved in the MTS algorithm. 1) choose an initial resolution, (the level in the quadtree) at which pixel blocks will be examined For each temperature 2) decrease the temperature at the cooling rate of a For R iterations 3) randomly select a node (block), C, in S , 4) use R to determine if splitting or merging of node C is to occur if split a) examine the children of block C using (10) if s d = 0 b) if 6(s, > T ) = 1, split and assign unknown labels c) else split and update labels else merge

a) examine block C and its neighbors using (IO) if md = 0 b) if 6(m, > r) = 1, merge and assign unknown label c) else merge and update label 5 ) update U with (6) 6) if no splits or merges occurred (U is minimum), then all blocks labeled so exit 7) reached maximum iterations so assign closest texture class to the label of any unknown blocks.

V. EXPERIMENTAL PROCEDURE This section details the experimental method used to evaluate the MTS and COIS algorithms and discusses the results obtained with these algorithms. Section V-A identifies the control parameters which affect the MTS and COIS algorithms as well as the control parameters determining the types of synthetic test images used in the experiments. Section V-B describes the experimental environment in which the MTS and COIS algorithms were evaluated. A. IdentGcation of Control Parameters

Control parameters are those variables in an algorithm which affect the ability of the algorithm to segment an image. The COIS algorithm has a number of control parameters to consider. They are summarized in the following list: 1) O-the block size used to sample texture, 2 ) a-the label lattice resolution, 3) r/-the neighborhood size, 4) p-site visitation schedule, 5) A-the penalty schedule, 6) p-number of labels, 7) c,-the thresholds. affects how much detail in the image is detected as well as how much computation is needed at each site in the label lattice. As the block size increases the detail level decreases and the amount of computation increases. a controls the resolution of the label lattice and thus determines how many label sites are used. As 0 increases, the distance between label sites increases, resulting in fewer label sites and less (if any) overlap of adjacent blocks. The amount of computation is also affected by n. When a is small there is a large number of label sites. This requires more computation because there are more sites to comptue the texture measures. rl controls how many sites are involved in assigning a label to a particular label site. The neighborhood also determines which sites in the lattice are used when calculating the KS measure. A neighborhood does not have to consist of physically adjacent sites in the lattice. In fact, the neighborhood in the COIS algorithm consists of randomly selected sites. This is done to achieve a more global estimate of texture in the image. p and A are schedules which affect how sites are visited and how much of an effect illegal label formations have on the labeling process, respectively. The site visitation schedule determines the order in which the label sites have their labels evaluated at each iteration. This schedule could call for some regular pattern of site visitation or it could simply visit sites

-

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

randomly. The penalty schedule X determines how quickly constraints are applied to the minimization function. The faster the schedule, the sooner illegal label formations are ruled out. However, if X is increased too quickly, it is possible to get stuck in a local minimum. This will result in a segmentation which does not accurately reflect the regions existing in the image. p determines how many different regions can exist in the image. A small number of labels will keep the amount of computation per label site lower. The number of differentiable regions will also be low. This is advantageous in images with a small number of differentiable patterns but results in loss of detail if different regions have to be given the same label. The thresholds {c2} are perhaps the most important parameters. They control the sensitivity of the KS measure to differentiate among blocks (see Section 111). The MTS algorithm has two control parameters to consider. They are summarized in the following list: 1) a-the cooling schedule, 2) 0-the number of configurations per iteration. Q determines the rate at which the temperature is lowered at each iteration of the annealing process. When the temperature is cooled too fast it is possible to get stuck in a local minimum. When the temperature is cooled very slowly unnecessary computation (excessive splitting and merging) is incurred. R determines how many state changes are possible at each temperature. When this number is too low it is possible to again get stuck in a local minimum. As with the cooling schedule, when the number is large, unnecessary computation is incurred. Experiments were performed on both synthesized images and real ultrasound images. Real ultrasound images were used to determine if the algorithms could segment the types of images for which their application is intended. Synthesized images were used because they provided a controlled environment in which the algorithms could be evaluated. The controlled environment is described in terms of the ground rruth of the synthesized image. In the ground truth, each pixel’s location and intensity is known. As well, the region to which each pixel belongs is also known. Thus, it is possible to determine which pixels (or block of pixels) have been partitioned into the proper regions in the segmented image. There are several parameters which affect the ground truth of synthesized images: 1) Il-the size of the texture region, 2) 12-the number of distinct texture regions, 3) 13-the texture in a region, 4) I4-the orientation of the texture, 5 ) 0.9. At this point there was a sharp rise in the amount of time required to produce the segmentations (the

i(

= 100,

= 0.99.0.9.0.7 and 0.3.

vegmentation times increased up to 400% for a: = 0.99 and

R = 500). Also, as 62 increased, the amount of time increased. This is intuitive because when R gets larger, more pixel blocks are examined, thus increasing the amount of computation. The number of configuration changes performed was relatively low until (I > 0.95. This suggested that using a schedule with (1 > 0.95 would inevitably result in unnecessary, random configuration changes during the segmentation process. The segmentation results for the synthetic images implied that a relatively fast schedule (low values for a and R) could be used for segmenting images. The segmented real images

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

117

Fig. 9. Segmentation results for real image, I = 2 0 . II = 40: (1)-(4)

..._. ................,......,....,...............

.......................................

n = 300,a

= 0.99, 0.9, 0.7 and 0.3.

..... ....

Fig. 10. Segmentation results for real image convolved with Sobel operator: (1)-(4)R = 300. n = 0.99, 0.9, 0.7 and 0.3.

Fig. 11.

Segmentation results for stretched real images, R = 300. n = 0.9, (1) I = 0 . U = 60, (2) I = 0 . II = 100, (3) I = 2 0 . I I = 60, (4) / = 2 0 . 11 = 100, (-5) segmentation results of (I),(6) segmentation results of (2). (7) segmentation results of (3). (8) segmentation results of (4).

in Figs. 8-10 and the segmentation times and configuration changes, however, implied that a slower schedule (higher values for Q and 0) was necessary to achieve a good segmentation. Since the ultimate goal is to produce an accurate 3D model of an ovarian follicle based on a series of real (not synthetically contrived) ultrasound images, the following parameters were determined to produce an effective schedule for the MTS algorithm: 1) Q = 0.9 and 2 ) R = 300. These parameter settings were used for the remainder of the experiments. It is interesting to note that these parameter settings were not based solely on the results of the synthetic images. Even though the synthetic images contained textures sampled from the real ultrasound images, their segmentation results were not the same (though similar) as those from the

real images. A qualitative analysis of the segmentation results from the real images was also used to determine the parameter settings for a and R. This shows that it is of utmost importance for any segmentation algorithm to be tested with both real and synthetic images. A sample segmentation sequence of a real ultrasound image is illustrated in Fig. 12. Fig. 12(2) shows the initial guess at the segmentation. At this point the label for each block is unknown. As the segmentation proceeded, labeled blocks began to identify the follicular region and the surrounding tissue region Fig. 12(3)-(23). The final segmentation is shown in Fig. 12(24). C. Experiment 2: Image Bias

To determine if the synthetic images used in Experiment 1 were biased towards the segmenting mechanism of the MTS

I18

IEEE TRANSACTIOKS ON MEDICAL IMAGING, VOL. 12, NO. I , MARCH 1993

Fig. 14. Segmentation results for I = 20. I I = 4 0 . 0 = 300.0 = 0.9: (1) test image, shifted left by: (2) 4 pixels, (3) 6 pixels, (4) 2 pixels, ( 5 ) segmentation results of ( I ) , (6) segmentation results of (2), (7) segmentation results of (3), (8) segmentation results of (4).

Fig. 15. Segmentation results for the Sobel operator, R = 3 0 0 . 0 = 0.9: ( I ) test image, shifted left by (2) 4 pixels, (3) 6 pixels, (4) 2 pixels. ( 5 ) segmentation results of ( I ) , (6) segmentation results of (2),(7) segmentation results of ( 3 ) , (8) segmentation results of (4).

Fig. 12. Segmentation sequence of a real image for I = 20. I I = 40. R = 300, r ) = 0.9: ( I ) the stretched image, (2) initial segmentation with no label assignment, (3)-(23) solution after every second iteration, and (24) final segmentation.

Fig. 13. Segmentation results for 12 = 300. o = 0.9: ( I ) original teat image, shifted left by: (2) 4 pixels, (3) 6 pixcls, (4) 2 pixels, ( 5 ) segmentation results of ( I ) , (6) segmentation results of (2). (7) segmentation results of (3). (8) segmentation results of (4).

Therefore, it was only necessary to examine the mislabeling error to see if the synthetic test images were biased towards the MTS algorithm. Figs. 13, 14,and 15 show the resulting segmentations of the original, stretched, and convolved images respectively. Images (1)-(4)in each figure show the synthetic image shifted by 0,

used in Experiment I . As a result, subsequent experimentation used the same set of synthetic images as in Experiment 1. D. Experiment 3: Level Itidepetidence

algorithm (see Section VI-A), it was necessary to perform some experiments with different versions of the synthetic images. It was determined in Experiment 1 that the contents of the synthetic image being segmented had little effect on segmentation time and the number of configuration changes.

The purpose of this set of experiments was to determine if the level at which the segmentation is initiated affects the final segmentation results. The MTS algorithm uses a quadtree structure for keeping track of the different levels of resolution at which the pixel blocks in the image are being examined. The

__

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

n

n

-

I? -

0

Sobel operator

0

2

4

6

P i x e l s Shifted

Fig. 16. Mislabeling error for synthetic images shifted by 0, 2, 4, and 6 pixels: 12 = 300 and o = 0.9.

119

original image the mislabeling error increased from 30 to 40% the level decreased. The variability in mislabeling error was attributable to the accuracy in which each texture was characterized at different resolutions. The real ultrasound images were segmented in the same manner to determine if the results would be consistent with those achieved for the synthetic images. As with the processed synthetic images, the segmentations initiated at varying levels for the real processed images were quite similar. The variability in the segmentations for the original real image was attributed to the randomness and inadequate texture characterization (as with the original synthetic image). The distribution of the configuration changes for each segmented image were examined and found to be consistent with those obtained for the synthetic images. As the level increased, the amount of merging increased, the amount of splitting decreased and the number of random configuration changes remained relatively constant.

E. Experiment 4: Fixed Feature entire image is represented as level 1 in the quadtree. Lower levels in the tree have their nodes representing smaller pixel blocks. Level n is the lowest level in the tree with each of its nodes representing pixel blocks of size N/2(”-1) = 4 where N is the size of the image. By initiating the same segmentation at varying levels, it is possible to determine if the results are independent of the initial level chosen. The set of three synthetic images was segmented first with a = 0.9 and R = 300. The initial level was vaned from 2 to 6 ( n = 6). For the processed images (the stretched image with 1 = 20,u = 40, and the image convolved with the Sobel operator) the segmentations were almost identical for each different level, suggesting that the level at which segmentation is initiated does not have an effect on the final segmentation (the slight differences in the segmentations for the original synthetic image may have been a result of randomness and the inadequate characterization of the textures in their original form). Comparison of the types of configuration changes (split, merge and random) and the distribution of the configuration changes for each segmented synthetic image were examined to determine the effect of the selection for the initial segmentation level. As the level decreased from 2 down to 6, the amount of merging increased while the amount of splitting decreased. This is an intuitive result because when the initial level is high, larger blocks are examined, requiring more splitting than merging to achieve the appropriate detail in the segmentation. Conversely, when the level increased from 6 up to 2 , the amount of splitting decreased as more merging was required to achieve the same detail in the segmentation. In both cases, the number of random configuration changes remained relatively constant (slight fluctuations due to the randomness involved). The effect of the initial level on the mislabeling error was also examined. For the processed images, the error was constant (the stretched image M I% mislabeling error and the convolved image 1 1% mislabeling error) except for a slight decrease with the convolved image when level = 6. For the

It was found in an earlier study that the choice of features to characterize texture was dependent on the size of the texture block [6]. Hence, the segmentation algorithm should account for this fact by selecting the best feature at a given resolution for determining homogeneity of a texture block. The purpose of this set of experiments was to determine if the use of different texture features at varying resolutions produces better results than using a fixed texture feature. In these experiments, images were segmented three different times. The first segmentation was performed using the feature f c which most accurately characterized the texture blocks at the largest resolution (64 x 64 for the synthetic images and 128 x 128 for the real ultrasound images). The second segmentation used the feature which most accurately characterized the texture blocks at the smallest resolution (4 x 4). The third segmentation was performed using a varying feature. The resulting segmentations produced for the original, enhanced, and convolved synthetic images indicated that varying the feature produced the best results. Using the original synthetic image as input, mislabeling errors of 66.02% (fc = lo), 66.6% (fi = 5 ) , and 29.98% (varying fi) were produced. When the enhanced synthetic image was used, mislabeling errors of 50.98% (fi = 6), 6.64% (fi = 23), and 1.17% (varying fZ) were produced. Finally, with the synthetic image, which was convolved with the Sobel operator, mislabeling errors were 43.75% (fi = lo), 7.62% (fi = 9), and 8.5% (varying fc).In Fig, 17, the segmentations for the real ultrasound images all have qualitatively the most accurate segmentations when the feature is varied.

F. Discussion of Experimental Results The purpose of Experiment 1 was to determine the most effective settings for the scheduling parameters a and 0 . By performing numerous experiments on both synthetic and real images it was observed that a relatively slow rate of cooling ( a = 0.9‘3) consistently produced the most accurate

120

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. I , MARCH 1993

Fig. 17. Segmentation of real ultrasound images with R = 3 0 0 . 0 = 0.9: ( I ) original image, (2) f ; = 10, (3) ft = 5, (4) varying f , (5) stretched image with I = 20. U = 1 0 , ( 6 ) f L = 6 , (7) f, = 2 3 , (8) varying f, (9) convolution of the Sobel operator with the original image, (10) f; = 10, ( I 1) f t = 9, (12) varying f t .

segmentation results. The value chosen for R had little effect on the segmentation results as compared to the effect of N. It should be noted, however, that the largest images segmented were 256 x 256 pixels. For larger images it is probable that the effect of R will become more apparent since the number of blocks to examine will increase significantly. Examination of the segmentation results for the synthetic and real images suggested that a schedule with a faster rate of cooling could be used without a significant loss in the accuracy of the segmentations. This small loss was traded for a substantial decrease in segmentation time. As a result ck = 0.9 and R = 300 were used for the remaining experiments. In Experiment 2, bias in the synthetic images towards the MTS algorithm was examined. The texture boundaries in synthetic images seemed to lie in the exact places which would produce the most favorable segmentations for the MTS algorithm. It was determined that the small amount of bias present in the synthetic images was not large enough to cause discrepancies between the results obtained for the synthetic images and the results obtained for the real ultrasound images. Thus, the original versions of the synthetic images were used in subsequent experiments without special consideration to bias. The choice for the initial level at which to begin segmentation was examined in Experiment 3. Since the MTS algorithm examines pixel blocks at varying resolutions, the algorithm should produce the same segmentations independent of the initial level chosen. By segmenting the same image at different initial levels, it was found that the choice for the initial level did not affect the final segmentation results for almost all cases. The multiresolution mechanism of the MTS algorithm was further examined in Experiment 4. By fixing the feature detector for some of the segmentations, comparisons were made to determine if the use of a varying feature detector pro-

duced better results than those obtained with the fixed feature detector. Examination of the segmentation results for the real ultrasound images showed that by varying the feature detector, significantly better segmentation results were obtained. VII. COMPARISON OF THE MTS AND COIS ALGORITHMS The MTS and COIS algorithms were evaluated in a common computing environment. Section V-B described this environment, The experiments used in comparing the MTS and COIS algorithms, along with their results, are described in the following sections. A . Iinpleinentation of the COIS Algorithm The COIS algorithm was implemented using the region partition model. As in [l], the neighborhood size r/ was set to 20, the penalty schedule X was fixed with Xk = 0 for the first ten iterations and increased by 1 for each subsequent iteration. The same penalties for forbidden label configurations were used (see Section 111). The number of labels p was set to 2 (follicle or nonfollicle texture). The site visitation schedule p was determined at each iteration by randomly ordering the sites in the label lattice with one site visited at a time. This schedule is different from the one in [l] where the sites were visited in a raster scan fashion. The most important parameters for the COIS algorithm were the thresholds {c,}l 5 i 5 rn, which controlled the sensitivity of the KS measure to differentiate among blocks (see Section 111). Since the COIS algorithm uses a fixed block size (in [ I ] all experiments had B = 21), it was necessary to determine the block size which produced the best segmentation results according to the types of images being segmented. This required a different set of thresholds for each block size.

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

121

Fig 18 Segmentation of the real image using the COIS algorithm with I = 2 0 11 = 40 U = 6 and (1) j = 5 . ( 2 ) 3 = 7. ( 3 ) 3 = 13. ( 4 ) 3 = 15. U = 7 and ( 5 ) 3 = 5 . ( 6 ) j = 7 . ( 7 ) 3 = 1 3 . ( 8 ) 3 = 1 5 . u = 9 and (9) 3 = :.(lo) j = 7 . ( 1 1 ) 3 = 1 3 . ( 1 2 ) 3 = 15.a = 11 and (13) 3 = 5 ( 1 4 ) 3 = 7 ( 1 5 ) 3 = 1 3 . ( 1 6 ) 3 = 15

For the experiments, the thresholds were chosen as follows: 1) Fix the block size ,O, 2 ) Sample homogeneous texture blocks of size ,O from the images, 3) Compute the KS distance for all block pairs, 4) Update the histogram H with each KS distance, 5) Compute the cumulative distribution function S from histogram H and set the threshold to s, such that c g , , H ( j ) = 0.97. Thus, no more than 3% of the KS distances for blocks from the same texture would exceed the threshold. Thresholds for the four transforms applied to the images in [ 11 as well as raw gray scale values were used in the experiments ( m = 5 ) for each fixed block size.

B. Experiments Before a comparison of the MTS and COIS algorithms could be made, it was necessary to perform a number of experiments with the COIS algorithm. The first set of experiments was performed to determine how the parameters cr and r", affect the performance of the COIS algorithm as well as to determine which combination of these parameters produced the best segmentation results. The second set of experiments followed exactly the same procedure as in the firbt set except that real images were segmented.

C. Experiment 5: Real Images The purpose of Experiment 5 was to determine if real images could be segmented with the same quality and in the same amount of time as the synthetic images.

The real images in Fig. 4(1) and 4(6j were segmented 16 times each with a different combination of (a.P) where (7 E { 6 . 7 . 9 . 11) and p E {5,7. 13,15}. The resulting segmentations for the stretched image in Fig. 4(6) are shown in Fig. 18. Similar results were obtained for the original real image. Fig. 19 shows the segmentation times for the real images at varying block sizes. Figure 19( 1) depicts the Segmentation times for the original real image in Fig. 4( 1j. Here, the block size has a relatively small effect on the segmentation times. In Fig. 19(2), the segmentation times for the stretched real image in Fig. 4(6) are shown. The block size has a more profound effect on the segmentation times with larger blocks ([j = 13,p = 15) requiring less segmentation time. D. Discussion of Results

For the MTS algorithm, synthetic images were segmented with R = 300 and 0 = 0.9. The same set of synthetic images were segmented using the Cols algorithm with cr = 3 and ,O = 13. The MTS algorithm produced the lowest mislabeling error among the segmentations (1.16) while the COIS algorithm achieved the lowest mislabeling error of 9.73%. Based on the mislabeling error the MTS algorithm can be judged more accurate because it located all texture regions with only 1.16% error (98.84% accuracy). Examination of the segmentation times for the MTS and COIS algorithms with the above parameter settings shows the superiority of the MTS algorithm. On average the MTS algorithm required =I75 s to segment the synthetic images while the COIS algorithm required ~ 6 8 0 0s to segment the synthetic images. The MTS algorithm produced segmentations

122

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 1 , MARCH 1993

Fig. 19. Time to segment real image using the COIS algorithm: (1) original real image, ( 2 ) 1 = 20.11 = 40.

in a fraction of the time needed by the COIS algorithm. For the real images segmented in Experiment 5, the MTS algorithm segmented the images in Fig. 4(1) and 4(6) in 438 and 103 s while the COIS algorithm (with @ = 6) required 6723 and 7257 s, respectively. It is obvious that the multiresolution approach used in the MTS algorithm significantly reduced the number of blocks that have to be examined during the segmentation process, and thus required less time to segment an image.

VIII. CONCLUSION Numerous experiments were performed with the MTS algorithm to determine its ability in segmenting both synthetic and real images. In Experiment 1 it was found that the scheduling parameters a and R could be chosen such that the segmentations produced were accurate and the time required to produce the segmentations was small. In Experiment 2 it was determined that the synthetic images used in the experiments were not significantly biased towards the MTS algorithm while in Experiment 3 it was found that the initial level at which the segmentation was started did not affect the final segmentation. In Experiment 4 it was determined that the multiresolution mechanism performed better when varying the feature measure rather than fixing the feature measure for different resolutions. Experimentation with the COIS algorithm and its parameter settings suggested that for the set of images segmented, there was an optimal block size (13 x 13) which produced a good segmentation in a consistent amount of time. As well, the accuracy of the segmentations increased as the resolution at which blocks were examined increased (i.e., low (T values). However, as the resolution increased the segmentation time also increased. Thus, producing a segmentation with a low mislabeling error required a significant amount of time. A number of conclusions may be inferred by examining the segmentation results from the MTS and COIS algorithms. The MTS algorithm has been shown to produce more accurate results in less time than a current texture segmentation algorithm, the COIS algorithm. The lower mislabeling error achieved by the MTS algorithm can be attributed to the fact that it makes better use of a priori knowledge than the COIS algorithm. The

MTS algorithm uses a feature table which characterizes each unique texture at varying block sizes while the COIS algorithm uses a threshold at a fixed block size to signal nonuniqueness among different textures. By ranking the features in the feature table only the best feature at each resolution is used by the MTS algorithm. Second, the amount of time required to segment images was related to the number of blocks that have to be examined during the segmentation process. Because of the multiresolution approach the MTS algorithm characteristically examined far fewer blocks than the fixed number of blocks examined by the COIS algorithm. As the desired level of accuracy for segmentation was increased, the difference in the number of blocks examined by each algorithm was quite significant. Thus, the COIS algorithm will require much more time than the MTS algorithm to segment an image. Conversely, the COIS algorithm produces a much higher mislabeling error when segmenting an image in the same amount of time as the MTS algorithm. This is because blocks have to be examined at a lower resolution which results in the higher mislabeling error. There are a number of advantages with the MTS approach: it can segment images in the presence of noise (evident through testing of the ultrasound images), can adapt to different types of images, is unaffected by linear changes in the illumination, and uses local measures in a global context. Most importantly, the MTS approach can use a priori information about the textures it is segmenting to reduce or eliminate the dependence on any thresholds. The research in this paper represents a first attempt in applying multiresolution texture segmentation to diagnostic ultrasound images. The resulting segmentations of the ultrasound images can also be transformed into a three-dimensional model of a follicle [4] thus providing a nondestructive means of examining the follicle in much more detail than is available in a single two-dimensional cross section.

IX. ACKNOWLEDGMENT The authors would like to thank Dr. C. Meyer and the anonymous referees for their comments.

MUZZOLINI et al.: MTS WITH APPLICATION TO DIAGNOSTIC ULTRASOUND IMAGES

REFERENCES [ I ] D. Geman, S. Geman, C. Graffigne, and P. Dong, “Boundary detection by constrained optimization,” IEEE Trans. Pattern Anal. Machine Intell., vol. 12, pp. 609-638, July 1990. [2] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes in C. Cambridge, England: Cambridge University Press, 1988, pp. 343-352. 131 P. J. M. van Laarhoven and E. H. L. Aarts, SimulafedAnnealing: Theory and Applications. New York: Kluwer-Academic, 1988. 141 R. E. Muzzolini, “Segmentation of Ultrasound Images Using Texture Discrimination,” M.Sc. thesis, Dep. Computational Sci., Univ. Saskatchewan, Saskatoon, Sask., Canada, Oct. 1991. 151 N. Friedland and D. Adam, “Automatic ventricular cavity boundary detection from sequential ultrasound images using simulated annealing,” IEEE Trans. Med. Imag., vol. 8, pp. 344-353, Dec. 1989. [ 6 ] R. E. Muzzolini, Y. H. Yang, and R. A. Pierson, “A framework for the evaluation of feature detectors with application to diagnostic ultrasound images,” in IEEE Med. Zmag. Con$. Santa Fe, NM, Nov. 1991. [7] C. Bouman and B. Liu, “Multiple resolution segmentation of textured images,” IEEE Trans. Partern Anal. Machine Intell., vol. 13, pp. 99-1 13, 1991. [8] H. Samet, “Neighbor finding techniques for images represented by

[9] [IO] [ll] [I21

[I31 [ 141

[IS] 1161

I23

quadtrees,” Comput. Graph., Image Processing, vol. 18, pp. 37-57, 1982. H. Samet and R. E. Webber, “Hierarchical data structures and algorithms for computer graphics-Part 1,” IEEE Trans. Comput. Graph. Appli., pp. 48-68, May 1988. -, “Hierarchical data structures and algorithms for computer graphics-Part 2, IEEE Trans. Comput. Graph. Appli., pp. 59-75, July 1988. A. D. Gross and A. Rosenfeld, “Multiresolution object detection and delineation,” Comput. Vision, Graph., lmugr Processing, vol. 39, no. 1, pp. 102-115, 1987. T. H. Hang and A. Rosenfeld, “Compact region extraction using weighted pixel linking in a pyramid,” IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-6, pp. 222-229, Mar. 1984. M. F’ietikainen and A. Rosenfeld, “Image segmentation by texture using pyramid node linking,” IEEE Trans. Sysr., Man, Qbernef., vol. SMC-11, pp, 822-825, Dec. 1981. H. J. Antonisse, “Image segmentation in pyramids,” Comput. Graph. Image Processing, vol. 19, pp. 367-383, 1982. P. Meer, “Stochastic image pyramids,” Comput. Vision, Graph. Image Processing vol. 45, pp. 269-294, 1989. V. Cantoni and S. Levialdi, Pyramidal Systems for Computer Vision. New York: Springer-Verlag, 1986, pp. 215-288.