A multiscale dynamic programming procedure for ... - CiteSeerX

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 19, NO. 2, FEBRUARY 2000

127

A Multiscale Dynamic Programming Procedure for Boundary Detection in Ultrasonic Artery Images Quan Liang, Inger Wendelhag, John Wikstrand, and Tomas Gustavsson*

Abstract—Ultrasonic measurements of human carotid and femoral artery walls are conventionally obtained by manually tracing interfaces between tissue layers. The drawbacks of this method are the interobserver variability and inefficiency. In this paper, we present a new automated method which reduces these problems. By applying a multiscale dynamic programming (DP) algorithm, approximate vessel wall positions are first estimated in a coarse-scale image, which then guide the detection of the boundaries in a fine-scale image. In both cases, DP is used for finding a global optimum for a cost function. The cost function is a weighted sum of terms, in fuzzy expression forms, representing image features and geometrical characteristics of the vessel interfaces. The weights are adjusted by a training procedure using human expert tracings. Operator interventions, if needed, also take effect under the framework of global optimality. This reduces the amount of human intervention and, hence, variability due to subjectiveness. By incorporating human knowledge and experience, the algorithm becomes more robust. A thorough evaluation of the method in the clinical environment shows that interobserver variability is evidently decreased and so is the overall analysis time. We conclude that the automated procedure can replace the manual procedure and leads to an improved performance. Index Terms—Artery, boundary detection, dynamic programming, imaging, ultrasonic.

I. INTRODUCTION

N

ONINVASIVE methods of measuring intima–media thickness (IMT) and lumen diameter (LD) of carotid and femoral arteries from -mode ultrasonic images play an important role in epidemiological studies and clinical trials relating to atherosclerotic disease [1], [2]. Measurements of IMT and LD are defined as the average distance of interfaces between vessel tissue layers [3], [4]. In order to determine the interface location, computer-based interactive tracing systems are commonly used. Early research has shown that there is a highly significant correlation between the measurements so obtained and those obtained from histological evaluation [5], [6]. The manual tracing approach, however, is time consuming Manuscript received July 28, 1998; revised December 13, 1999. This work was supported in part by the Swedish Research Council for Engineering Sciences under Grant No. 97-620, and in part by the Swedish Heart-Lung Foundation, the Swedish Medical Research Council under Project B96-27X-10880-03A, and Astra Hässle, Mölndal, Sweden. The Associate Editor responsible for coordinating the review of this paper and recommending its publication was M. Sonka. Asterisk indicates corresponding author. Q. Liang and *T. Gustavsson are with the Department of Signals and Systems, Chalmers University of Technology, Sweden (e-mail: [email protected]; [email protected]). I. Wendelhag and J. Wikstrand are with the Wallenberg Laboratory for Cardiovascular Research, Sahlgrenska University Hospital, Göteborg University, Sweden (e-mail: [email protected]; [email protected]). Publisher Item Identifier S 0278-0062(00)02299-0.

and based on subjective operator assessment and therefore inevitably results in inter- and intraobserver variability. Furthermore, manual tracing may cause a drift in measurements over time. Our previous studies of the inter- and intraobserver variability of this manual method can be found in [3] and [7]. Efforts have been made to make the measurement less operator dependent by introducing automated image analysis procedures. This paper presents a new automated procedure which is essentially different from other published automated methods. It is based on multiscale dynamic programming (DP) and includes optional modification by a human operator. In the DP procedure, the local measurements of the echo intensity and the intensity gradient of the image, and the boundary geometrical constraint are included as weighted terms in a global evaluation function, the cost function. The optimal solution of the cost function is the desired boundary. The human intervention, if needed, is also incorporated in the framework of DP. In handling images of varying quality, the present multiscale DP method requires few human modifications and normally no operator guidance. Variability due to human factors is thus reduced and the whole reading process is significantly simplified and less time consuming. The paper is organized as follows. Section II presents the artery boundary detection problem and gives a brief review of previous work. Section III explains the rationale behind our method. In Section IV, we present the proposed method as a multistep procedure, and describe related techniques. We give a brief account of the method and results of a clinical evaluation of the technique in Section V. Details of the clinical evaluation are reported in separate papers [8], [9]. A more in-depth discussion of our technique is presented in Section VI.

II. THE BOUNDARY DETECTION PROBLEM A. Ultrasonic Artery Images Fig. 1(a) shows a representative image of a carotid artery. The femoral artery has a similar appearance. The echoes in the region of interest can be schematically grouped into seven echo zones Z1–Z7 [Fig. 1(b)]. Previous studies [3], [4] have shown that the leading edge (upper side) of Z3, Z5, and Z7, denoted as I3, I5, and I7, can be mapped to the near-wall intima–lumen interface, the far-wall lumen–intima interface, and the far-wall media–adventitia interface, respectively. Consequently, the distance between I3 and I5 represents the LD and the distance between I5 and I7 is the far-wall IMT. With this understanding, the determination of ultrasonic measurement of the artery becomes

S0278–0062/00$10.00 © 2000 IEEE

128


(a) Fig. 1.

(b)

(a) Carotid artery image. (b) Definition of echo zones and interfaces.

equivalent to accurately detecting the echo boundaries I3, I5, and I7. B. Previous Research Efforts have been made by numerous investigators worldwide to try to find an approach for boundary detection in ultrasonic superficial artery images which is less reliant on human operators. The first attempt in this respect was published in 1992 by Touboul et al. [10]. The measurement was performed on regular arterial segments where the intensity profile showed clear two-pulse patterns which corresponded to echoes from the lumen–intima and media–adventitia interfaces. At locations assigned by the operator, the IMT was automatically computed as the distance between the midpoints of the rising slopes of the two intensity pulses. Gariepy et al. in 1993 presented a system by which the operator traced the position of the wall and then the computer located the lumen–intima and media–adventitia interfaces by discriminating changes in gray levels [11]. From these detected interfaces, the LD and far-wall IMT were computed. In 1994, Selzer et al. described a multistep procedure [12]. First, the approximate position of the boundary was manually traced. Guided by this approximation, an edge-detecting procedure generated a set of conditional edges by applying a maximal local intensity gradient criterion. These edges were tested for edge strength so as to eliminate weak edges. Accepted edge pixels on the I5 and I7 interfaces were paired. The IMT was finally computed as the mean distance between accepted pixel pairs. The above three methods all required an initial human operator setting, which can potentially introduce variability due to subjective judgement. The determination of the boundary location was entirely based on local evaluation of a single image feature, that is, either the echo intensity or the intensity gradient. These methods can derive accurate reproducible results if the echo image shows clear and smooth interfaces. Unfortunately, in clinical practice, most of the images to be processed contain speckles and echo dropouts and the interfaces are irregular, as shall be seen in Section III. It therefore becomes impossible to correctly detect the boundary in these regions using the simple criterion for the methods described.

In 1994, we suggested a global optimization approach based on DP, which took echo intensity, intensity gradient, and boundary continuity criteria into account [13]. Kozick reported a similar procedure in 1996 [14]. However, the latter still required an initial human setting which is not needed in our method. This also applies to the algorithm presented by Liang et al. [15] who employed DP in conjunction with a quality assurance procedure. Since our initial report in 1994, further improvements in the robustness and accuracy of our algorithm have been made through introducing new techniques. In the following we describe our improved method in detail.

III. RATIONALE The existence of ultrasonic speckle noise makes the accurate definition of a boundary very difficult. Ideally, the echo streak from a smooth interface should have similar axial intensity profiles at different cross sections. However, due to scattering, speckles and other imaging artifacts, the shape of the profile may vary significantly. Fig. 2 shows an example of a carotid artery far-wall image and its intensity profiles at different cross sections. The main waves in the intensity profiles correspond to the echo from interface I7. It can be seen that the point of maximal gradient on the left slope of the main wave could be near the beginning of the slope or close to the peak. A line connecting these points does not form a meaningful boundary. Moreover, in bulb images or images where plaques are present, the far-wall boundary is usually not perpendicular to the propagation direction of the incident acoustic wave. The echo therefore becomes extremely weak or disappears entirely over a wide range of locations in the longitudinal direction. In these regions, it becomes impossible to determine the artery boundaries by using a conventional edge concept [16]. Rather than focusing entirely on points of maximal gradient, we could try to mimic the boundary finding mechanisms associated with a human expert. Although a most difficult task, it is essential to understand, at least to a certain degree, these interpretation mechanisms and find ways of implementing them in a computer-based boundary detection algorithm. In this context, a few points can be noted.

LIANG et al.: A MULTISCALE DYNAMIC PROGRAMMING PROCEDURE FOR BOUNDARY DETECTION IN ULTRASONIC ARTERY IMAGES

(a)

129

(b)

Fig. 2. Maximum gradient points near the interface I7. (a) Far-wall image with overlaid maximum gradient points near I7. (b) Equally spaced cross-sectional intensity profiles with associated points of maximum gradient. A line connecting these points does not form a meaningful boundary.

1) Cost Function and Dynamic Programming: In human boundary determination, the global view prevails over the local view. This suggests that the automated detection algorithm should also invoke an optimization procedure including a global criterion, which in our case is a cost function. To efficiently search for the global optimum of the cost function, DP [17], [18] is a unique tool. 2) Multiple Image Features: A human observer perceives multiple image features, such as echo intensity and intensity gradient. Also, boundary formation is considered. The cost function in the automated procedure also takes similar factors into account by including them as cost terms. 3) Multiscale Detection: Research shows that in the human vision system, spatial frequency components are detected independently in different channels [19]–[21]. Perceptually, the “where” information, which is contained mostly in the lower frequency components, drives the attention window to direct the subsequent processing for obtaining the “what” information [22]. The latter concerns the detail of the object and is mostly contained in the higher frequency components. A similar approach for machine detection of the artery boundaries would be a multiple-scale detection procedure. The approximate location of the artery wall is detected in the coarse-scale image where the lower frequency components are emphasized. This now guides a detailed artery boundary search in the fine-scale image where all the higher frequency components are retained. 4) Fuzzy Expression: Human perception processes often involve imprecise descriptions of the input information. If we let our automated procedure represent image features in fuzzy expressions, by adjusting the fuzzy membership function we may have more alternatives in imitating the human interpretation of the images. 5) System Training: A priori knowledge plays an important role in the observer’s perceiving and decision making process. In the automated system, such knowledge can be included in the design of the cost function terms. Moreover, it is believed that the expert data, that is, the manual tracing of the boundary by

an expert, contain vast human knowledge. They can be used to train the system. 6) Human Intervention: As pointed out above, the boundary search is based on a broad and global view of the image. If the automated detection system needs human assistance in ambiguous cases, this assistance should also be given within the framework of global optimality. Therefore, the human intervention is formulated and included in the cost function. IV. THE AUTOMATED ARTERY BOUNDARY DETECTION METHOD Based on the foregoing principles, a multiscale DP procedure for detecting the boundaries is described in Sections IV-A–D. Section IV-E explains the computing of the measurements. Sections IV-F and IV-G describe two important aspects, system training and handling of human intervention, in implementing the system. In this work, three types of superficial arteries, the common carotid artery, the carotid artery bulb, and the common femoral artery, were involved. In the following context, abbreviations CCA, Bulb, and CFA are used to refer to these three types of images respectively. A. Fuzzy Expression Before describing the cost function which is the core in the DP method, we look at how image features are represented in cost functions. In ultrasonic images, real object information is transformed, distorted, or corrupted. Still, a human observer has the ability to interpret these vague uncertain imprecise messages and make a valid decision based on them. Fig. 3 shows an example of the manually traced interface I7. Note that the pixels of the line are not always immediately above a bright area. An experienced operator understands that due to the complex defects in the echo, the dim echo at may not necessarily imply less ability to attract the boundary than the bright echo at . The message from a local image feature is not used as a rigid criterion in determining the boundary but as a variable which

130


reflects the degree to which a pixel is a boundary pixel. Fuzzy set theory [23], [24] offers a powerful tool for describing such imprecise knowledge. To apply the concept of a fuzzy set at the image feature level [25], an image feature can be represented as a membership value representing the degree to which a pixel under consideration is being viewed as a boundary pixel. Consider the boundary pixel set as a fuzzy subset where is composed of all the pixels in the area of interest. as a function We write the membership function of pixel of an image feature . In the cost function, instead of the membership function is using feature value used. It offers a tuning point to adjust the cost function so as to better reflect an experienced observer’s interpretation of the image feature. For example, the expert would generally mark an artery boundary above a bright streak of echoes, while faint echoes are less likely to be marked. However, when the echo’s brightness reaches a certain level, the expert will regard it to be as important as those brighter echoes when making a decision on the location of the boundary. (see, e.g., Fig. 3) The type of shown in Fig. 4(a) reflects this membership function expert’s interpretation of the image feature. After selecting the type of membership function it is important to determine the used in the function. Unfortunately, experts’ exthreshold periences are often expressed in a vague way, rather than analytically. Hence, we use a large collection of boundary lines drawn by the expert, analyze the intensity around the boundary especially at places such as point in Fig. 3, and empirically determine this threshold. Fig. 4(b) is another type of membership function used in this work, which functions in a way that is contrary to the type of function in Fig. 4(a). The use of the fuzzy expression in the cost function was proven, in the training, to be crucial (see Section IV-F).

(b) Fig. 3. (a) Far-wall image. (b) The manually traced interface I7. The average intensity below A is higher than that below B and so is the intensity gradient. The expert, however, determines that the interface passes B instead of A. This is not only due to a line smoothness constraint; the weak echo near B also contributes to the decision.

so that a stronger image feature at will yield a lower output, hence, a lower local cost which means a stronger atis tracting force exerted on the polyline. The term related to the geometrical force. A larger difference between yields a higher the vertical positions of and value and hence, a higher local cost which means the connecis less favored. The effect of this term tion between and is the is to keep the line smoother. The desired boundary which minimizes optimal (4)

B. The Cost Functions and Cost Terms Cost functions are built for each of the interested boundaries I3, I5, I7, the auxiliary boundary I2 (see Section IV-D), and the near- and far-wall estimates (see Section IV-D), in each type of the images (i.e., CCA, Bulb, and CFA). 1) Definition of the Cost Function: In an image grid of consider a deformable polyline containing nodes, one in each column composed of candidates, i.e., The cost function denoted as or is defined as a sum of local costs along a candidate boundary or the polyline

(1) where the local cost at

(a)

is composed of two terms (2) (3)

are image feature terms is in which the number of features used). They are transformations of image (see below) and are specifically designed feature values

2) Definition of the Cost Terms: The cost terms in a cost function associated with a certain boundary must reflect the characteristics of the geometrical formation of the boundary and image features in the neighborhood. These characteristics differ from boundary to boundary. Hence, the cost terms used may also differ from boundary to boundary. As an example, we list the description of the cost terms used in the cost function for I7 in the CCA image. In these terms, intensity and gradient values by their respective maxare all normalized to the range of and imum values in the whole image. The expressions are membership functions which are illustrated in Section IV-A and Fig. 4. : Boundary smoothness. In favor of a Term: smoother line. Representation: Square of the finite difference of the distance to a reference line at node , i.e., (Fig. 5). : Intensity gradient. In favor of a line located at Term: a higher intensity gradient position. grad where grad is the Representation: downward intensity gradient, which corresponds to the upper edge of an echo. The intensity gradient is estimated in a 5 5 neighborhood window and normalized. It is the slope at the center of a third-order polynomial surface fitted to the intensity


(a)

131

(b)

Fig. 4. Membership functions. f = Value of image feature, normalized; 1 (f ); 2 (f ) = membership function of a pixel belonging to a fuzzy boundary subset a in terms of image feature f ; T ; T = threshold.

and not a realistic alternative. To solve this problem, we rewrite (1) in a recursion form and generalize it by replacing with

Fig. 5.

Finite difference of distance d at p : d(p )

0 d(p

):

values within this window. The threshold used in the memberis ship function Term: : Brightness below. In favor of a line immediately above a wide strong echo. int where int is the avRepresentation: (here ) pixels right below and erage intensity of is . normalized. The threshold used in : Darkness above. In favor of a line right below Term: a narrow dark streak. int where int is the average Representation: pixels right above and normalintensity of (here, is . ized. The threshold used in C. Dynamic Programming the optimum in terms For a given cost function of minimizing the cost is searched. Ideally, the local-cost surface around a desired boundary is convex, which forces the polyline to go along a path approximating the valley. However, for a real image, the convex is always severely distorted. In the case shown in Fig. 6, the interference originating from noise and the nearby boundary I5 appear as local-cost valleys or local minima. The search for the solution must be carried out thoroughly in the searching space to avoid having it locked by a local minimum. A straightforward exhaustive search in the whole space, in in terms of this case, has a computational complexity numbers of cost function evaluations, which is too expensive

(5) -stage process, The summation in (1) is expressed as an . With the so that a DP procedure can be applied to derive the following derived DP procedure, the computational complexity . can be dramatically reduced to We express the candidate minima of the cost function of polyas a function of the end node line (6) By applying (5), we can express it as a multistage cost accumulation process (7)

(8) costs are accumulated according to (7) To search for the location of which defines and (8). At each stage as in (8) is recorded for each instance of . Then the

132


(a)

(b)

(c)

(d)

Fig. 6. Local-cost surface for interface I7 and its cross-section profiles. (a) A far-wall image. (b) The local-cost surface for interface I7. (c) and (d) Two cross sections of the local-cost surface. cost valley corresponding to interface I7.* cost valley due to interference from interface I5. + cost valley due to interference from speckles. Note that interferences appear as local-cost minima which could be smaller than the local minimum due to I7 and may trap the boundary. A global evaluation of the cost function could help to solve this problem.

=

nodes of the optimal boundary by back tracking

=

defined in (4) can be derived

(9)

(10)

D. Multiscale Dynamic Programming 1) The Scale Space: The presence of speckles has a significant impact on the correctness of the detection. In the methods discussed in Section II-B [10]–[12], [14], [15], manually set points or initially traced boundaries were used to avoid this problem. In order to facilitate an automated detection without an initial setting, we performed multiscale DP based on the properties of the scale space [26]–[28]. The scale space of an image is constructed through convolution of the image with a two-dimensional (2-D) Gaussian density kernel with zero mean and standard deviation, i.e., the scale σ. In the case of an artery image, the echo has a strong tendency to display horizontal streaks, and therefore, the behavior of the 2-D image scale space is similar to that of a -direction one–dimensional (1-D) signal scale space. Hence, as σ increases and the number of maxima of intensity and intensity derivatives decreases, no new extrema emerge. Detailed object structures

=

vanish, while gross structures persist. When σ decreases, the extrema can be continuously tracked until they reach the extrema in the original image where = 0. Based on these features, in our method a coarser scale artery image is used to identify the approximate position of the walls and then, under the guidance of the wall estimates a finer scale, the original image, is used to localize the interfaces of the detailed layer structure of the wall. In Fig. 7, a vertical cross-section of an artery image is represented as a 1-D signal. It is shown how the estimates of the walls in the coarse-scale (points and can guide the detection of the relevant interface I7 (point , and the auxiliary interface I2 (point which is also of interest for detecting I3. 2) The Multistep Procedure: The multistep procedure for detecting the boundaries is illustrated in Figs. 8 and 9. 3) Coarse-scale dynamic programming: In the coarsescale, by analyzing the shape of the horizontal projection of the intensity, the horizontal vessel center line can be easily determined, even if the vessel is not well placed horizontally [Fig. 9(a)–(b)]. DP is performed in the upper and lower part of the image separated by the center line to locate the approximate positions of the near- and the far wall [Fig. 9(c)]. When applying the DP procedure, a vertical window scans from left to the right, starting at column 1 and ending at column . At each column scanned, the optimized connection is searched for each point in that column and the cost accumulated. The from which optimized connection is the connection to minimizes the accumulated cost for the section of a candidate [see (8)]. boundary starting at column 1 and ending at point


133

=

Fig. 7. The scale-space image of a 1-D signal: vertical cross-section of an intensity profile of an artery image. (Bottom) Solid line the 1-D signal at its finest scale; dashed line the same signal at the coarse scale. (Top) The scale-space image of the signal. The lines are tracks of the extrema which are zero crossings of the second derivatives of intensity. At coarse scale the number of extrema diminishes. The far wall (a) and near wall (b) can easily be identified. As the scale reduces, the tracks of the extrema lead to the location of the relevant interface I7 (c) and the auxiliary interface I2 (d).

=

Fig. 8. The multistep detection procedure.

At the end of the scanning, the minimum is searched in the accumulated cost function. The position of the minimum cost point is the right-hand side end of the desired boundary [see (9)]. By backtracking, the wall estimate is derived [(see (10)]. In coarse-scale DP, the reference line used is the horizontal line. Note that in the coarse scale, the significant depression of speckles and the detailed wall structures notably improves the robustness of the wall estimation. 4) Fine-Scale Dynamic Programming: The detection of I3 and I5 is difficult due to the fact that zones Z3 and Z5 are weak and the nearby zones Z1 and Z7 are strong. On the other hand, I2 and I7 can easily be identified. This leads to the detection strategy. In the wall regions, I2 and I7 are detected first. They are detected in narrow neighborhood bands of the two wall estimates with the trajectories of the estimates as reference lines [Fig. 9(d)]. After that, with the smoothed lines of I2 and I7 as reference lines, in the two lumen–intima interface regions formed below I2 and above I7, I3 and I5 are detected. [see Fig. 9(e)]. As the vertical sizes of these two regions are small, and as they are put in proper position, interference from speckles is mostly sidestepped and interference from Z1 and Z7 is avoided. Here, preknowledge about the spatial relation between the interfaces greatly helps the correct detection of the lines. One advantage of using the DP method in this context is that it always gives a single and continuous line as the solution. This not only ensures

that the detected boundary will bridge the disjointed echo, but it also gives the algorithm the ability to combat the nearby speckle interference. E. Measurements A linear regression line is derived from the detected I7. Its slope is used as the estimated vessel slope. Perpendicular to this slope, the distance between I3 and I5, as well as the distance between I5 and I7, is measured between paired pixels. These distances represent LD and far-wall IMT, respectively. Mean, maximum, and minimum values are extracted. F. Training and Validation In a system built as the one described above, the weights of the cost terms in the cost function have yet to be set. They are determined through a training procedure which utilizes expert data. 1) Data Preparation: Images of three types, i.e., CCA , the Bulb , and the CFA (image set size , were recorded with an ultrasound scanner (Acuson 128) equipped with a 7-MHz linear transducer. An experienced technologist manually traced the artery boundaries. The boundary coordinates and measurements of LD and IMT were stored on disk for subsequent use as expert data.

134


(a)

(b)

(c)

(d)

(e) Fig. 9. Boundary detection in an artery image. (a) The original image. (b) A low-pass filtered image and its horizontal intensity projection. (c) Estimating the wall position in the coarse image. (d) Detecting interface I2 in the near-wall region and I7 in the far-wall region in the fine-scale image. (e) Detecting interfaces I3 and I5 in lumen-intima interface regions in the fine-scale image.

2) Training: Each of the image sets obtained (CCA, Bulb, or CFA) was split into two equally sized subsets. One for training and the other for validation. The weights of the cost functions were determined for each boundary in each image type as follows (Fig. 10). The cost function defined in (1)–(3) has a weight set with discrete weights constrained by (11) (12)

For each setting of , boundaries of an artery interface were detected in all of the training images and then compared with the corresponding expert data. Errors were defined as the mean of the squared distances in the vertical direction between the corresponding pairs of pixels on the two lines. The overall error

between the detected result and the expert data, given a certain setting of weights was defined as

(13)

is the error at the th pair of pixels in the th where was applied, and is the length image when weight set of the boundary in the th image. The optimal weight vector, , was searched for exhaustively in terms of minimizing in the weight space confined by (11) and (12). The resulting weight set was used for subsequent validation. To reduce the computation required in the searching, a coarse step size of 0.05 was used initially. In the vicinity of the estimated best point, a fine step size of 0.01 was then used. Fig. 11 shows an example, . Note that due to the constraint (12), while the where total number of parameters is three, the number of independent


135

used in the coarse scale to determine the estimation of the nearand far-wall locations, no expert data were available. Hence, all the weights were determined empirically. As a matter of fact, these weights can easily be chosen by trial and error without much effect on the result of the estimation. The estimates of the wall locations were detected and stored for every image, and would be used in the training procedure carried out in the fine-scale image. G. Human Intervention

Fig. 10.

The training and validation procedure.

parameters is two and the searching space is the triangular area and and the line . enclosed by the axes 3) Validation: The DP procedure with the optimal weight set was applied to the validation set. The detected boundaries were again compared with the manually defined boundaries. The error in (13) was computed. If satisfied a predefined criterion, which was the square of 0.5 pixels in our case, then the weight set was accepted. If not, this implied that the cost terms designed were not adequate in describing the characteristics of the interface and needed to be modified. After that, the correlation coefficient between the measurements (LD, IMT) of the detected results and expert results was computed. The result of was required to exceed 0.98. If it did not, further steps to adjust the cost terms were needed. This training and validation procedure was repeated for each of the interfaces I3, I5, and I7 and for each type of the images, i.e., CCA, Bulb, and CFA. The experiment showed that, before the membership funcwas kept at above a high level, tions were used, the error which meant that the training was never successful. Visually, without the fuzzy expression, detected boundaries tended to snap to those pixels where the intensity and/or intensity gradient is the highest, which is not always the way the experts mark the boundaries. The use of a fuzzy expression and a and made the successful proper setting of the thresholds training possible. was In the training procedure, an error surface obtained (Fig. 11). The sensitivities of the error with respect were obtained by the partial derivatives to . For most of the boundaries in different types of artery images, in the parameter space at around the point with the best combination of weights [i.e., the lowest point of the surface in Fig. 11(b)], had similar sensitivities to the weights of the intensity and intensity gradient terms and was less sensitive to the weight of the continuity term. Note that the training procedure was applied only to cost functions used in the fine-scale image. For the cost functions

In the case of obvious detection errors, a human intervention mechanism is needed. We present a new approach whereby the human intervention functions under the framework of the cost minimization. In this method, the operator specifies a certain point in the th column of the searching area and forces the polyline to pass through it. This human intervention factor is introduced into the cost function in the form of an external force . It is a monotonously interm creasing function of the distance between the candidate node and the manually set point . This extra in the th column, term exerts a force pushing the node toward . The optimum solution of the cost function is resought. Due to the smoothness constraint enforced by the geometrical force term , a single setting of point has a global effect. Not only is node forced to approach , but the whole trajectory of the boundary is empirically set for adjusting the is affected. The weight leads to a stronger strength of the pushing force. A higher force and vice versa. The operator can set more than one modification point, but there is a maximum of one for each node. , then the cost Assuming the points are set at function in (1) is revised to

(14)

ensures that an extra term where the unit sample function is added only to the nodes which are in the columns that have operator-set modification points. Fig. 12 demonstrates how this mechanism works. In Fig. 12(a), the solid line is the detected boundary passing through the local cost surface (image-feature-related costs only). It is determined in such a way that it follows the surface valley and, at the same time, contains less curvature. In Local costs of all points Fig. 12(b), the operator sets a point , the value of in column 5 are added with a term varies its position in column 5. This means which varies as that a high wall with a U-shaped gap centered at is built on the cost surface. The shape of the gap pushes the boundary toward point . The described scheme for modification is essentially a user guided auto-detection. It is fundamentally different from the scheme based on pixel-by-pixel correction. It has the following characteristics.

136


(a)

Fig. 11.

K + 1 = 3) using (a) a coarse step and (b) a fine step.

(b)

Search for weight values (

(a)

(b)

(c)

(d)

2

Fig. 12. External force pushes boundary towards a specified location. (a) The local-cost surface of size 7 8, which is composed of image-feature-related costs only and the optimal boundary. (b) The same cost surface as in (a) but with a human-intervention-related cost added and the changed optimal boundary. (c) An erroneous detection at the near wall. (d) A single click at point changes a major part of the boundary to the desired location.

a

First, the high-level human correction is naturally introduced into the cost function which is originally dominated by low-level image features. The detected boundary is a mathematically optimal solution to the modified cost function (14). As such, a single setting of a modification point has a global impact on the detection, which may change a great part or the whole of the trajectory of the boundary [Fig. 12(c)–(d)]. Hence, very few modification points are needed, even for modifying a totally wrongly detected boundary. Second, for most parts of the boundary which horizontally deviate from the modification point, it is still the image features that determine the accurate location of the boundary. In these

parts, the operator intervention is used merely as a guide for the boundary detection. Third, at the section of the boundary around the user set point, due to the flattened shape around the bottom of the gap, the boundary can still be attracted by strong nearby image features instead of passing exactly though the user set point. The operator can therefore be fairly relaxed about setting the modification point and let the image features determine the accurate location of the boundary. These characteristics do not only significantly reduce the workload of the operator but, more importantly, they reduce the variance due to the subjectiveness of the human intervention


137

Fig. 13. Arteries and detected boundaries. (Row 1) CCA. (Row 2) Bulb. (Row 3) CFA. (Column 1) Original image. (Column 2) Image with detected boundary overlaid.

in the modification process, since fewer input points and less accuracy of these points are required, V. RESULTS A. Examples Fig. 13 shows three examples of boundaries which have been correctly detected. In the example of the CCA, a rough plaque on the far wall is correctly picked out from the noisy background. In the Bulb image, echoes in zone Z5 are weak and difficult to distinguish from speckles and echoes in zone Z7. Still, the detected boundary is in agreement with an expert interpretation. In the CFA image, wide gaps in the echo zone Z3 are filled at the near wall and a continuous and reasonable estimation of I3 is obtained. B. Clinical Evaluation In order to assess the performance and usefulness of the trained and validated system in a real application, a thorough evaluation of the method was carried out at the Wallenberg

Laboratory for Cardiovascular Research, Sahlgrenska University Hospital, Göteborg University, Sweden. A full description of the evaluation study is given elsewhere [8], [9]. In the following, we briefly describe the experiment and present the main results. 1) Method: New images were registered from a consecutive using the same group of patients and control subjects ultrasound scanner as the one used in the training. From each subject, nine images, three for each of the CCA, the Bulb, and the CFA, were recorded. Two readers, both with previous experience, analyzed the images using both the manual and the automated systems. Reader 2 analyzed only CCA images. A third reader, with no previous experience, also analyzed the images, but using only the automated system. In using the automated system the operator had the option of modifying the detected boundaries. Measurements for a subject were derived by averaging measures from the three recorded images. The presence of a strong echo from the wall of the adjacent vein and other disturbing echoes from tissues below the far wall of the artery sometimes interfered with the correctness of the

138


TABLE I COMPARISON BETWEEN THE MANUAL AND AUTOMATED ANALYZING SYSTEMS

wall estimation. When this happened, the reader had the option of drawing a box which enclosed the near and far walls of the artery, but excluded the vein wall and other interference from outside of the vessel. Within this box, the wall locations were automatically searched and the boundaries were automatically detected. 2) Statistics: We report the main results for the first two readers in this section. More results of this clinical evaluation study including all three readers have been given in [8] and [9]. for differences beMeans and standard deviations tween the readers and between the methods were calculated. were then calculated Interobserver and intermethod error . The coefficient of according to the formula describes the difference as a percentage of variation and was calculated according to the pooled mean value . For the significance of the the formula difference between the two analyzing methods and between two readers, Wilcoxon’s signed-rank test for matched pairs was used. The association between the manual and automated measures was characterized using Pearson’s correlation coefficient .

Table I presents a comparison of the measures from the manual and the automated system obtained by Reader 1, the most experienced among the three readers. The correlation coefficient for the two analyzing methods was at least 0.98 for all of the measurements. Table II describes the reading variability for the two experienced readers, Readers 1 and 2, when measuring the CCA. With the manual system the differences between the means of the measurements from two readers were significant, while with the automated system the differences is notably improved. were much smaller and the Fig. 14 plots the mean of IMT data by Reader 1 versus Reader 2. With the manual system, Fig. 14(a) shows that the Reader 1 tended to obtain smaller measurements than Reader 2. However, when using the automated method, the results from the two operators were closer, and the plotted points in Fig. 14(b) are close to the ideal regression line and evenly distributed on both sides of it. The percentage of images for which the user drawn box was needed to exclude interference from objects outside of the artery was 11%, 6%, and 18% for the CCA, Bulb, and CFA, respectively. This indicates that the algorithm for automated estima-


TABLE II READING VARIABILITY WHEN MEASUREMENTS WERE PERFORMED BY TWO READERS MANUAL AND THE AUTOMATED ANALYZING SYSTEM

(a) Fig. 14.

139

WITH THE

(b)

Variability between two experienced readers in the manual (a) and the automated (b) analyzing system.

tion of the near wall and far wall was successful in most images and especially for the CCA and the Bulb. The percentage for the Bulb was lower than that for the CCA. This was because in the Bulb only the far wall was considered.

Manual corrections of the automatic outlining of the intima–lumen or the media–adventitia interface of the far wall were performed in 17% of all the CCA images. Most corrections were minor in nature (12% of all the images). Manual

140


corrections of the interface in the far wall of the Bulb were performed in 58% (minor changes in 21%) of all images, and in 67% (minor changes in 13%) of all images of the CFA. The average time used for measuring a common artery image using the automated system was 0.7 min. It was about 3.5 min with the manual system. In a real application a complete analysis also includes the choice of frozen images from the videotape and digitization and also the evaluation of the real-time images on the tape. For such complete reading, the ratio of analysis time for the automated versus the manual system was 1:3. VI. DISCUSSION A. Reducing the Variability The key to reducing interobserver variability is a reduction in the amount of human intervention. In the present method this reduction is accomplished by adopting three approaches. First, the vessel wall position is automatically detected in the coarsescale image and, hence, the manual initial tracing is avoided. Second, by applying DP in the fine scale image, a single smooth boundary instead of separated boundary segments is derived. This enables the detection algorithm to resist most of the interference from speckles and dropouts in the wall region. As a result, fewer manual modifications are needed after the detection. Third, when modification is inevitable, it is important to minimize the number of modifications and make the results less dependent on the exact location of the modification points. In this algorithm, one or a few points can move a severely displaced boundary into its correct position. This not only makes the operation easier but, more importantly, it reduces operator variability. As the interobserver variability is reduced, it is expected that the new system will also reduce intraobserver variability and prevent the problem of drift in measurements over time in a longitudinal study. B. Quality Assurance Some investigators [12], [15] proposed the application of a quality assurance procedure to ensure that the boundary detection algorithm only chooses those pixel pairs along the boundaries which present measurement values within a certain expected range. In our method, such computerized quality assurance is not needed. First, because the boundary smoothness constraint takes care of the majority of the outliers. Second, because in practical use there will always be a human operator selecting the image to be measured and then accepting, rejecting, or modifying the result of the automated detection. This serves as the ultimate quality control. We find this approach to quality assurance acceptable because our previous studies [8], [9] as well as other studies using this technique [29], [30] have shown that this type of human interaction does not have a significant influence on measurement variability. However, it is highly recommended that laboratories entering the field of automated IMT measurement should carry out inter- and intraoperator variability studies before starting clinical studies. This is an important quality assurance at another level which concerns the automated boundary detection algorithm only to a lesser extent. Rather, it concerns the overall measurement protocol, including the ultrasound examination, selection of images to be measured, and the mea-

Fig. 15. Schematic illustration of the difference in boundary location in the automated and the manual system.

surement region if this is not automatically selected or otherwise consistently defined. C. The Use of Human Knowledge In the present method, human a priori knowledge is incorporated into the algorithm and human perception ability is imitated to some extent. The preknowledge of the echo pattern and the geometrical shape of the interface is embodied in the cost function. The employment of the membership function offers another tuning point for the algorithm to better imitate the fuzzy interpretation of a human observer. Through the training process, the system learns from the expert data. It has been proven that this approach to artery boundary detection improves the robustness and reliability of the automated detection. D. The Differences In some cases, the automated algorithm locates the boundary differently from the manual tracing. From Table I it can be observed that with the automated detection system, the IMT measure is higher than that obtained with the manual system. A possible explanation for this difference is outlined in Fig. 15. At the media–adventitia interface the automated detection matches the manual detection well. For the lumen–intima interface, however, due to the weak echo the visibility threshold can be well above the point of the maximal gradient. In this case, the operator tends to set the interface point closer to the top of the echo. This gives a thinner IMT than that obtained in the automated measurements. However, this difference is clinically acceptable as long as the system performs consistently. E. Optimality With DP, a path corresponding to a minimal cost of the cost function, hence, a mathematically optimal solution can always be derived. However, the quality of the path as an artery boundary relies on the quality of the designing of the cost function. To improve the designing of the cost function, we use the training procedure based on human expert data. When the trained system is applied to new images, the optimality of the detected boundary from a clinical point of view is monitored


Fig. 16.

141

Multiple optimal paths. In the strictly homogeneous region, path 1 and path 2 have exactly the same cost. Therefore both are optimal.

by the reader of the system. In poor quality images, when a mathematically optimal solution does not give a clinically satisfactory result, human intervention can be introduced by inserting the external force terms into the cost function to derive an optimal solution in both the mathematical and the clinical senses. From the way the cost function is designed, there is no guarantee that the minimum is unique. The costs are computed in floating-point precision. Two paths passing through regions with even very slightly different intensity patterns will result in different costs. However, two paths share same cost does happen when the line passes through a strictly homogeneous area where there is no active image information, as in the case shown in Fig. 16. The line following path 1 will give exactly the same cost as that going through path 2. The current algorithm uses the first reached optimal path as the optimal path, which should more accurately be called an optimal path. In the ultrasonic artery images, a strictly homogeneous region is usually very small and will therefore usually not significantly affect the measurement. In cases where the image exhibits large echo dropouts, the strictly homogeneous region can be large and the algorithm may pick an inappropriate path among the paths with same costs. Then high-level human knowledge needs to be incorporated through the human intervention mechanism in this method. F. Applicability Experiments have shown that when applying the present system to images of the Bulb and the CFA, manual correction after detection was necessary for a large percentage of the images. The main reason was the poorer quality of these images. The frequent occurrence of plaque was another reason for incorrect detection. In some of the Bulb images large plaque, probably due to the constitution of fibrous tissue and lipid, appeared as an over brightened echo area with varying shape. Beneath it there always existed a wide dark shadow area. There is hardly any clue to suggest the position of the media–adventitia interface. In cases such as these an experienced human operator, by consulting the playback video of the moving image sequence, can make a valid decision none the less. This is, however, beyond the capability of the current detection method. A possible solution could be an automated recognition of the plaque based on an investigation of the statistical properties of the plaque echo.

All the images used for training and evaluation of the system were recorded by an Acuson 128 scanner. In principle, we believe that retraining of the system is needed when the ultrasonic equipment is changed. However, an evaluation of the same system without retraining was carried out at the Department of Epidemiology and Biostatistics, Erasmus University Medical School, The Netherlands, where the ultrasonic scanner used was an ATL3000. A variability study including images from 99 subjects, generally showed results consistent with those in our study [29]. G. Conclusion In conclusion, we have proposed a method based on multiscale dynamic programming to automatically measure ultrasonic artery images. The human knowledge of the artery image is incorporated in the system, which makes the system capable in processing images of different quality. Human factors in the determination of the boundaries are reduced. Evaluation of the system shows reduced interobserver variability as well as overall analysis time. The automated artery boundary detection system can replace the old manual system in a clinical application environment. ACKNOWLEDGMENT The authors wish to thank Dr. M. Bots and Dr. A. Iglesias del Sol at the Department of Epidemiology and Biostatistics, Erasmus University Medical School, The Netherlands for providing the evaluation data of the automated detection method. The authors acknowledge the excellent technical assistance of C. Schmidt and A. Fröden. REFERENCES [1] J. T. Salonen and R. Salonen, “Ultrasound B-mode imaging in observational studies of atherosclerotic progression,” Circulation, vol. 87, no. suppl. II, pp. 1156–1165, 1993. [2] J. Wikstrand and I. Wendelhag, “Quantitative ultrasonography of carotid and femoral arteries,” in Vascular Diagnostics, P. Lanzer and J. Rösch, Eds. Berlin, Germany: Springer-Verlag, 1994, pp. 129–138. [3] I. Wendelhag, T. Gustavsson, M. Suurkula, G. Berglund, and J. Wikstrand, “Ultrasound measurement of wall thickness in the carotid artery. Fundamental principles and description of a computerized analysing system,” Clin. Physiol., vol. 6, no. 11, pp. 565–577, 1991. [4] P. Pignoli, E. Tremoli, A. Poli, and R. Paoletti, “Intimal plus medial thickness of the arterial wall: A direct measurement with ultrasound imaging,” Circulation, vol. 74, pp. 1399–1406, 1986. [5] M. Wong, J. Edelstein, J. Wollman, and M. G. Bond, “Ultrasonic-pathological comparison of the human arterial wall, verification of intimamedia thickness,” Arterioscler. Thromb., vol. 13, no. 4, pp. 482–486, 1993.

142

[6] J. Persson, J. Formgren, B. Israelsson, and G. Berglund, “Ultrasound-determined intima-media thickness and atherosclerosis, direct and indirect validation,” Arterioscler. Thromb., vol. 14, no. 2, pp. 261–264, 1994. [7] I. Wendelhag, O. Wiklund, and J. Wikstrand, “On quantifying plaque size and intima-media thickness in carotid and femoral arteries. Comments on results from a prospective ultrasound study in patients with familial hypercholesterolemia,” Arterioscler. Thromb. Vasc. Biol., vol. 16, no. 7, pp. 843–850, 1996. [8] I. Wendelhag, Q. Liang, T. Gustavsson, and J. Wikstrand, “A new automated computerized analyzing system simplifies reading and reduces the variability in ultrasound measurement,” Stroke, vol. 28, no. 11, pp. 2195–2200, 1997. [9] C. Schmidt and I. Wendelhag, “How can the variability in ultrasound measurement of intima-media thickness be reduced? Studies of interobserver variability in carotid and femoral arteries,” Clinical Physiol., vol. 19, pp. 45–55, 1999. [10] P. J. Touboul, P. Prati, PO.-Y. Scarabin, V. Adrai, E. Thibout, and P. Ducimetiere, “Use of monitoring software to improve the measurement of carotid wall thickness by B-mode imaging,” J. Hypertension, vol. 10, no. suppl. 5, pp. S37–S41, 1992. [11] J. Gariepy, M. Massonneau, J. Levenson, D. Heudes, and A. Simon and the Groupe de Prevention Cardio-vasculaire en Medecine du Travail, “Evidence for in vivo carotid and femoral wall thickness in human hypertension,” Hypertension, vol. 22, no. 1, pp. 111–118, 1993. [12] R. H. Selzer, H. N. Hodis, H. Kwong-Fu, W. J. Mack, P. L. Lee, C. R. Liu, and C. H. Liu, “Evaluation of computerized edge tracking for quantifying intima-media thickness of the common carotid artery from B-mode ultrasound images,” Atherosclerosis , vol. 111, pp. 1–11, 1994. [13] T. Gustavsson, Q. Liang, I. Wendelhag, and J. Wikstrand, “A dynamic programming procedure for automated ultrasonic measurement of the carotid artery,” in Proc. IEEE Computers Cardiology 1994, pp. 297–300. [14] R. J. Kozick, “Detectinh interfaces on ultrasound images of the carotid artery by dynamic programming,” SPIE, vol. 2666, pp. 233–241, 1996. [15] W. Liang, R. Briwning, R. Lauer, and M. Sonka, “Automated analysis of brachial ultrasound time series,” in Proc. SPIE Conf. Physiol. Function Multidimensional Images, vol. SPIE 3337, San Diego, CA, Feb. 1998, pp. 108–118. [16] T. Gustavsson, R. Abu-Gharbieh, G. Hamarneh, and Q. Liang, “Implementation and comparison of four different boundary detection algorithms for quantitative ultrasonic measurements of the human carotid artery,” in Proc. IEEE Computers n Cardiology 1997, Lund, Sweden, pp. 69–72.


[17] R. E. Bellman and S. Dreyfus, Appled Dynamic Programming. Princeton, NJ: Princeton University Press, 1962. [18] A. A. Amini, T. E. Weymouth, and R. C. Jain, “Using dynamic programming for solving variational problems in vision,” IEEE Trans. Pattern Anal. Machine Intell., vol. 12, pp. 855–867, Sept. 1990. [19] A. K. Jain, Fundamentals of Digital Image Processing. Englewood Cliffs, NJ: Prentice-Hall, 1989. [20] D. J. Sakrison, “On the role of observer and a distortion measure in image transmission,” IEEE Trans. Commun., vol. COM-25, pp. 1251–1567, Nov. 1997. [21] H. R. Wilson and F. Wilkinson, “Evolving concepts of spatial channels in vision: From independence to nonlinear interactions,” Perception, vol. 26, pp. 939–960, 1997. [22] I. A. Rybak, V. I. Gusakova, A. V. Golovan, L. N. Podladchikova, and N. A. Shevtsova, “A model of attention-guided visual perception and recognition,” Vision Research—Special Issue on Model of Recognition, 1998. [23] L. A. Zadeh, Fuzzy sets and their applications to cognitive and decision processes. New York, NY: Academic, 1975. [24] A. Kaufmann, Introduction to the Theory of Fuzzy Subsets. New York, NY: Academic, 1975, vol. I. [25] N. R. Pal and S. K. Pal, “A review on image segmentation techniques,” Pattern Recognit., vol. 26, pp. 1277–1294, 1993. [26] A. P. Witkin, “Scale-space filtering,” in Proc. Int. Joint Conf. Artif. Intell. Karlsruhe, Germany, 1983, pp. 1019–1021. [27] J. Babaud, A. P. Witkin, M. Baudin, and R. O. Duda, “Uniqueness of the gaussian kernel for scale-space filtering,” IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-8, pp. 26–33, Jan. 1986. [28] A. L. Yuille and T. A. Poggio, “Scaling theorems for zero crossing,” IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-8, pp. 15–25, Jan. 1986. [29] M. Bots and A. Iglesias Del Sol, “Private Communication,” unpublished. [30] E. Stensland-Bugge, K. H. Bonaa, and O. Joakimsen, “Reproducibility of ultrasonographically determined intima-media thickness is dependent on arterial wall thickness,” Stroke, vol. 28, no. 10, pp. 1972–1980, 1997.