Point based methods for automatic bronchial tree matching and labelling Thomas B¨ ulowa , Cristian Lorenza , Rafael Wiemkera , and Janett Honkob a Philips
Research Laboratories, Roentgenstrasse 24-26, 22335 Hamburg, Germany Semiconductors, Stresemannallee 101, 22529 Hamburg, Germany
b Philips
ABSTRACT When addressing the task of automatic tracheo-bronchial tree matching it seems natural to make use of the tree’s graph structure and branching characteristics. In tracheo-bronchial trees that were automatically extracted from multi-slice CT data, however, the graph information is not always reliable, especially for noisy or low-dose data which makes the abovementioned class of approaches prone to error in these situations. In this work we investigate what can be gained by using the spatial position of the bronchial centerline points. For this purpose we introduce, investigate, and compare two approaches to tree matching that are based on the use of centerline point positions alone with no additional connectivity information. As features we use (1) the 3D shape context and (2) statistical moments of the local point distribution. The 3D shape context has recently been introduced as a regional shape descriptor. It is based on a spherical histogram with logarithmic sampling in the radial direction. Both methods are used in order to match an automatically extracted tree to a manually labeled model tree which results in an automatic anatomical labeling of the data tree. Six tracheo-bronchial trees were matched to a given model tree. The data trees covered a range from high quality data to poor quality data. Furthermore two of the cases exhibited strongly distorted anatomy. It could be shown that the 3D shape context feature labeled 69% of the branches correctly with one of 34 anatomical labels. In the case of the statistical moment feature 40% of the branches were labeled correctly. We conclude that the set of centerline points alone allows correct labeling of a large portion of lung segments. We propose to combine the valuable local shape information in future work with connectivity and branching information, where the latter is reliably available. Keywords: Tree matching, tracheo-bronchial tree, pulmonary vessel tree, 3D shape context, anatomical tree labeling, multi-slice CT
1. INTRODUCTION Being able to match corresponding anatomical tree structures (either intra-patient or inter-patient) is beneficial for a variety of applications. In this paper we focus on the tracheo-bronchial tree extracted from multi-slice CT data. As application we consider matching tracheo-bronchial trees interpatiently. Matching a tracheo-bronchial tree to an anatomically labeled model tree allows us to automatically assign anatomical labels to the data tree. Previous approaches to bronchial tree matching1 and anatomical labeling1, 2 make explicit use of the graphstructure of the tree, containing branching points as vertices and branches as edges. However, there exist situations where the automatically extracted graph structure does not reliably correspond to the anatomical tree structure, while the positions of bronchi or vessel centerline points can still be extracted with reasonable accuracy.3 This can be the case if the tracheo-bronchial tree is extracted from noisy, especially low-dose CT data possibly leading to false topology or disconnected subtrees. If the quality of the data does not allow for the extraction of a connected tree, centerline points can still be extracted by thresholding of a bronchi-enhancing filter. In the extraction of the pulmonary vessel tree from multi-slice CT data, arteries and veins are in many cases not separable on basis of their Hounsfield values. This leads to the risk of artery/vein crossings being mislabeled as branching points in the tree.9 Considering the above it seems worthwhile to investigate how well anatomical tree structures can be matched purely on the basis of the set of the centerline points of the tree without taking into account the apparent graph Corresponding author: Th.B., E-mail:
[email protected]
Seed points Initialize one segment per seed point
seed point
Add segments to the segment queue Prune tree
centerline point
Queue empty ?
Yes
No Retrieve head segment from the queue
branching point
segment
connected front component
Reset segment and adapt expansion parameters
Expand and continuously evaluate segment Evaluation result:
Prune tree Done
Continue expansion Adapt parameters Reject segment Accept segment
Initialize one new segment per connected front component
Figure 1. Sketch and flow chart of the tree segmentation algorithm.
structure. This is the purpose of this paper. We compare two different shape features – the 3D shape context and statistical moments – using the set of centerline points and propose a matching scheme based on these shape features. Recently, the 3D shape context was applied to matching lung surfaces in order to assess breathing motion.4
2. TREE EXTRACTION For simultaneous segmentation and centerline extraction of tree structures we apply a front-propagation-based method.5 We briefly outline this framework, that has previously been applied to the segmentation of the tracheobronchial tree,6, 7 the coronary arteries,8 and the pulmonary vessel tree.9 The general procedure consists of three levels — the voxel level, the segment level, and the tree level. Voxel level. Starting from an initial seed point a fast marching front propagation process is started. In order to maintain mostly flat front portions, the front propagation velocity is set constant v ≡ 1. A voxel acceptance criterion – in the most simple case, a HU threshold criterion – decides whether a voxel is included or not. While propagating through the tree, the front is regularly checked for connectedness in regular intervals. If the front disintegrates into two or more disconnected components this is interpreted as a bifurcation in the tree. In this case the segment currently grown is finalized and one new segment per connected front component is initialized. Segment level. During segment expansion, it is checked by a termination criterion whether expansion of the current segment should be continued or terminated. Typical termination causes are disintegration of the front or the reaching of a maximally allowed segment length. A segment evaluation criterion regularly evaluates and has the possibility to reject a segment. For an accepted segment a new segment is initialized for each connected front component. Rejected segments on the other hand do not initialize children. Segment evaluation criteria can, e.g., be of geometric nature (max segment radius, elongated shape, ...), or gray-value based (vesselness, orientedness, ...). Segments that turn out unacceptable can be regrown using an automatically adapted set of parameters for the various criteria. These parameters are also adapted whenever a new segment is inherited from a parent segment. Tree level. At this level decisions that cannot be made looking at a single segment are made. E.g., a subtree can be pruned if the density of branching points gets too high, indicating leakage. Along with the segmentation the tree structure including the branching points and the vessel centerlines are built up. While the front propagates centerline points are estimated as centers of gravity of the propagating wave front. This centerline information will be used exclusively in the matching application. The described process is sketched in Fig. 1.
• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
Segment 1 Segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Segment 8 Segment 9 Segment 10 Segment 11 Segment 12 Segment 13 Segment 14 Segment 15 Segment 16 Segment 17 Segment 18 Bronchus 1 Bronchus 2 Bronchus 3 Bronchus 4 Bronchus 5 Bronchus 6 Bronchus 7 Bronchus 8 Bronchus 9 Bronchus 10 Bronchus 11 Bronchus 12 Bronchus 13 Bronchus 14 Bronchus 15 Bronchus 16
Left upper lobe apicoposterial segment Left upper lobe anterior segment Lingula Left lower lobe superior segment Left lower lobe medial basal segment Left lower lobe anterior basal segment Left lower lobe lateral basal segment Left lower lobe posterior basal segment Right upper lobe apical segment Right upper lobe posterior segment Right upper lobe anterior segment Right middle lobe lateral segment Right middle lobe medial segment Right lower lobe superior segment Right lower lobe medial basal segment Right lower lobe anterior basal segment Right lower lobe lateral basal segment Right lower lobe posterior basal segment Trachea Left main bronchus Right main bronchus Bronchus intermedius Left upper lobe bronchus Left lower lobe bronchus Right upper lobe bronchus Right middle lobe bronchus Right lower lobe bronchus Bronchus leading to segment cluster (5), 6-8 Bronchus leading to segment cluster 1+2 Bronchus leading to segment cluster 7+8 Bronchus leading to segment cluster 9+10 Bronchus leading to segment cluster 15-18 Bronchus leading to segment cluster 15-17 Bronchus leading to segment cluster 16+17
Table 1. The 34 anatomical labels that were used in the validation of the bronchial tree matching application. Figure 3 shows a tracheo-bronchial tree extracted from multi-slice CT data, which has been manually labeled according to the above scheme. The coloring scheme used for the visualization of the labeling result is indicated by the bullets in the left hand column of the table (electronic paper version only).
2.1. Anatomical labeling of the Tracheo-Bronchial Tree Each branch of the tracheo-bronchial trees extracted from CT datasets is labeled as one of 34 anatomical structures. For a list of the used structures see Table 1.
3. SHAPE FEATURES In order to detect correspondences between two given trees we start by extracting features representing the shape at each given centerline point of a tree. The features need to be designed in such a way that on the one hand they capture the local shape as seen from a given centerline point while on the other hand they are robust to distortions.
Two different local shape features were investigated: • The 3D shape context. • Statistical moments of the point distribution.
3.1. The Shape Context The 2D shape context as a regional descriptor of shape was introduced by Belongie et al. 10 Assume a 2D shape is described by the set of its contour points. A log-polar histogram is positioned at a reference point on the objects contour (see Fig. 2 left) and the number of remaining contour points is counted per histogram bin which gives a feature vector (see Fig. 2 middle) representing the shape at the reference point. The 3D shape context is a straightforward extension of the 2D shape context. 11 Assume we are given a set of points P = {p1 , . . . , pn }, pi ∈ R3 , e.g. sampled from the surface of a 3D object O or being centerline points of a tree structure. The local shape as seen from a point q ∈ P is captured in the 3D shape context at q. The shape context is a 3D spherical histogram (see Fig. 2 right) in which the displacement vectors from the reference point q to the remaining points pi ∈ P are counted. The displacement vector di = pi − q can be expressed in spherical coordinates as cos(ϕi ) sin(ϑi ) di = ri sin(ϕi ) sin(ϑi ) cos(ϑi )
(1)
The membership of a displacement vector di to a histogram bin is governed by di ∈ bin(j, k, l)
⇔
ϕi ∈ [Φj , Φj+1 ), ϑi ∈ [Θk , Θk+1 ), ri ∈ [Rl , Rl+1 )
(2)
where the angles ϕ and ϑ are quantized linearly Φj
=
Θk
=
2πj , J 2πk , K
j ∈ {0, . . . , J}
(3)
k ∈ {0, . . . , K}
(4)
and the radius r is quantized logarithmically
l Rl = exp ln(rmin ) + ln L
rmax rmin
.
(5)
The radii Rl are sampled logarithmically in order to allow for larger distortions with increasing distance from the center. The 3D shape context at q is defined as X SCq (j, k, l) = w(di ), (6) di ∈bin(j,k,l)
p where w(di ) is the weight with which di contributes to the sum. We tested wi = 1/ a V (j, k, l), where V (j, k, l) is the volume of bin(j, k, l). Among a ∈ {1, 2, 3, 4} the value a = 3 worked best and was used in all our experiments. In the following SCq will denote the J × K × L-dimensional vector constructed by concatenating the elements SCq (j, k, l).
In other applications a local coordinate system is used which aligns the shape context’s z-axis with the normal of the object surface. In our application the patient coordinate system is used as reference frame. In all our experiments the number of bins in the shape contexts are set to (J, K, L) = (9, 9, 8) and minimal and maximal radius of the shape context were rmin = 3mm and rmax = 250mm.
Figure 2. The 2D shape context histogram (left and middle), the 3D shape context (right). (Figures reproduced from 10, 11 )
3.2. Statistical Moments Statistical Moments of a point distribution provide some information about the distributions shape. Depending on the order of the calculated moments this information is either coarse or detailed. In general 3D moments of order n = j + k + l, (n ∈ N) of a 3D density function f (x, y, z) are defined by12 ZZZ mjkl = xj y k z l f (x, y, z) dx dy dz (7) R3
Replacing the continuous distribution f by a discrete set of points this reads13 mpqr =
N X
xjk ykk zkl
(8)
i=1
In the current application the reference point q is used as coordinate origin such that the coordinates x, y and z in eq. 8 are given by pi − q = di = (xi , yi , zi ) (9) To calculate the feature vector at a reference point based on moments, the 3D tree structure and the relative positions of all remaining points must be given. Then all moments of order up to some value n = n max are calculated and the feature vector is constructed by concatenating all acquired statistical moments. All moments in a feature vector are adjusted depending on the order n = j + k + l by µjkl =
√
j+k+l
mjkl
(10)
in order to involve all moments equally in the shape distance calculation. The moment of order zero – the number of points in the tree – was not used as a feature. This leads to an s-dimensional feature vector f = (µ100 , µ010 , µ001 , µ200 , . . . , µ00nmax ),
(11)
where s is given by s=
(nmax + 3)! −1 3! nmax !
(12)
3.3. Matching Cost Given feature vectors representing the shape at points pi ∈ P, qj ∈ Q we need a cost function in order to quantify the cost of matching pi to qj . For both shape features, the shape context and the statistical moments feature,
we tested three different cost functions: The L1 -distance, the L2 -distance, and the χ2 -distance. cij = C L1 (pi , qj ) =
D X d=1
cij = C
L2
(pi , qj ) =
|f d (pi ) − f d (qj )|
D X d=1
d
d
(f (pi ) − f (qj ))
(13)
2
!1/2
(14)
D
cij = C
χ2
(pi , qj ) =
1 X (f d (pi ) − f d (qj ))2 2 f d (pi ) + f d (qj )
(15)
d=1
where f (p) = (f 1 , . . . , f D )(p) is a D-dimensional shape feature vector (shape context or moments). While the χ2 -distance yields the best results for the shape context in our experiments the L1 -distance was optimal for our statistical moment feature experiments. These choices were used in all following tests.
4. TREE MATCHING Assuming we are given two trees T target and T data the task is to relate each point of T data to some point of T target . We do not require a one-to-one relation between the two trees. There might be portions missing in the data tree so that some points of T model are not matched by any of the points of T data . Also, we do allow that more than one points of T data choose the some target point as their equivalent.
4.1. Point to Point Matching Given a target and a data tree T target T data
= {pitarget | pitarget ∈ R3 , i ∈ {1, . . . N target }} = {pdata | pdata ∈ R3 , i ∈ {1, . . . N data }} i i
(16) (17)
any data point pdata is matched to the target point pjtarget giving the minimal matching cost to pdata among all i i points in the target tree. , pltarget )). (18) j = arg mintarget (C(pdata i l∈{1,...,N
}
Here C can be any of the matching cost functions defined in Sect. 3.3.
4.2. Branch to Branch Matching Although we deal with situations where we cannot trust that the extracted algorithmic tree structure corresponds to the actual anatomical tree structure and thus investigate point based matching algorithms, it still seems to be reasonable to assume that a branch of the extracted tree corresponds to a branch in the anatomic tree. A branch is defined as the part of a tree that starts at a bifurcation and ends at the next bifurcation. Thus, each branch bi consists of a number of centerline points bi ⊂ T , does not overlap with any other branch bi ∩ bj = ∅ for i 6= j, and the tree is completely made up of the branches ∪bi = T .
As in point to point matching any data point pdata ∈ bdata can be matched to a target point pktarget ∈ btarget . i j l target data data Thus, pi votes for the branch bl . If the branch bj is to be matched to a target branch each point of the data branch votes for one of the branches in the target tree. The data branch is simply matched to the target branch receiving the majority of votes. If the maximal number of votes is received by two or more branches the branch with the minimal sum of point matching costs is chosen.
5. RESULTS In this work we used seven thoracic scans acquired by a Philips Brilliance 40-slice CT scanner. In-plane resolution was 0.7 mm, the scans were reconstructed at a slice thickness of 0.9 mm and 0.45 mm slice spacing. The tracheobronchial trees were extracted from the data by the method described in Sect. 2 and manually labeled according to the scheme shown in Table 1. Each branch was assigned one of the 34 listed labels. One of the trees – the one with the most complete segmentation – was used as a model. The remaining six tracheo-bronchial centerline trees were matched to the model tree using the proposed shape features. We used feature vectors in an (9 · 9 · 8) = 648-dimensional feature space for the 3D shape context. As minimal and maximal radius we used rmin = 3mm and rmax = 250mm. The χ2 -distance was used as matching cost function. For the six trees shown in Fig. 4 the percentage of correctly labeled segment was 71%, 82%, 80%, 74%, 47%, and 61%. Thus, on average 69% of the branches were labeled correctly. Note that this includes trees exhibiting strongly distorted anatomy. For the statistical moment feature we used a 64-dimensional feature space corresponding to moments up to order 6. Higher moments did not improve results. Here, the L1 -distance was used as matching cost function. Using the statistical moment feature 40% of the branches were correctly labeled with the results for single trees ranging from 12% to 85%.
6. CONCLUSION Of the two shape features that were compared, the 3D shape context clearly yields superior results to the statistical moment based shape feature for the application of point set based tracheo-bronchial tree matching. It could be shown that based purely on shape features computed from the point set of bronchi centerline points without using the tree structures more than 2/3 of the bronchial tree can be labeled correctly according to the nomenclature containing 34 different structures. This shows that much of the relevant information is in the point positions itself. This approach might be equally useful in the matching of pulmonary vessel trees at different inhalation states since in this application the correct anatomical tree topology is harder to extract than for the tracheo-bronchial tree. Furthermore the presented approach is applicable in cases where only positional information is available, e.g., in the form of a thresholded vessel- or bronchi-enhancing filter. Future work will address pulmonary vessel tree matching. Another focus will be on the integration of the point set information with the branching structure of the tree where the latter is available and reliable in order to boost the overall performance of automatic tree matching and labeling.
ACKNOWLEDGMENTS We thank our colleagues at Philips Medical Systmes, Healthcare IT, Advanced Development, Best, The Netherlands and Philips Medical Systmes, Clinical Science CT, Cleveland OH, USA for providing data and support.
REFERENCES 1. J. Tschirren, G. McLennan, K. Pal´ agyi, E. Hoffman, and M. Sonka, “Matching and anatomical labeling of human airway tree,” IEEE Trans. Medical Imaging 24, pp. 1540–1547, 2005. 2. K. Mori, S. Ema, T. Kitasaka, Y. Mekada, I. Ide, H. Murase, Y. Suenaga, H. Takabatake, M. Mori, and H. Natori, “Automated nomenclature of bronchial branches extracted from CT images and its application to biopsy path planning in virtual bronchoscopy,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2005, J. Duncan and G. Gerig, eds., LNCS 3750, pp. 854–861, Springer, 2005. 3. R. Wiemker, A. Ekin, R. Opfer, T. B¨ ulow, and P. Rogalla, “Unsupervised extraction and quantification of the bronchial tree on ultra-low-dose vs. standard dose CT,” in Medical Imaging: Physiology, Function, and Structure from Medical Images, A. Manduca and A. A. Amini, eds., Proc. SPIE 6143, 2006. 4. M. Urschler and H. Bischof, “Assessing breathing motion by shape matching of lung and diaphragm surfaces,” in Medical Imaging: Physiology, Function, and Structure from Medical Images, A. A. Amini and A. Manduca, eds., Proc. SPIE 5746, pp. 440–452, 2005.
Figure 3. The manually annotated tracheo-bronchial tree used as model. The used coloring scheme is listed in Table 1 (electronic paper version only).
Figure 4. The data trees that were matched to the model tree shown in Fig. 3. Of the assigned anatomic labels 71%, 82%, 80%, 74%, 47%, and 61%, were correct.
5. T. B¨ ulow, C. Lorenz, and S. Renisch, “A general framework for tree segmentation and reconstruction from medical volume data,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2004, C. Barillot, D. Heynor, and P. Hellier, eds., LNCS 3216, pp. 533–540, Springer, 2004. 6. T. Schlath¨ olter, C. Lorenz, I. Carlsen, S. Renisch, and T. Deschamps, “Simultaneous segmentation and tree reconstruction of the airways for virtual bronchoscopy,” in Medical Imaging: Image Processing, Proc. SPIE 4684, pp. 103–113, 2002. 7. R. Wiemker, T. Blaffert, T. B¨ ulow, S. Renisch, and C. Lorenz, “Automated assessment of bronchial lumen, wall thickness and bronchoarterial diameter ratio of the tracheobronchial tree using high-resolution CT,” in Proc. Computer Assisted Radiology and Sugery, CARS, pp. 967–972, 2004. 8. C. Lorenz, S. Renisch, T. Schlath¨ olter, and T. B¨ ulow, “Simultaneous segmentation and tree reconstruction of the coronary artery tree in MSCT images,” in Medical Imaging: Physiology and Function: Methods, Systems, and Applications, Proc. SPIE 5031, pp. 167–177, 2003. 9. T. B¨ ulow, R. Wiemker, T. Blaffert, C. Lorenz, and S. Renisch, “Automatic extraction of the pulmonary artery tree from multi-slice CT data,” in Medical Imaging: Physiology, Function, and Structure from Medical Images, A. A. Amini and A. Manduca, eds., Proc. SPIE 5746, pp. 730–740, 2005. 10. S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object recognition using shape contexts,” IEEE Trans. Pattern Analysis and Machine Intelligence 24(4), pp. 509–522, 2002. 11. A. Frome, D. Huber, R. Kollure, T. B¨ ulow, and J. Malik, “Recognizing objects in range data using regional point descriptors,” in Proc. ECCV, pp. 533–540, 2004. 12. J. Mangin, F. Poupon, D. Rivi`ere, A. Cachia, D. Collins, A. Evans, and J. R´egis, “3D moment invariant based morphometry,” in Medical Image Computing and Computer-Assisted Intervention, R. Ellis and T. Peters, eds., LNCS 2879, pp. 505 – 512, Springer, 2003. 13. B. Buston, D. Keim, D. Saupe, T. Schreck, and D. Vrani´c, “An experimental comparison of feature-based 3D retrieval methods,” in 2nd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT’04), pp. 215 – 222, 2004.
