space. The most well-known is the narrow band approach [5,6]. This method has ... level set method works particularly well with the marching cubes algorithm,.
A New Marching Cubes Algorithm for Interactive Level Set with Application to MR Image Segmentation David Feltell and Li Bai {dzf,bai}@cs.nott.ac.uk School of Computer Science, University of Nottingham, UK
Abstract In this paper we extend the classical marching cubes algorithm in computer graphics for isosurface polygonisation to make use of new developments in the sparse field level set method, which allows localised updates to the implicit level set surface. This is then applied to an example medical image analysis and visualisation problem, using user-guided intelligent agent swarms to correct holes in the surface of a brain cortex, where level set segmentation has failed to reconstruct the local surface geometry correctly from a magnetic resonance image. The segmentation system is real-time and fully interactive.
1
Background
The level set method, introduced in [1], has shown great promise for medical image analysis [2,3,4]. However, threshold based techniques suffer when classes (e.g. tissue types) have inhomogeneous voxel intensities across the input image. In some cases the areas showing error are immediately apparent in a 3D rendering of the final segmentation. It would then be advantageous to allow for manual identification followed by autonomous correction of these areas in an efficient and fully interactive manner. 1.1
The Level Set Method
We can define an implicit surface as an isosurface embedded in a scalar field, {p | φ(p) = 0} - the zero-level set of φ. We further assume that φ < 0 indicates inside and φ ≥ 0 indicates outside a volume and that the surface is advected in the normal direction. Then we arrive at the level set equation: ∂φ(p) = |∇φ(p) |F (p) ∂t
(1)
where F is an arbitrary function controlling the speed of surface points. Updating the values of φ at each point in space requires a numerical technique to evolve the zero-level set from an initial specification. In the naive case, the whole of space must be evaluated at each iteration, giving (in three dimensions)
O(n3 ) complexity, rather than just the area of interest - the O(n2 ) surface. Algorithms have been developed to overcome these issues by only performing updates on regions near the surface, rather than integrating over the entire space. The most well-known is the narrow band approach [5,6]. This method has since seen several developments that trade accuracy for efficiency by reducing the width of the band, along with algorithmic changes. In-particular, the sparse field method [7] has been widely adopted, especially for interactive-rate applications [8,9] and is used in this work. In order to allow local and interactive modification of the level set surface, the sparse field algorithm needs to be modified. A modified sparse field algorithm that allows local modification of the level set surface is described in [10]. The level set method works particularly well with the marching cubes algorithm, as the signed distance scalar field provides the required inside/outside corner statuses. That is, each lattice node stores its distance to the object’s surface, with nodes inside the object having negative (or positive, if so defined) sign. By looking for adjoining nodes with opposite sign we can find those edges that the object’s surface bisects. With this information we can construct a triangle mesh representative of the isosurface, as detailed below.
2
An Extended Marching Cubes Algorithm
In order to visualise the local modification of the level set surface, we also require a method of rendering the surface in 3D as it is being updated. To reconstruct a surface from volumetric data, the original marching cubes method divides the data volume into grid cells and polygonises each cell individually. Each cell is defined as 8 corners that together form a cube, with the specific triangle configuration determined by which edges of the cell are bisected by the surface. That is, a continuous surface is approximated by finding the bisection points along each edge of each cell that the surface cuts through. To determine how the bisection points should be connected to form a triangle mesh, the marching cubes method makes use of two look-up tables. The first look-up table takes a bitwise encoded index of corner inside/outside status (8 bits) and returns a bitwise encoded list of edge vertices (12 bits), with each ‘on’ bit corresponding to an edge vertex that must be calculated. The second look-up table details how those vertices should be linked together to form the final triangle mesh polygonisation of the cell. As with the level set method, the naive marching cubes implementation has O(n3 ) time complexity. By exploiting a narrow band data structure the complexity is reduced to O(n2 ). If we are only modifying a few localised areas on the surface, whilst the majority of the surface remains static, we would like to forgo the necessity of cycling through the entire band just to update these regions. Therefore, we have further modified the method to reduce the complexity to O(1) for a single small region, or O(n) for n such regions. We have integrated this new marching cubes algorithm with the localised sparse field algorithm in
Figure 1. Illustration of the vertex index look-up grid, G. Each grid point p stores three indices G(p,0) , G(p,1) , G(p,2) , one for each cardinal direction. Each index acts as a reference to a vertex in the V vertex array.
[10], allowing for real-time visualisation of deformations by agent swarms, even for large surfaces.
2.1
Selecting Modified Mesh Fragments
We create a lattice of mesh fragments, M, each element of which contains a polygonisation covering a set of φ lattice nodes. In other words, we break up the complete mesh into smaller fragments and only repolygonise those fragments that are responsible for one or more of the cells that have been modified (e.g. by agents deforming the surface), rather than repolygonising the whole surface/space. To find and repolygonise invalidated mesh fragments, we maintain the mesh fragment lattice as well as a matching lattice of boolean flags B in memory, which are initially all f alse. As input we require a set of φ points that have been invalidated (i.e. changed in value), X ⊂ Z3 . We start by defining an empty array of fragment locations U ⊂ Z3 . We then cycle through all the points in X to find which fragments have been invalidated and append them to U. Since each fragment covers multiple φ lattice points, the B lattice is required to ensure no duplicate fragment locations are added to U. Now we have the set of invalid mesh fragments U, we reconstruct all marked fragments, M(p∈U ) , and reset corresponding boolean flags, B(p∈U ) = f alse. Reconstruction of a fragment proceeds by polygonising each cell that it covers, one at a time. Each fragment stores an array of vertices V ⊂ R3 ×R3 (position and normal), which is shared amongst all cells of the fragment, allowing vertices to be reused rather than recalculated, where possible. In addition, each fragment stores a triangle array, T ⊂ Z3 - a list of 3-tuple indices referencing vertices in V. The procedure for polygonising each cell is detailed in the following subsection.
2.2
Polygonising a Cell
The first step in polygonising a cell is to calculate the index i into the look-up tables: ξ, the edge table; and Λ, the triangle table (available, for example, from [11,10]). The next step is to compute the vertices (bisection points) lying along the edges of the cell. This results in a 12 -element array, v ⊂ Z - one element for each edge of the cell being polygonised, with each element storing an index into the vertex array V, or a N U LL value if a vertex is not required along that edge. The vertices are then connected to form the triangle list. Each of these steps are detailed below. Calculating indices into the look-up tables: each corner of the cell is tested in a specific order and the index - an 8-bit value i - is modified in a bitwise fashion depending on the result. We first test each corner in turn, setting the respective bit in i to 1 if that corner is inside the volume, and 0 otherwise. When i is resolved from a bit string into an integer we get our index into the look-up tables. The 12-bit value stored in the edge look-up table at ξ(i) can then be used to rapidly determine which edge vertices are required. First, we must define the 12 element array v for this cell. If a bit in ξ(i) corresponding to a given edge is switched on, then the proper polygonisation of this cell requires a vertex that is positioned along that edge. Fetching/calculating an edge vertex: by defining a cell edge as a start point, p ∈ Z3 , and a direction, d ∈ {0, 1, 2}, we can define a lattice G that stores an index into V for each positive cardinal direction, as shown in Fig. 1. If the vertex has already been created and added to V, then its index will be stored in G and we can simply return the value at G(p,d) . Otherwise we must calculate the vertex, append it to the vertex array V, and store its index in G for future queries. The use of G allows us to reuse vertices rather than duplicate them for every triangle that shares this vertex in this and neighbouring cells. By using G we can ensure in O(1) time that no unnecessary recalculations are performed and allow the use of index buffers in the graphics device memory, reducing the (typically large) memory footprint. The calculation of a new vertex v, such that vpos gives the vertex position and vnorm gives the vertex normal, is performed by linearly interpolating the values of φ at the two endpoints of the edge, such that φ(vpos ) = 0. Similarly we can calculate the vertex normal using ∇φ at each endpoint. Once v is calculated we append it to the end of the vertex array V and update the look-up lattice at G(p,d) with the index of this vertex (which is simply the size of the array V minus 1). Triangulating a cell: each triangle is defined as a 3-tuple, τ ∈ Z3 , of indices referencing vertices stored in V. Triangles are defined using the second look-up table, Λ, where Λ(i) gives the triangle list for this particular cell configuration and each element j of Λ(i) , Λ(i,j) ∈ {0, 1, . . . , 10, 11}, gives an index into v.
Figure 2. A partial polygonisation of an arbitrary isosurface. A sample set of the triangle mesh configurations given in the look-up table Λ are combined to show how they fit together to create a coherent triangle mesh.
Let (a, b, c) = (Λ(i,j) , Λ(i,j+1) , Λ(i,j+2) ). The next triangle required to polygonise this cell is constructed from the indices stored in the array v, such that τ = (v(a) , v(b) , v(c) ). τ is then appended to the mesh fragment’s triangle array, T . We then increment, j ← j + 3, to the next triangle and repeat until all triangles for this cell are fetched, as flagged by some N U LL value at Λ(i,j) . Figure 2 shows how several triangle mesh configurations for individual cells join to create a representative triangle mesh hull of the entire isosurface. Once all these steps are complete, U determines which fragments need updating in the graphics hardware, using the fragment’s arrays V and T directly as vertex and index buffers, respectively. Figure 3 shows the results of a simple experiment contracting a cube under mean curvature flow. The cumulative time taken to search for and repolygonise affected mesh fragments is plotted over 1500 time steps for different sized fragments. The largest 100x100x100 fragment covers the entire 100x100x100 lattice, so is the equivalent of performing the marching cubes algorithm on the entire lattice, creating a single mesh fragment as in the naive marching cubes method. It is clear that the smallest fragment sizes perform significantly better, tending to zero seconds per update as the zero-level set becomes smaller. A mesh fragment size of 50x50x50 (giving eight fragments) actually performs slightly worse than simply polygonising the entire φ lattice as a single mesh. The target shape is centred in the lattice, so the extents reach into all eight regions and as a result all eight fragments must be repolygonised every frame. This shows an associated overhead involved in breaking the mesh into fragments, mostly as a result of repeated effort at the border between fragments where vertices are not shared and must be recalculated.
3
Level Set Image Segmentation
For our example we are using user-directed agent swarms to correct a level set segmentation of a MR brain scan. This initial segmentation result is obtained
Figure 3. Cumulative time taken in seconds, over 1500 time steps, to find and repolygonise affected mesh fragments during contraction under mean curvature flow. The experiment takes place in a 100x100x100-node lattice with a 50x50x50 box centred within the lattice, which then undergoes mean curvature flow. Each curve represents mesh fragments of varying size, from 100x100x100 (labelled x100) - the entire lattice as in the naive marching cubes method, down to 3x3x3 (labelled x3) - the smallest workable size.
using the standard sparse-field level set method [7]. We define a simple threshold based speed function for use in the level set equation: F (p) =
1 (1 − α) D(p) + α κ(p) 2
(2)
∇φ Where: κ = ∇ • |∇φ| is the mean curvature, implemented using the difference of normals method (covered very nicely in [15]); and: |µ − I(p) |n D(p) = 2 −1 (3) n + |µ − I(p) |n
Where: I is the input image, assumed to be greyscale, such that ∀p, I(p) ∈ [0, 1]; µ is the ideal value; is the acceptable error; and n controls the steepness of the distribution. This function gives a value in [−1, 1), negative when the data point is within the threshold and positive otherwise. With our choice of signed distance representation (positive outside and negative inside the volume) a negative value for D indicates an outward expansion. To summarise, we have a global ideal voxel intensity and a threshold. Within that threshold the surface expands, although mediated by a small curvature to smooth over noise and prevent leaks. Similarly, outside of the threshold the surface contracts, similarly mediated by the curvature term. The result of this global ideal-value-and-threshold approach gives a nice approximation, when the various free parameters are chosen well, and is relatively fast computationally. However, even after tuning the parameters until they produce the best possible segmentation, it is apparent that this approach is weak when confronted with intra-class inhomogeneity across voxel intensities. In this case the surface prematurely stops expanding in some areas. When viewed in 3D we clearly see ‘holes’
(a) Ground truth
(b) SPM [12]
(c) FSL [13]
(d) Level set
Figure 4. A 3D rendering of the segmentation of grey matter from a BrainWeb [14] simulated MR image. Images show results using different software packages, as well as the hand-segmented ground truth. All clearly show holes in the superior end of the segmentation, including the ground truth.
in the surface. The 3D view shows these holes much more clearly than a 2D slice-by-slice comparison. Figure 4 shows the superior end of the final segmentation result from three segmentation methods, the SPM [12] and FSL [13] software packages, our level set implementation, plus the ground truth. The ground truth, FSL and SPM volumes are stored as fuzzy objects. For these volumes, the surface is extracted as the 0.5-curve, providing the best balance between under and over classification, whereas the level set surface is extracted as the zero-curve (zero level set). We can see in all cases the segmentation suffers from holes in the final surface extraction, with the ground truth actually being the worst offender. We would like to enable users to interactively guide the correction of these holes in real-time. Using mechanisms found in previous work on situating virtual swarms in a deformable environment [16,17,10,18], we embed multiple agents on the surface, whose movements are weighted toward areas that the user identifies with a click of the mouse. The surface in that area is corrected by the swarm, with the results of the swarm’s modifications immediately visible to the user in real-time, thanks to our extensions to the marching cubes algorithm. The following section details the methods we used to embed the agents and have them interactively modify the surface about a user-selected location.
4
Populating the Zero-Level Set with Agent Swarms
Agents can be situated on the zero-level surface with an R3 position and velocity, moving across and modifying the surface using interpolation methods from/to the Z3 -located φ lattice nodes. This is quite straightforward since the level set representation allows us to locate the surface in space and calculate the surface normal at any point very easily. Fetching the φ value at a given R3 location can use standard trilinear interpolation. This easily extends to calculating interpolated surface normals or any other numerical value as required for movement and sensory information. Modifying the surface (that is, the zero-layer φ nodes) at a given R3 location is less straightforward. To do this, we update the values at all zero-layer points within a given range of the agent’s position x by a value A(x), weighted by distance from the agent using a normalised 3D Gaussian distribution. In other words, A(x) gives the amount to raise/lower the surface by. The decision each iteration by each agent on whether to raise/lower the surface uses population-level sampled averages of voxel intensity and standard deviation, so that decisions on surface modifications can be made on a local basis and adapt as the user directs the agents around the surface. For specifics on the A function used in this example see [18]. The φ locations modified by the agents are stored each frame and passed as the X parameter to our marching cubes algorithm as detailed in Sect. 2.1. For movement across the surface, a movement potential function P (x) is specified that represents the ‘forces’ pulling agents in certain directions. The function P does not in general produce a valid movement direction vector along the surface, so it must be resolved to lie along, that is, tangential to, the zerolevel isosurface. Also, the agents will tend to ‘drift’ from the zero-curve over time and could overshoot the narrow band and end up in undefined territory, so a term must be added to compensate for this:
x ←− x + P − ∇|φ| (∇|φ| • P ) − ε∇|φ| (4) where 0 < ε 1 controls the strength of the pull back onto the zero-curve T ∂|φ| ∂|φ| i + j + k . and ∇|φ| = ∂|φ| ∂x ∂y ∂z Again, for implementation details of the P function used in this example see [18].
5
Results and Discussion
Figure 5 shows the result of running the semi-autonomous swarm surface correction algorithm on a simulated MR image taken from BrainWeb [14]. A grey matter level set segmentation created using a sub-optimally low threshold was populated by 200 agents. Lowering the threshold results in more and larger holes in the segmentation than the optimal choice (which still exhibits holes, but less obviously so). The user need only click the mouse whilst pointing in the general
(a)
(b)
Figure 5. Image showing level set surface with visible holes at the superior end of a grey matter segmentation. These holes are repaired by 200 agents directed by a nonexpert user. (a) shows the initial surface, with multiple visible holes; (b) shows the final surface with these holes corrected.
(a)
(b)
Figure 6. A level set segmentation and swarm based correction of grey matter from a real T1 MRI scan, as opposed to a BrainWeb simulation: (a) shows a portion of the initial level set surface; and (b) shows the swarm corrected surface.
area of interest and the agents very rapidly ‘fill’ nearby holes. To show how this methodology is not restricted to idealised BrainWeb images, a further illustration is shown in Fig. 6, where the process of level set segmentation and swarm correction is applied to a real T1 MR image of a brain. The test platform is an Intel Xeon 3GHz CPU, with 4GB RAM and a NVIDIA Quadro FX 5800 GPU. As a straightforward marching cubes mesh visualisation tool, the system achieves between 14000 and 15000 FPS when rendering a grey matter segmentation from a 181x217x181 MR image. Once 200 agents are added and moving without modifying the surface, the system runs at between 1800 and 1900 FPS. Whilst repairing a surface this figure is reduced to between 100 and 300 FPS. More specifically, when agents are coded to raise or lower the surface randomly with a 0.1 probability, that is, on average each agent raises or lowers the surface every 10 time steps, the result runs at 110 FPS. When this probability is raised to 0.5 the result is 55 FPS. The non-linear correlation is due to agents modifying overlapping regions, meaning less level set surface points and thus associated marching cubes mesh fragments need be processed.
References 1. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79(1) (1988) 12–49 2. Zhukov, L., Museth, K., Breen, D., Whitaker, R., Barr, A.: Level set modeling and segmentation of DT-MRI brain data. Journal of Electronic Imaging 12(1) (2003) 125–133 3. Cates, J.E., Lefohn, A.E., Whitaker, R.T.: GIST: an interactive, GPU based level set segmentation tool for 3D medical images. Medical Image Analysis 8(3) (2004) 217–231 4. Shi, Y., Karl, W.: A fast level set method without solving PDEs. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Volume 2. (2005) 97–100 5. Adalsteinsson, D., Sethian, J.: A fast level set method for propagating interfaces. Journal of Computational Physics 118 (1995) 269–277 6. Sethian, J.: Level Set Methods and Fast Marching Methods. Cambridge University Press (1999) 7. Whitaker, R.T.: A level-set approach to 3D reconstruction from range data. Computer Vision 29(3) (1998) 203–231 8. Museth, K., Breen, D.E., Whitaker, R.T., Mauch, S., Johnson, D.: Algorithms for interactive editing of level set models. Computer Graphics Forum 24(4) (2005) 821–841 9. Lefohn, A.E., Cates, J.E., Whitaker, R.T.: Interactive, GPU-based level sets for 3D segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI 2003). (2003) 564–572 10. Feltell, D.: Self-organised virtual swarms in a continuous deformable environment. PhD thesis, The University of Nottingham, Nottingham, UK (2010) 11. Bourke, P.: Polygonising a scalar field. http://local.wasp.uwa.edu.au/ ~pbourke/geometry/polygonise/ (May 2009) 12. Friston, K., Ashburner, J., S.J., K., Nichols, T., Penny, W., eds.: Statistical Parametric Mapping: The Analysis of Functional Brain Images. Academic Press (2007) 13. Zhang, Y., Brady, M., Smith, S.: Segmentation of brain MR images through a hidden Markov random field model and the expectation maximization algorithm. IEEE Transactions on Medical Imaging 20(1) (2001) 45–57 14. McConnell BIC: BrainWeb: Simulated brain database, Montreal Neurological Institute. http://www.bic.mni.mcgill.ca/brainweb/ (2010) 15. Lefohn, A.E., Kniss, J.M., Hansen, C.D., Whitaker, R.T.: A streaming narrowband algorithm: Interactive computation and visualization of level sets. IEEE Transactions on Visualization and Computer Graphics 10 (2004) 422–433 16. Feltell, D., Bai, L., Jensen, H.J.: An individual approach to modelling emergent structure in termite swarm systems. International Journal of Modelling, Identification and Control 3(1) (2008) 29–40 17. Feltell, D., Bai, L., Soar, R.: Level set brain segmentation with agent clustering for initialisation. In: International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS), Funchal, Madeira, Portugal (2008) 18. Feltell, D., Bai, L.: 3D level set image segmentation refined by intelligent agent swarms. In: IEEE World Congress on Computational Intelligence, Barcelona, Spain (2010)
