Automatic Classi cation of Bacteria Culture Images Anil K. Jain and Lin Hong Pattern Recognition and Image Processing Lab. Department of Computer Science Michigan State University East Lansing, MI 48824 email: fjain,
[email protected] November 26, 1996 Abstract Automatic image analysis plays an important role in biological, medical, and ecological applications. However, because of the complexity and diculty of automatic image analysis, no currently available image processing systems can achieve a desired level of performance on di erent types of images. The design of an image analysis system needs to be application-oriented and conform to the requirements of the task. In this report, an image analysis system whose sole purpose is bacteria image analysis is introduced. The details of its implementation are given, and its performance on bacteria culture images, provided by the Center for Microbial Ecology, Michigan State University, is evaluated.
Research
Supported by NSF grant
1
2
1 Introduction Automatic image analysis has important applications in biological, medical, and ecological domains, such as automatic detection of tumors or other diseases from X-ray images of patients, grain size evaluations of materials, etc. Advances in computer hardware and software technologies have made it possible to achieve this goal by using current image processing technology. In fact, many universities, research centers, and companies have already developed a number of image analysis systems, which are for an automatic or semi-automatic analysis of various types of images. New generations of image analysis systems continue to appear in the market for the growing numbers of applications. Unfortunately, automatic image analysis is an extremely dicult task. Over the past 30 years, a number of researchers have devoted a substantial amount of e ort to automatically analyze input images. However, they have not been successful in building a general purpose system. The state-of-the-art in image processing technology is such that it does not allow us to automatically analyze all the images or even images in a speci c application area. For example, in biomedical applications, given an image which contains di erent types of bacteria, it is desired to estimate the percentage of bacteria in each category. However, following problems are typically encountered in practical applications:
Uneven illumination Occlusion Out of focus
3
Smear In the presence of these artifacts, it is extremely dicult for a segmentation algorithm to automatically separate the objects of interest from the background. Figure 1 shows such an example. Another dicult problem is to automatically assign a given set of segmented objects into di erent categories. Currently, interactive techniques have to be used to overcome these dicult design problems that can not handled by automatic algorithms.
(a) bacteria image (b) classi cation result Figure 1: An example of a bacteria image which is extremely dicult to be analyzed automatically; (a) input bacteria image; this image contains a number of bacteria which overlap and occlude each other; some bacteria are not in the focal plane; also there are some objects smearing the bacteria which makes it dicult to tell the exact shape of the smeared bacteria; (b) classi cation result; the classi cation result has been obtained by a semi-automatic technique which includes a substantial amount of user interaction. Currently available image analysis systems can be divided into two categories. The rst category, also called development-oriented image analysis systems have been mostly developed in universities or research centers, such as HIPS, XITE, etc. An image analysis
4 system in this category is usually presented as a collection of software implementations of various image processing algorithms. It is primarily intended to be used as a library for programmers. It provides a basic functionality often needed in image processing. It can be used by researchers in various application domains such as robotics, high energy physics, oil exploration, industrial inspection, medical imaging, etc. The image analysis systems in this category usually provide only general-purpose image processing routines which can be thought of as the basic modules of high-level image processing tasks. They incorporate only minimal interactive tools and lack the common properties of current application-oriented software. Therefore, it is dicult for users who are not computer literate, which is generally the case in the application areas, to eciently use these libraries in their own applications. Lack of user-friendliness is the main drawback of an image analysis system in this category. The other category of image analysis systems includes those currently available in the market. They can be called application-oriented image analysis systems, and usually feature a carefully designed graphical user interface (GUI). The motivation behind an image analysis system in this category is to provide users with image processing routines suitable for various types of image analysis and integrate them into a graphical user interface which helps the user to use these image processing routines easily in their work. An image analysis system in this category normally includes three kinds of tools in the software package: (i) general tools, which consist of general image processing routines, such as simple image transformation, ltering, edge detection, etc.; (ii) speci c tools, which are used for the automatic analysis of some speci c types of images; and (iii) interactive tools, which allow users to operate
5 on the images interactively. In a concrete application, users may rst try to use the most suitable automatic image processing tools for their images. If the automatic tools fail to produce the expected results, the users can recapture the images, make them suitable for the automatic processing or edit the nal results directly to t their own requirements by applying the interactive tools provided by the image analysis system. Unlike a generalpurpose image analysis system, an image processing system in this category usually aims at speci c application areas. As mentioned above, the state-of-the-art in image processing technology is such that it has eluded a general purpose image analysis system. Therefore, a suitable approach to applying image processing techniques to di erent application areas is to design specialized systems for each application area, and make the systems e ective in handling a wide variety of problems. This strategy has been adopted by most image analysis system designers, and has become the common design motivation behind the currently available application-oriented image analysis systems.
1.1 Application-Oriented Image Analysis System Design An image processing task can usually be presented as a sequential, feed-forward, modular process. When images are entered into an image analysis system, a sequence of modules will be called upon to process the incoming images. Each module takes the intermediate output of the previous module and transforms it into representations which are suitable for the later modules, and the last module produces the nal results. Figure 2 shows such an example. In
6 order to make an application-oriented image analysis system re ect this inherent structure of image processing tasks, its design should conform to the following speci cations: 1. Provide an input/output module to manage all the image les, I/O devices, etc.; 2. Provide an image processing module which includes both the commonly-used routines and special routines for the speci c image processing tasks; 3. Provide an interactive environment to integrate all the functions of the image analysis system into a compact package. 4. Provide a mechanism which makes it easy to use new features/modules. Bacteria Image
fm(x,y)
f(x,y)
Load
Smooth
Threshold
Features Precentage of bacteria in each class
Classification
fs (x,y)
Measurement
Figure 2: A typical bacteria image analysis owchart; f (x; y) is the input image; fm(x; y) is the smoothed image; and fs(x; y) is the segmented image. The input/output module can be thought of as an interface between the image processing system and its operating environment. It is mainly responsible for the management of image les, acquisition devices, and output devices. Typically, the input/output module should support a number of commonly-used le formats, and should handle a variety of special types of images, such as sequences of images, multiple band images, etc. The design of this module is relatively invariant across di erent application areas.
7 The image processing module is the kernel of an image analysis system. It can be divided into several sub-modules: (i) common image processing sub-module, (ii) editing sub-module, (iii) measurement/classi cation sub-module, and (iv) application-dependent sub-module.
The common image processing sub-module implements a number of general-purpose image processing algorithms, including ltering, convolution, image arithmetic, image transformation, and standard segmentation algorithms. Because hundreds of image processing algorithms are available in the literature, it is impossible and unnecessary to incorporate all of them into this sub-module. The sub-module should be both complete and concise. To be complete, it must have all the utilities commonly used in image processing; conciseness means that unnecessary image processing algorithms should not be included. Like the input/output module, the design of this sub-module is relatively constant across application areas, di ering mainly in the speci c algorithms included. The editing sub-module is responsible for providing tools which are used to edit input images, intermediate results and nal results interactively. In practice, interactive editing is a very important component of an image analysis system. Sometimes, given a speci c type of images, it is very hard or even impossible to analyze them automatically by using currently available image processing algorithms. For example, uneven sample illumination, shadowing, partial occlusion, clutter, or noise may result in erroneous segmentations of images, which in turn leads to erroneous classi cation/measurement results. Although researchers in image processing are designing new algorithms to solve these problems, their e orts have not been successful. In these cases, the only way we can analyze such types of images is to use
8 some semi-automatic approaches, which means using interactive editing tools to change some portions of the input image and make them suitable for automatic processing. So, in an application-oriented image analysis system, it is very important to provide an ecient interactive editing tool. The measurement/classi cation sub-module is usually the output of an image analysis system. It is typically responsible for producing the nal result of an image processing task. After input images have been processed by various image processing routines, and the objects of interest in the images have been segmented from the background, a measurement routine is called to estimate di erent object attributes such as area, perimeter, elongation, etc. These attributes or features are then entered into a classi er, which assigns objects to di erent classes based on a speci c application context. One of the most dicult and important problems in the design of a classi er is the selection of the features or descriptors [4]. Over the past 30 years, extensive research has focused on the development of ecient and reliable methods for the selection of salient features in the design of classi ers [11, 4]. However, none of the proposed methods eciently solve the problems for all applications. Thus, in the implementation of an image processing software, it is critical to select a set of suitable features and to design the corresponding classi er based on the application requirements. The application dependent sub-module is designed for a given application task, which is dicult or sometimes impossible to ful ll by using general-purpose image processing algorithms. Because this sub-module is problem-dependent, its implementation should conform to the requirements of the speci c application.
9 The interactive environment is responsible for the integration of all the routines of the image processing software into a compact software package and to provide a graphical user interface allowing users to easily execute di erent routines. For an application-oriented image analysis system, a well-designed interactive environment is as important as the image processing routines themselves. Without a well-designed interactive environment, it is very dicult for users to apply the system to their image analysis problem, especially when users are not computer specialists. Advances in human-machine interaction have enabled us to implement the interactive environment as a graphical user interface (GUI), which has become a de facto standard for application-oriented software. The GUI notion of \what you see is what you get" supports the interactive environment design over any other type of human-machine interface design.
1.2 Bacteria Image Analysis System An application-oriented image analysis system whose sole purpose is bacteria image analysis is currently being implemented in the Pattern Recognition and Image Processing Laboratory, Department of Computer Science, Michigan State University. The design of the system completely meets the requirements of an application-oriented image analysis system. It provides suitable functionality for the proposed bacteria image analysis tasks. Figure 3 shows the overall system diagram. The following list of the modules have been implemented:
I/O module Currently supports HIPS le format
10
Image processing module
{ Smoothing { Edge detection { Morphological operations { Segmentation Interactive thresholding Niblack thresholding
{ Measurement Object Filter Object Splitter Active Contour Automatic Measurement Interactive Measurement
{ Classi cation { Interactive Editing Image tools Object tools Graphical user interface
11 Currently, our system is capable of measuring 23 di erent parameters which can be interactively chosen in an image analysis process. These parameters are area, perimeter, average grey scale density, x value of the center of mass, y value of the center of mass, average curvature, angle, thinness, circularity, length of bounding box, width of bounding box, area of bounding box, ratio of bounding box, minimum radius, maximum radius, ratio of minimum/maximum radius angle, angle of minimum radius, angle of maximum angle, longest dimension, average cell width, aspect ratio, cell curvature, and distance to nearest neighbour.
Graphical
User Interface Image Processing Modules
I/O Module
Image Preprocessing
Image
Object
Object
Interactive
Segmentation
Measurement
Classification
Editing
Figure 3: Bacteria Image Analysis System Diagram In the following sections, we will describe several aspects related to the development of the bacteria image analysis system. Section 2 brie y describes several general image
12 preprocessing routines implemented in the system. Section 3 describes the segmentation algorithms. In Section 4, the implementation details of the measurement module of the system are described. Section 5 introduces the features used in classi cation. Section 6 details the design of the interactive environment of the bacteria image analysis system, and the last section contains the summary and a discussion.
2 Image Preprocessing The purpose of image preprocessing is to improve the visual appearance of an image, or to convert the image to a form that is suitable for a later processing stage [18]. Generally, it consists of three types of image manipulation processes: image smoothing, edge detection, and morphological operations.
2.1 Smoothing 2.1.1 Smoothing lter In image processing, there is always some noise presented in input images which may lead to unexpected errors in the nal result. It is a practical requirement to include some noise cleaning tools in an image analysis system. Normally, additive noise has a higher spatial frequency spectrum than the image itself. Hence, a simple low-pass ltering can e ectively remove such noise from the image [18]. We have implemented a smoothing lter in our system. The operation of the W W smoothing
13 lter is given by the following formula:
g(x; y) = W1
W
+ 2
X
2
W
+ 2
X
i=? W2 j =? W2
f (x + i; y + j );
(1)
where f (x; y) is the input image gray level at pixel (x; y) and g(x; y) is the smoothed image gray level at pixel (x; y), W is the size of the window which is an adjustable parameter. In our implementation of the smoothing lter algorithm, we have adopted a sliding window technique [18] which is computationally ecient in situation where the window size is very large, for example 128 128. Figure 4 shows the result of applying a 7 7 smoothing lter.
(a) input image (b) smoothed image Figure 4: Operation of a smoothing lter; window size is 7 7.
14
2.1.2 Median lter Generally, the smoothing lter discussed above is capable of eciently removing the additive noise presented in an input image. However, this type of lter tends to blur the input image which may lead to undesired errors under some circumstances. A median lter can remove noise from input images with a minimum amount of blurring e ect. It performs much better than the smoothing lter under most circumstance. Therefore, it is necessary to include a median lter in our system. The operation of a W W median lter can be de ned as follows:
g(x; y) = MEDIAN i;j 2R x;y f (i; j ); (
)
(
)
(2)
where f (x; y) represents the grey level value of the input image at pixel (x; y) and g(x; y) represents the grey level value of the smoothed image at pixel (x; y), R(x; y) prepresents a
W W window centered at pixel (x; y), and MEDIAN means the median of the grey level values within the speci ed window. Generally, a median lter operation is time-comsuming, especially when the window size is large. However, its high computational complexity can be greatly reduced by using a sliding window technique when input images are binary images. Figure 5 shows the results of applying a 5 5 median lter on two images.
15
(a) input image
(b) ltered image
(a) input image (binary) (b) ltered image (binary) Figure 5: Operations of a median lter; window size is 5 5.
2.1.3 Nonuniform Background Correction Due to uneven illumination conditions, un at samples, or variations in thickness of the slides, the acquired images often have an uneven background. Such an image can be considered as a low-variation background image added to an ideal foreground image in a constant background. By applying the smoothing lter with a very large window size (for example 128 128), the low-variation background image can be estimated. An estimation of the foreground image is obtained by subtracting the background from the input image. This is the main idea of nonuniform background correction algorithm implemented in our system.
16 Mathematically, it can be represented as follows:
f (x; y) = f (x; y) ? W1
W
+ 2
X
0
2
W
+ 2
X
i=? W2 j =? W2
f (x + i; y + j ) + ;
(3)
where f (x; y) is the corrupted input image and f (x; y) is the output de-shaded image. W 0
is the size of the averaging window (normally larger than 128); and is an o set. Figure 6 shows an example of applying the de-shade lter to an image with uneven background.
(a) input image (b) de-shaded image Figure 6: Example of nonuniform background correction lter; window size = 128 128, = 128.
2.2 Edge Detection Edges, or sharp discontinuities in image intensity values, are important characteristics of an image because they provide an indication of the physical extent of the objects within the image [18]. There are two generic approaches to the detection of edges in an image:
17 di erentiation and model tting. In the di erentiation approach, a spatial di erentiation is performed on the original image f (x; y) to produce a gradient image g(x; y), where high values indicate the presence of edges in the input image. Next, a thresholding operation is executed to determine the locations of signi cant gradients. The nal result is usually an edge map which indicates the locations of edges in the image. There are two major classes of di erential edge detectors: rst-order derivative methods and second-order derivative methods. In a rst-order derivative method, a digital approximation ofthe rst-order spatial di erentiation is performed on an image to get the edge gradients, which are then compared to a threshold. An edge at a pixel is retained if the gradient exceeds the threshold. For a second-order derivative method, an edge is judged to be present if there is a signi cant spatial change in some given direction of the second derivative. The second approach to edge detection involves tting an edge model to a local region of a pixel. An edge is assumed to be present at that pixel if the model t is suciently accurate. A binary edge map is nally generated to indicate the location of edges in the image. A large number of edge detection methods have been proposed over the past years [18, 3]. Generally, an edge detector should be developed to satisfy the following three criteria: 1. Good detection. A maximum number of edge points must be detected with a minimum number of false detection. 2. Good localization. Edge points marked by the edge detector should be as close to the center of the edge as possible.
18 3. Single response. There should be a single response to a true edge. Due to the diculty in determining the optimal parameters associated with each edge detection technique and the lack of a de nitive performance criteria, it is impossible to nd a universal edge detection algorithm. To meet our performance requirements, we have implemented several most widely used rst-order derivative edge detection algorithms. They are the Sobel, Prewitt, and Roberts edge detectors. Also, Canny edge detector which is based on a second-order derivative is implemented in our system. The Sobel edge detector computes the magnitude and direction of the image gradient by applying the horizontal and vertical di erentiation masks as given below: 2 6 6 6 6 6 6 6 6 4
3
2
7 7 7 7 7 7 5
6 6 6 6 6 6 6 6 4
?1 ?2 ?1 77 0
0
0
1
2
1
and
?1
0
1
?2
0
2
?1
0
1
3 7 7 7 7 7: 7 7 7 5
The formulas for the gradient magnitude and direction are given by q
Msobel = GH + GV ; 2
2
GV ; sobel = arctan G H
(4)
where GH is the horizontal and GV is the vertical gradient value. It is computationally expensive to calculate the gradient magnitude by using the above formula, therefore it is
19 often approximated by
Msobel = jGH j + jGV j ;
(5)
where jj represents the absolute value. Like the Sobel edge detector, the Prewitt edge detector computes the magnitude and direction of the image gradient by applying the following horizontal and vertical di erentiation masks:
2 6 6 6 6 6 6 6 6 4
3
2
7 7 7 7 7 7 5
6 6 6 6 6 6 6 6 4
?1 ?1 ?1 77 0
0
0
1
1
1
and
?1
0
1
?1
0
1
?1
0
1
3 7 7 7 7 7: 7 7 7 5
The formulas for the gradient magnitude and direction of the Prewitt edge detector are the same as for the Sobel operator: q
GV ; prewitt = arctan G
Mprewitt = GH + GV ; 2
2
H
(6)
where GH and GV are the horizontal and vertical gradient values, respectively. Robert's gradient is a simple 2 2 mask that computes the di erence between a pixel and one of its neighbors. Robert's 2 2 mask is given bellow: 2 6 6 6 4
0 ?1 1 0
3 7 7 7: 5
20 Figure 7 shows the Sobel, Prewitt, and Robert's gradient magnitude images for a typical bacteria image. The Canny edge detector has been developed to satisfy the three edge detection criteria mentioned above [3]. The image input f (x; y) is rst convolved with a Gaussian lter G(x; y) that has a width W and standard deviation : (f G)(x; y) =
W X
+
W X
+
u=?W v=?W
f (x + u; y + v)G(u; v):
(7)
Next, the edge points are found by computing the zeros of the second-order directional derivative of the convolved image @@n22 (G f ) in a given direction n. Figure 8 contains an example of the Canny edge detector on a bacteria image.
2.3 Binary Morphological Operation Binary morphological operations are a type of ltering technique which can enhance or identify a speci c form or a structured object within an image based on the geometrical relationship or connectivity of pixels which are expected to belong to the same class [4]. The speci c form of the structural object of interest is represented by a structuring element, which plays the role of a kernel in a convolution operation. The basic morphological operations are dilation, erosion, opening, closing, and thinning. We have implemented all these ve basic binary morphological operations in our image analysis system. Binary dilation of an object increases its size by marking the background pixels adjacent
21
(a) input image
(b) Sobel operator
(c) Prewitt operator (d) Robert operator Figure 7: Outputs of Sobel, Prewitt, and Robert gradient operators. (a) input image; (b) Sobel magnitude value; (c) Prewitt magnitude value; (d) Robert magnitude value.
22
(a) input image
(b) magnitude image
(c) direction image (d) thinned edge map Figure 8: Example of Canny edge detector; W = 15, = 2.0; (a) input image; (b) magnitude image (c) direction image; (d) thinned edge map.
23 to an object boundary as object pixels. It can be de ned as the union of the sum of all the pixels, a, in an object, A, with all the pixels, b, in the structuring element, B , s.t.,
A B = fa + b : a 2 A; b 2 B g:
(8)
Binary erosion of an object reduces its size by removing the object pixels adjacent to its boundary. The complement of the eroded image is obtained by dilating the background
A B = fa + b : a 2 AC ; b 2 B gC ;
(9)
where AC andfgC are the complements of A and fg, respectively. Binary opening is de ned as the erosion of the object, followed by dilation. Opening can be used to remove small speckles or small protrusions and narrow openings of objects. The formula for a binary opening is de ned as:
A B = (A B ) B;
(10)
where B is the structuring element. Binary closing operation is de ned as the dilation of the object, followed by erosion. Closing is used to remove small inward bumps and small holes in objects. The formula is:
A B = (A B ) B;
(11)
24 where B is the structuring element. Closing is the dual of opening operation. Binary thinning successively deletes all the border pixels of a region in a symmetric way until only its skeleton is left. This skeleton, also called the medial axis, preserves the connectivity of the original region. A large number of thinning algorithms have been published [4]. In our image analysis system, we have implemented one of the fastest thinning algorithms proposed in the past years [15]. Figure 9 and Figure 10 contain examples of morphological operations on a binary image.
3 Image Segmentation One of the most important tasks in image processing is to partition an image into homogeneous regions [18]. This procedure is called image segmentation. Image segmentation is a very dicult problem. Although over the past 30 years a large number of image segmentation methods have been proposed, there is still no general methodology for image segmentation [22, 10, 20, 19, 6, 18]. Image segmentation algorithms usually assume some speci c type of image model and are generally restricted to those types of images that t the assumed model exactly. These algorithms fail on images which do not t the assumed model. A very important problem in the design of an image analysis system is to determine the image models and the corresponding segmentation algorithms which are suitable for the image of interest. The bacteria images of interest to us contain objects placed in a background with a
25
(a) input binary image
(b) dilation
(c) opening (d) closing Figure 9: Examples of morphological operations on a binary image; all the structuring elements in the above morphological operations are of size 3 3.
26
(a) segmented ngerprint image (b) thinning result Figure 10: An examples of a thinning operation on a binary image. di erent intensity. For such images, intensity is a distinguishing feature that can be used to segment the objects from its background. Therefore, a thresholding technique becomes a strong candidate for an ecient segmentation of these images. Thresholding is the process of partitioning pixels in the images into object and background classes based upon the relationship between the gray level value of a pixel and a parameter called the threshold value. Because of its eciency in performance and its simplicity in theory, thresholding techniques have been studied extensively and a large number of thresholding methods have been published [22, 19]. These published methods can be divided into two categories: global methods and locally adaptive methods. Global methods compute a single threshold value for the entire image, and pixels having a gray level value less than the threshold value are marked as belonging to one class, otherwise the other class. Locally adaptive methods, on the other hand, compute a threshold value for each pixel on the basis of information contained
27 in a local neighborhood of the pixel. The output of a thresholding operation is typically a binary image. Based on the properties of the images of interest to us, we have implemented two di erent thresholding algorithms. One of the methods is the interactive global thresholding and the other method is the Niblack's locally adaptive thresholding [16].
3.1 Interactive Global Thresholding Global thresholding assumes that the input images have an even background. A threshold value is selected which minimizes the segmentation error. The selection of the threshold value is critical for good segmentation. A number of analytical approaches to automatic selection of threshold value have been proposed [22, 19, 6]. Most often, these methods can automatically select a threshold value based on the minimization of some criteria function. However, in practice, it may be very dicult to get the desired segmentation result for a variety of input images by using an automatically selected threshold value. Some user interaction may be necessary for the correct segmentation of the images. In our image analysis system, we have implemented an interactive global thresholding tool, by which a user can interactively select the optimal threshold value based on his/her own experience. The main idea of an interactive global thresholding is to provide the thresholding result to a user in real time. Because the segmentation result can be seen immediately by the user when he/she selects a speci c threshold value, it is very easy for users to select the optimal threshold value in their applications. Figure 11 shows an example of interactive thresholding
28 on a bacteria image.
(a) input image (b) segmented image Figure 11: Example of interactive thresholding; threshold value = 75; although speckles spread all over the segmented image, they can be easily removed by the postprocessing module which will be introduced in next section.
3.2 Adaptive Thresholding Because of uneven illumination conditions, un at samples, variations in thickness of the slides, and input noise, the acquired images often have an uneven background. The nonuniform background correction technique can, to some extent, remove the uneven background from the image. However, under some circumstances, it may be impossible to completely remove it. This usually leads the failure of the global thresholding technique to get the correct segmentation result. In this case, a locally adaptive thresholding technique may be used to overcome the problem. Niblack's thresholding method is computationally fast and
29 easy to implement [22, 16]. The main idea of Niblack's thresholding method is to vary the threshold value over the input image, based on the local mean and local standard deviation. The threshold value at a pixel (x; y) is computed by
T (x; y) = m(x; y) + ks(x; y);
(12)
where k is an adjustable parameter which depends on the image content, m(x; y) and s(x; y) are the sample mean and standard deviation of gray level values in a local window of size W
W centered at pixel (x; y), respectively, which can be computed by the following formulas: m(x; y) = W1
v u u u s(x; y) = u t
1 W
W
+ 2
X
2
W
+ 2
X
2
W
+ 2
X
i=? W2 j =? W2
W
+ 2
X
i=? W2 j =? W2
f (x + i; y + j );
(13)
(f (x + i; y + j ) ? m(x; y)) : 2
(14)
We have designed a menu by which a user can conveniently select the values of k and W . Figure 12 contains an example of Niblack's thresholding on a bacteria image.
30
(a) input image (b) segmented image Figure 12: Example of Niblack thresholding; k = 0.002, W = 15; in the segmented image, a large number of noise, speckles or ghost areas are segmented to be objects, however, they di erentiate from the bacteria in shape and size, a postprocessing algorithm based on these two measures can eciently delete all of them, which will be introduced in next section.
4 Measurement After an image has been segmented into objects and background, the next step is to extract the objects from the background the compute the values of the salient features that are most suitable for di erentiating the objects from each other. This is what we call object feature measurement. Mostly, the purpose of object feature measurement is to aid in classi cation. In an ideal case, each object in a segmented binary image is represented as a connected component. However, in practice, there is usually noise, speckles and conjugations in the segmented binary image. Some postprocessing methods need to be used to remove all these unwanted components. We have integrated several postprocessing methods in the feature
31 measurement sub-module. These include: object lter, object splitting and active contour. After each object is identi ed, the object feature measurement procedure is called to measure the values of the corresponding features. Currently, our system is capable of measuring 23 di erent feature values. In practice, due to some undesired e ects, an automatic measurement sub-module may not always work correctly. In this case, an interactive measurement will be much helpful. Therefore, in our system, we have provided two types of measurement sub-modules, one is an automatic measurement sub-module which can operate without any user intervention and the other is an interactive measurement sub-module which operates interactively. Finally, in practice, there occasionally are some requirements that can not be handled by those general-purpose sub-modules. Therefore, some specialized sub-modules need to be incorporated in our system.
4.1 Object Filter The rst postprocessing method used in the image analysis system is a three-level object lter which hierarchically removes those connected components from the segmented binary images that are not likely to be objects of interest. The rst level of object lter is based on area thresholding which removes all the connected components whose area is either less than t or larger than t , t < t . Connected components 1
2
1
2
with small areas can be thought of as noise or speckles and connected components with very large areas might actually correspond to \ghost" objects. The area thresholding can, to some extent, e ectively achieve this goal without deleting objects of interest.
32 Based on the observation that the circularity of objects of interest are always in some prede ned range, we have used a thresholding on this attribute as the second level of object lter. All the connected components with a circularity value outside of a prede ned range are removed from the segmented image. Curvature thresholding, the third level of object lter, removes all the connected components with a high average curvature value. This is based on the observation that the boundary of objects of interest are smoother than the noisy and \ghost" areas. The curvature is de ned as the rate of change of slope in a given boundary. For each pixel on the boundary of a given object, the curvature c is de ned as
c = (left ? right + 360) mod 360;
(15)
where left is the slope of the vector joining the current pixel and a boundary pixel located
k pixels to the left, and right is the slope of the vector joining the current pixel and a boundary pixel located k pixels to the right. In our implementation, k is an adjustable parameter, which depends on the size of the objects in the image. Its default value is 3 and this value is suitable for most applications.
4.2 Object Splitting A given connected component might correspond to two or more overlapping or touching objects. Based on the observation that the boundary of an object should be smooth, a connected component will correspond to more than one object if its boundary contains sharp
33 turns. We have implemented two automatic connected component splitting algorithms. Both of them are heuristic-based algorithms. One is proposed by Jain and Dubuisson [10]. The other one is an algorithm based on 4-3 Chamfer distance transformation. Figure 13 contains an example of the rst two levels of postprocessing on a segmented bacteria image.
4.3 Active Contour Under certain circumstances, a segmented object boundary may not t the true object boundary precisely which results in incorrect measurement values. In order to make the measurement values correctly re ect the object attributes, an active contour or snake algorithm is applied to obtain an exact boundary. An active contour or snake can be thought of as an energy minimizing spline in the
x ? y image plane [13, 14]. Its behavior is controlled by the internal and external forces. The internal forces serve as a smoothness constraint and the external forces make the snake linger on the salient image features, such as edges, corners, etc. Representing a snake in the parametric form v(s) = (x(s); y(s)), where s is the spatial index, the potential energy function of a snake can be de ned as follows,
Esnake =
Z 0
1
Eint (v(s)) + Eimage (v(s)) + Econ(v(s))ds;
(16)
where Eint is the internal force, Eimage is the image force, and Econ is the constraint force. Image forces can be represented as a linear combination of forces due to lines, edges,
34
(a) input image
(b) segmented image
(c) after object ltering (d) after object splitting Figure 13: An example of postprocessing on a segmented image. The segmented image in (b) is obtained by applying the interactive thresholding method on the input image in (a), the threshold values of area lter are 64 and 4096 pixels, respectively, the circularity is between 0:1 and 1, and the curvature threshold value is 30 .
35 corners, etc., which can be computed from the input image independently. Constraint forces can arise due to any kind of constraint on the shape or other features of the snake. These forces are application dependent and are de ned completely by the users. External energy is comprised of a rst-order and a second-order term. The former makes the snake act like a membrane, and the second term makes it look like a thin plane. It can be de ned as follows,
Eint (v(s)) = w (s)jvsj + w (s)jvssj ; 1
2
2
2
(17)
where w (s) is the tension of the snake, and w (s) is its rigidity. 1
2
Mathematically, the minimization of energy function Esnake can be done by solving a set of Euler equations [13, 2, 14],
?w xss + w xssss + Exext = 0;
(18)
?w yss + w yssss + Eyext = 0;
(19)
1
1
2
2
where Eext is the sum of Eimage and Econ, and x(y)ss and x(y)ssss are the second and fourth order derivatives of the parametric curve, respectively. In practice, a snake has to be discretized in digital image space. High-order derivatives of the discrete data thus have to be estimated. However, estimation of high-order derivatives
36 of discrete data tends to be extremely inaccurate, and it results in the snake to have unpredictable behavior and to be very unstable. Amini et al. [2] proposed a dynamic programming algorithm for minimizing the energy function at the expense of numerical complexity. It promotes stability by reducing the required higher order of di erentiability to rst and second order di erentiability and ensures convergence of the energy minimization process in a nite number of steps. Amini's algorithm works well in the ideal case. Unfortunately, it su ers seriously from the problem of local minima in the presence of noise. In our implementation, a simulated annealing technique was used which overcomes some of these limitations.
4.3.1 Snake Parameters Leymarie and Levine [14] point out that a very important aspect concerning the snake model is the selection of its di erent parameters. First of all, the number of snake points (snaxels) is an important factor. However, since this parameter is application dependent, it is very dicult to choose an optimal number of snaxels. Therefore, it has to be left as a user speci ed parameter. Second, tension and rigidity weights can be de ned as the distance between the snaxels and their corresponding true positions in the image, which in turn can impose their in uence naturally on the snake behavior. In addition, making the ratio (w (s)/w (s)) zero 1
2
at some points can make the snake t the corners or cusps. Third, in order to make the snake stable in its dynamical behavior, it is very important to select proper energy force bounds [14]. Normally, this can be done by normalizing the energy forces and restricting their range to a prede ned window, which again must be speci ed by the user. Fourth, for
37 the snake to behave properly, a reinitializing process is required during the iterative process. Finally, the termination criterion is a key aspect to ensure the convergence of the snake to the global minimum. Leymarie and Levine [14] have proposed a steady-support criterion, which only minimizes the potential image energy Eimage as an alternative termination criterion. However, it can not ensure the global minimum either.
4.3.2 Snake Implementation We have implemented a snake by using the dynamic programming technique. The dynamic programming algorithm views the problem of energy minimization as a discrete multistage decision process, which starts from an initial point and makes decisions from a nite set of possible choices at each succession. At each stage, only a neighborhood of a point on the snake is considered. The energy function to be minimized can be represented as follows:
Esnake(i + 1; j; k) = min >
> :
0 otherwise:
b(x; y) = >
(22)
In our system, a fast algorithm has been implemented to computed the area of an object.
4.4.2 Perimenter The perimeter (P) of an object is the perimeter length of the object. Let (c ; c ; :::cn) denote 1
2
the boundary chain code of an object, the perimeter P of the object can be represented as follows:
P=
n X i=1
"
p
#
2?1 1+( )(1 ? (?1)ci ) : 2
(23)
In our system, after an object is located, a contour tracing algorithm [21] is applied to nd the boundary chain code of the object. After that, the above formula is used to computed the perimeter of the object.
41
4.4.3 Average Grey Scale Density The average grey scale density (AGSD) of an object de nes the ratio of grey density within the object to the background. It represents the optical density of each object individually rather than the optical density of the entire image. Average grey scale density is determined by measuring the average grey level value of the background and the average grey level value of each individual object within the image. This parameter is usually used to distinguish population of objects in a community which have the same morphology and di er in refractive indices. Let R denote the region occupied by an object and f (x; y) denote the grey level value of the image in which the object reside at pixel (x; y), the average grey scale density
AGSD of the object can be computed with the following fomula: PN PN
AGSD = PxN
=1
x=1
y=1 s(x; y ) ; y=1 b(x; y )
PN
(24)
where N N represents the size of the image, 8 > > >
> :
0
s(x; y) = >
otherwise;
(25)
and 8 > > >
> :
0 otherwise:
b(x; y) = >
(26)
42
4.4.4 Center of Mass The center of mass of an object is the center position of the object with respect to its mass. It represents the ideal position of the object in the image in which it resides. Let R denote the region occupied by an object, the horizontal value x and vertical value y of the center of mass of the object can be de ned as follows:
x= y=
PN PN
x=1 y=1 xb(x; y ) ; x=1 y=1 b(x; y ) PN PN x=1 y=1 yb(x; y ) ; PN PN x=1 y=1 b(x; y ) PN PN
(27) (28)
where N N is the size of the image and 8 > > >
> :
0 otherwise:
b(x; y) = >
(29)
The computation of the values of the center of mass in our system is performed in the area computation algorithm which is rather fast.
4.4.5 Average Curvature The average curvature of an object, to some extent, re ects the smoothness and regularity of the object boundary. In our system, the curvature at a given point on an object boundary is de ned as the rate of change of slope in degree. The average curvature of an object is, thus, de ned as the average rate of change of slope in degree along the object boundary. Let
43 (x ; y ); (x ; y ); :::(xn; yn) denote the coordinates of all the boundary pixels of an object, the 1
1
2
2
curvature at a given point (xi ; yi) is de ned as:
i =j tan? ( xyi k ) ? tan? ( xyi?k ) j; +
1
(30)
1
i?k
i+k
where k is a pre-de ned interval. The value of the average curvature of the object is computed by the following formula:
=
Pn
i=1 i :
(31)
n
4.4.6 Angle The angle of an object is de ned as the angle between the x-axis and the axis around which the object can be rotated with minimum inertia. Let f(xi ; yi); i = 1; :::; ng denote the set of coordinates of the pixels belonging to an object and (x; y) denote the position of the center of mass of the object. The covariance matrix of the object is de ned by 2 6 6 xy = 6 4
m; = 20
?m ; = 11
Pn
i (xi ? x)
?m ; =
2
11
Pn
i (xi ? x)(yi ? y)
m; = 02
3
Pn
i (xi ? x)(yi ? y ) 7 7 Pn
i (yi ? y)
2
7: 5
(32)
With this de nition of covariance matrix which is a real and positive semi-de nite, the angle
of the object can then be computed by: = 21 tan? ( m 2m? ;m ): 11
1
;
20
;
02
(33)
44
4.4.7 Thinness The thinness of an object is a kind of regularity measure of the object which is de ned as the ratio of the minimum eigenvalue and maximum eigenvalue of its covariance matrix. Let xy denote the covariance matrix of the object de ned above. Since this matrix is a real and positive semi-de nite matrix, usually, it has two non-zero real eigenvalues, min and max . The ratio of the minimum eigenvalue and maximum eigenvalue which is de ned as: = min max
(34)
1
is a measure re ecting the thinness of the object. For a perfect circle, =1 and for any 1
other object, 0 < 1. The more elongate an object is, the smaller the value of the 1
thinness of the object.
4.4.8 Circularity The circularity of an object is also a kind of regularity measure of the object. In our system, in order to facilitate the implementation of the classi cation map which will be introduced in the classi cation section, it is de ned as: = 4Area ; Perimeter 2
2
(35)
where Area represents the area of the object and Perimeter represents its perimeter. Obviously, this de nition is inverse to the traditional de nition of circularity. The value of
2
45 is expected to be close to 1 if the shape of the object resembles an ellipse.
4.4.9 Minimum Bounding Box The minimum bounding box of an object is de ned as the smallest rectangle which totally surrounds the object with minimum area (Figure 15). The length of minimum bounding box (LBB) is de ned as the length of the rectangle; the width of minimum bounding box (WBB) is de ned as the width of the rectangle; the area of minimum bounding box (ABB) is de ned as the area of the rectangle; and ratio of minimum bounding box is de ned as the ratio of the shortest side of the rectangle and the longest side of the rectangle.
Length of Minimum Bounding Box
Minimum Bounding Box
Object Width of Minimum Bounding Box
Figure 15: Minimum bounding box of an object.
4.4.10 Minimum/Maximum Radius The minimum radius (MINR) of an object represents the smallest distance between the boundary pixels of the object and its center of mass. Let f(xi ; yi); i = 1; :::; ng denote the set
46 of coordinates of the pixels on the boundary of an object and (x; y) denote the coordinates of its center of mass. The minimum radius of the object can be de ned as: N q
(xi ? x) + (yi ? y) : MINR = min i 2
2
=1
(36)
The minimum radius angle (MINRA) of an object is de ned as the angle between the minimum radius in outward direction and the x-axis in degree. It can be represented as:
MINRA = tan? ( xymin );
(37)
1
min
where (xmin ; ymin) are the coordinates of the boundary pixel which has the minimum distance to the center of mass of the object. The maximum radius (MAXR) of an object represents the largest distance between the boundary pixels of the object and its center of mass. Let f(xi ; yi); i = 1; :::; ng denote the set of coordinates of the pixels on the boundary of an object and (x; y) denote the coordinates of its center of mass. The maximum radius of the object can be de ned as: q
N (xi ? x) + (yi ? y) : MAXR = max i =1
2
2
(38)
The maximum radius angle (MAXRA) is de ned as the angle between the maximum
47 radius in outward direction and the x-axis in degree. It can be represented as:
MAXRA = tan? ( xymax );
(39)
1
max
where (xmax ; ymax) are the coordinates of the boundary pixel which has the maximum distance to the center of mass of the object. Ratio of minimum/maximum radius angle (RRA) of an object is de ned as the ratio as follows:
MINRA : RRA = MAXRA
(40)
4.4.11 Longest Dimension The longest dimension (LD) of an object is de ned as the largest distance between two boundary pixels of the object. It represents the length of the object. Let f(xi; yi); i = 1; :::; ng denote the set of coordinates of the pixels on the boundary of an object, the longest dimension
LD can be represented as follows: q
n n (xi ? xj ) + (yi ? yj ) : LD = max max i j =1
=1
2
2
(41)
4.4.12 Average Cell Width The average cell width (ACW) of an object is a measure which measures the average width of the object. The width of an object at a given boundary pixel is de ned as the distance
48 between the given boundary pixel and the boundary pixel on the opposite side of the object boundary (Figure 16). Let fwi; i = 1; :::; ng denote the set of the widths of all the boundary pixels of an object, the average cell width ACW of the object can be de ned as follows: n X ACW = n1 wi:
(42)
i=1
Cell
cell width
Figure 16: An illustration of average cell width.
4.4.13 Aspect Ratio The aspect ratio (AR) of an object is de ned as the ratio of the area of the object to the area of its minimum bounding box:
Area ; AR = ABB
(43)
where Area represents the area of the object and AOBB represents the area of the minimum bounding box of the object.
49
4.4.14 Cell Curvature The cell Curvature (CC) of an object is a measure that re ects the variance of its boundary pixels in position with respect to an ideal circle. In our system, it is de ned as the standard deviation of the distances between the boundary pixels of the object and the equivalent circle of the object. The equivalent circle of an object is de ned as the circle which has the same area as the object and resides at the center of mass of the object (Figure 17). Let
fdi; i = 1; :::; ng denote the set of the distances between the boundary pixels of an object and its equivalent circle, the average cell curvature CC of the object can be de ned as follows: v
uX un CC = n1 t (wi ? w) ; i
(44)
n X w = n1 wi:
(45)
2
=1
where
i=1
4.4.15 Distance to Nearest Neighbor The distance to nearest neighbor (DNN) of an object is the distances between the object and its nearest neighbor. Let f(xi; yi); i = 1; :::; mg denote the set of the coordinates of the center of mass of all object appearing in an image, the nearest neighbor distance DNNk of
50
Equivalent Circle
Object
Center of Mass
Figure 17: An illustration of the equivalent circle of an object. a given object k can be represented as follows: q
DNNk = min ((xi ? xk ) + (yi ? yk ) ): i6 k 2
=
2
(46)
4.5 Measurement Interface In the above sub-sections, we have introduced the procedured with which objects are extracted from segmented binary images and the feature measures which can be computed with our system. In order to make our system exible and easy of using in measurement of features in applications, we have implemented an interactive measurement interface which is capable of greately facilitating the operation of the measurement procedure. In this subsection, we will introduce some detail of this interface design. In practice, it usually happens that users need only a subset of feature measures of
51 the objects appearing in a given segmented binary image. In this case, we can, of course, provide them with all the feature measures of the objects. However, this approach tends to be inecient and inconvenient because it spends the system a large amount of resources to computed those unneeded feature measures and imposes a requirement on users which needs them to search in the data le to obtain the feature values they need. Therefore, it is a general requirement for an application-oriented image analysis system to provide a mechanism such that a user can select a subset of feature measures which he/she wants the system to compute. In our system, we have designed and implemented such a measurement interface with which a user can interactively select the feature measures. The layout of the measurement interface is shown in Figure 18. Another general requirement of an application-oriented image analysis system is to provide an interactive mechanism such that a user can interactively specify a region of interest in a given segmented image and makes the system compute the selected feature measures of the objects appearing only in the region of interest. In our system, we have implemented a interactive measurement sub-module which allows users to interactively specify the region of interest. Figure 19 shows such an example. Under certain circumstance, it is required for an application-oriented image analysis system to provide an interactive mechanism that enable a user to selectively compute the feature measures of the objects appearing in a given segmented binary image. Typically, such a requirement is to impose the system to provide a mechanism that is capable of measuring
52
Figure 18: The layout of the measurement interface implemented in our system. In the 4th window of the interface, there are 24 check buttons which correspond to the feature measures that can be computed with our system. If any of them are checked, the corresponding feature measures will be computed and stored in the measurement procedure.
53
(b) measured image (a) segmented image Figure 19: An example of interactively specifying the region of interest; (a) the region of interest speci ed interactively; (b) measurement of the objects appearing only in the region of interest. those objects appearing in a give segmented binary image whose feature values are within certain ranges. In our system, this is achieved with an interactive selection sub-module. The interactive selection mechanism provides user a graphic user interface which enable a user to interactively select the feature measures to which he/she want to impose restrictions and the corresponding ranges. Figure 20 shows the layout of the graphic user interface of this interactive selection sub-module and Figure 21 shows a result of such selection for a given segmented binary image.
4.6 Interactive Counting Given an image, sometimes it is very dicult or even impossible to analyze it automatically. On one hand, uneven sample illumination, shadowing, partial occlusion, clutter, or noise
54
Figure 20: The layout of the graphic user interface of the interactive selection sub-module. In the second window of the graphic user interface, there are 24 check buttons which correspond to the feature measures that can be computed with our system. If any of them are checked, the corresponding feature measures will be used to check whether their values are within the pre-de ned ranges. The ranges are displayed in the rst window, which can also be changed interactively. may result in erroneous segmentation of the given image, which in turn may lead to erroneous measurement results. On the other hand, under certain circumstance, it is impossible to nd a set of feature measures which are capable of eciently di erentiating the objects appearing in the given image, which means that we can not automatically select the objects we intend to measure from the given image. In these cases, the only way we can achieve our goals is to use some semi-automatic approaches, which means using some interactive tools to perform the tasks. Because of all of these reasons, it is necessary to provide some ecient interactive tools in an application-oriented image analysis system.
55
(a) segmented image (b) result of selection Figure 21: Interactively select objects with measure values within speci ed ranges. The selected feature measures are area (50-500), perimeter (30-200), and thinness (0.5-1.0).
In our system, we have designed and implemented an interactive counting sub-module which enable a user to interactively select objects from segmented images and makes the system compute the feature values of those objects. Figure 22 shows the layout of the graphic user interface of this interactive counting sub-module. The interactive counting submodule operates as follows: (i) a graphic panel that controls all the settings of the interactive counting sub-module is rst presented to a user and the user interactively specify the values of these settings according to the requirements of the applications; (ii) the user interactively select objects appearing in segmented images and the system computes the the measure values of the speci ed objecta and automatically processes them according to the speci ed values of settings. Figure 23 shows such an operation.
56
Figure 22: The layout of the graphic user interface of the interactive counting sub-module. In the second window of the interface, there are 24 check buttons which correspond to the feature measures that can be computed with our system. If any of them are checked, the corresponding feature measures will be computed and stored in the interactive counting procedure.
4.7 Application-Oriented Measurement In bacteria image analysis, it happens sometimes that for a given task it is dicult and even impossible to ful ll it by using those general-purpose image processing sub-modules. Application-oriented sub-modules are neccessary to be provided. These sub-modules are problem-dependent, their implementations should conform to the requirements of the speci c applications. In the following, we will introduce two application-oriented sub-modules implemented in our system.
57
(a) segmented image (b) result of selecting measurement Figure 23: Selective object measurement. In this example, the measured objects are selected by a user.
4.7.1 X-Y Spatial Distribution Analysis X-Y spatial distribution analysis is an important method for analyzing the spatial distribution of biomedical and biological phenomena. Generally, it is capable of predicting what type of spatial process such as random, clustered, regular, etc. best ts a given set of bacteria cultures and whether the spatial pattern changes in distribution. Also, it can reveal whether two types of bacteria culture are correlated with each other. In our system, a sub-module whose sole purpose is to obtain the parameters of the X-Y spatial distribution for a given set of bacteria cultures is implemented. According to the requirements of the application, the following parameters are computed:
Area density, which is de ned as the number of morphological units appearing in a unit area of the selected region of interest.
58
Microbial cover, which is de ned as the ratio of the integrated area of cells to the total area of the selected region of interest.
Distance to nearest neighbor, which is de ned as the distances between a given cell and its nearest neighbor within the selected region of interest.
Degree of order of distribution, which is de ned as the standard deviation of distances to nearest neighbor. In applications, for a given set of bacteria images, their morphological feature values are rst computed with the above general-purpose image processing sub-modules. After that, the XY spatial distribution analysis sub-module can then be called to estimated the parameters of the X-Y spatial distribution of the given set of bacteria images. Figure 24 shows an example of such an X-Y spatial distribution analysis on a set of images.
4.7.2 Morphological Diversity Analysis In ecological applications, it is often required to compute the diversity indices for a given set of bacteria images. Although the de nitions of these diversity indices depend on speci c applications, the computation of these diversity indices are usually based on the morphological measure values and the classi cation results of the bacteria appearing in the given set of bacteria images. In our system, we have implemented a sub-module whose purpose is to compute the following diversity indices:
59
40
35
Frequency Distribution
30
25
20
15
10
5
0 0
10
20
30 40 50 Distance to Nearest Neighbour
60
70
80
(b) frequency distribution of DNN (a) input image Figure 24: X-Y spatial distribution analysis on a set of images. In this test, we have two sets of bacteria images: a control set and a comparative set both of which consist of 4 bacteria images. The solid curve in (b) corresponds to the frequency distribution of DNN of the control set and the dash dot curve corresponds to the frequency distribution of DNN of the comparative set.
Species Richness (d), which is de ned as: S?1 ; d = log( N)
(47)
where S is the number of morphological types and N is the number of individual cells appearing in the set of input images.
Shannon-Weaver Index of Diversity (H ), which is de ned as: C (N log N ? X n log n ); H=N i i
(48)
60 where C is a constant, N is the number of individual cells appearing in the set of input images, and ni is the number of individual cells of the ith morphological type appearing in the set of input images.
Evenness (e), which is de ned as: e = logHS ;
(49)
where S is the number of morphological types and H is the Shannon-Weaver index of diversity.
Equitability (J ), which is de ned as: J = HH ; max
(50)
where H is the Shannon-Weaver index of diversity and Hmax is the theoretical maximal Shannon-Weaver diversity index for the population examined, which assumes each morphological type has only one member. After all the morphological measure values of objects appearing in a given set of input images and their classi cation results are obtained with the above general-purpose sub-modules, the Morphological diversity analysis sub-module can then be invoked to compute these diversity indices.
61
5 Classi cation After all the valid objects in a segmented binary image have been identi ed, the next step is to classify these objects into di erent categories according to their salient features. In our system, two di erent sets of shape features are used. One is a set of regularity measures of a given shape, which are thought as being suitable for classi cation of bacteria cultures [10]. The other is a set of moment invariants which are among the most popular shape features in literatures [1, 8].
5.1 Regularity Measure In our system, a classi er based on two simple regularity measure is implemented. Although it can not handle all the classi cation problems, it is simple and works well in most of the cases. The rst regularity measure used in the classi er is the ratio of minimum eigenvalue and maximum eigenvalue of its covariance matrix, which is de ned above as thinness of the object. The second regularity measure is the circularity value. Figure 25 contains an example of the classi cation result on a bacteria image based on the two simple regularity measures.
5.2 Moment Invariant Moment invariants which are invariant to rotation, translation and scaling of an object have been widely used as shape features in the literature [8, 1] for their good performance in discriminating shapes and their low computational complexity. Let f(xi; yi); i = 1; : : : ; N g
62 denote the set of coordinates of the pixels belonging to an object. The (m; n)th order spatial moment of the object can be de ned as N X
M(m; n) = (xi)m(yi)n:
(51)
M(1; 0) mx = M (0; 0)
(52)
M(0; 1) my = M (0; 0)
(53)
i=1
The ratios
and
of the rst-order to the zero-order spatial moments de ne the centroid of the object. With centroid de ned, the (m; n)th order spatial central moment of the object can be de ned as N X
U(m; n) = (xi ? mx)m(yi ? my )n: i=1
(54)
Its normalized version with respect to size can be represented by
U(m; n) ; V(m; n) = (M (0; 0))
(55)
63 where =
m+n 2
+ 1. Based on the normalized spatial central moments of order 1, order 2
and order 3, a set of features of the object can then be de ned as follows:
h = V (2; 0) + V (0; 2);
(56)
1
h = (V (2; 0) ? V (0; 2)) + 4V (1; 1) ; 2
2
(57)
2
h = (V (3; 0) ? 3V (1; 2)) + (V (0; 3) ? 3V (2; 1)) ;
(58)
h = (V (3; 0) + V (1; 2)) + (V (0; 3) ? V (2; 1)) ;
(59)
2
3
4
2
2
2
h = (V (3; 0) ? 3V (1; 2))(V (3; 0) + V (1; 2)) 5
((V (3; 0) + V (1; 2)) ? 3(V (0; 3) + V (2; 1)) ) + 2
2
3(V (2; 1) ? V (0; 3))(V (3; 0) + V (2; 1)) (3(V (3; 0) + V (1; 2)) ? (V (0; 3) + V (2; 1)) ); 2
(60)
2
h = (V (2; 0) ? V (0; 2))((V (3; 0) + V (1; 2)) ? (V (3; 0) + V (1; 2)) ) + 2
6
2
4V (1; 1)(V (3; 0) + V (1; 2))(V (0; 3) + V (2; 1));
(61)
h = (3V (2; 1) ? V (0; 3))(V (3; 0) + V (1; 2)) 7
((V (3; 0) + V (1; 2)) ? 3(V (0; 3) + V (2; 1)) ) + 2
2
(3V (1; 2) ? V (3; 0))(V (0; 3) + V (2; 1)) (3(V (3; 0) + V (1; 2)) ? (V (0; 3) + V (2; 1)) ): 2
2
(62)
This set of features have been shown [8] to be invariant to translation, rotation and scaling of the object. It is suitable to design a classi er based on them.
64
(a) classi cation map (b) 3-class classi cation Figure 25: Classi cation results; (a) x coordinate represents eigenvalue ratio, y coordinate represents inverse of circularity; the region marked with blue color represents the distribution of feature values (eigenvalue ratio and inverse of circularity) of those bacteria which have long tails, the region with green color represents distribution of feature value of bacteria with relatively elongated shape, the region with red color represents the distribution of feature value of bacteria which are round. In our system, a user can interactively plot a speci c region and designate the category which it belongs to. (b) 3-class classi cation results. Given a set of features, a feature space can be constructed by viewing each feature as an axis. Every valid object of interest in a segmented binary image can then be represented as a single point in this feature space. Based on the distribution of labeled points in the feature space, a classi er can be designed. It partitions the feature space into several non-overlapping regions where each region represents a category. Each object whose feature point falls in a certain region is said to belong to the corresponding category. There is no way to know in advance which type of partition of the feature space is suitable for a given application. Therefore we have implemented a user-interface which allows a user to de ne his/her own partition of the feature space interactively. Figure 25 shows such an example of interactively
65 specifying the decision boundaries. Our image analysis system has been tested on a set of real images which consist of bacteria images obtained from two di erent types of feed pattern. Each set has 8 1280 950 images with 20 to 100 bacteria in each image. The purpose of the image analysis is to determine whether these two feed patterns have any ecological in uence on this bacteria culture, which is re ected by the percentage change of three di erent types of bacteria. Because of some undesired artifacts in the captured images, e.g. images have a number of bacteria which are not in focal plane, it is dicult to analyze all of them automatically. Therefore, some user interaction is necessary. Figure 26 shows a classi cation result on a real bacteria image. A summary of the results on the 16 di erent images which we have processed is shown in Table 1. It shows that our system can get a desired recognition rate for the given set of bacteria images. In most circumstances, our system can operate automatically. However, due to certain inherent nature of the given images such as, occlusion, it is hard for our system to operate automatically on all of them. Some user interaction has to be applied. Table 2 shows the classi cation results on the test images. Since some user interaction is involved here, it should not be surprising to see that our system can achieve a 100% recognition rate. Image set Number Average CPU time Recognition Cells Requiring User per image (seconds) Rate (%) Interaction (%) 1 8 79 92 8 2 8 67 91 9 Table 1: A Summary of Real Image Analysis
66
Figure 26: Classi cation results of a real bacteria image
True Classi ed as Category Methanotroph Hyphomicrobium Heterotrophs Methanotroph 100% 0 0 Hyphomicrobium 0 100% 0 Heterotrophs 0 0 100% Table 2: Classi cation results of test images
67
6 Interactive Environment As discussed above, the automatic image analysis can be abstracted as a sequential, feedforward process. A sequence of image processing modules is called upon during an image processing task. Each module takes the intermediate output of the previous module and transforms it into a representation which is suitable for the later modules based on the speci ed parameters, and the last module produces the nal results. In performing such an image processing task, a ow of information and control is required between the operator and the image analysis system. The operator needs information from the image analysis system with respect to its status and the image analysis system generally cannot function without information from the operator with respect to commands and parameters. Furthermore, there is an exchange of dialog in which the image analysis system prompts the operator for input, the operator enters a desired function, the image processing system responds with the feedback, and so on. This type of model imposes speci c requirements and constraints on the human-machine interface design of an image analysis system. The main problem here is to meet the requirements within the set of speci ed constraints. An analysis of the requirements and the environmental constraints of the image processing procedure is the rst step in describing the human-machine interface of the image processing system. The second step is to de ne a formal speci cation of the interaction of image analysis system that meets the requirements. The nal step, of course, is to implement the speci cation [17]. Generally, there are three types of interface models which are widely adopted [17]. They
68 are command language model, menu selection model, and direct manipulation model. In our implementation, we have analyzed the inherent nature of the three types of interface models and based on them have implemented a graphical user interface that is believed to be most suitable for the bacteria image analysis system. The earliest and most fundamental design of human-machine interface is the command language design. It involves a dialog between the operator and the image analysis system. The structure of the interaction is such that the commands which represent image processing routines are generated by the operator and executed by the image analysis system. The operator working under such a type of user interface has to translate his/her intent into a series of image processing commands and present them to the interface. The image analysis system receives these commands and interprets them. If they can be parsed, then the commands are executed and the results or response are displayed to the operator. If they can not be parsed, error messages are generally presented to the operator. The operator must then reformulate the commands or give up. In general, this type of interaction can be viewed as an operator-driven model of interaction. The operator initiates and terminates the dialog and is more likely to change the course of dialog than the image processing system itself. Only in the case of errors, warnings, and anomalies, the system seems to alter the course of dialog. The major disadvantage of such a type of interface design is that it requires the operator to memorize all the command syntax to apply it eciently in application. The other problem concerning this type of interface design is that it is extremely inecient for some types of interactions such as object boundary editing, which requires updating the
69 object's boundary directly. Although one can argue that command language human-machine interface design still has its advantages under some circumstances, we have abandoned it in our system. The user interface design of the bacteria image analysis system uses the other two types of interactive models, menu and direct manipulation.
6.1 Menu Design Menu selection has been an increasingly popular mode of human-machine interaction for a number of reasons. It reduces the memory and learning requirement of command languages; it can eliminate syntactic errors; it has the potential of structuring the interaction in such a way as to be congruent with the task structure; and it has the potential of reducing operatorkeystroking input. The basic idea behind menu selection model can be characterized as a script like that of selecting items from a restaurant menu. At each point of interaction, the system presents a menu of options to the operator; the operator scans the list of the options and selects the desired option via some input device; and the system performs the function associated with the selected option. Menu selection inherently involves ow of information and control back and forth across the interface. It, to some extent, re ects the abstract model of image processing procedure. In our system, we integrate all the image processing routines into a menu system with a tree structure which is naturally consistent with the requirements of image processing tasks. The rst level of the menu tree in the bacteria image analysis system consists of six items associated with the six image processing modules implemented in the system. They are
70 the le management module, the edit module, the preprocessing module, the segmentation module, the measurement module, and the classi cation module. These modules in turn consist of submenus corresponding to the routines in the module. The le management includes all the routines related to image le and I/O device manipulation. The edit module incorporates all the editing tools as its menu items. The preprocessing module consists of di erent types of general image processing routines and is organized as a tree structure with each inner node as a submenu and each leaf node as a routine. The segmentation module includes all the items related to image segmentation routines. The measurement module provides a submenu for di erent parameter estimation and postprocessing. The classi cation module is the submenu that includes all the classi cation routines implemented in the image analysis system. With this hierarchical tree structure interface design, operators can easily go through the menu tree of the image analysis system to nd the desired routines and execute them without memorizing or learning the names and syntax of the commands. This approach is especially sound for computer nonspecialists. Figure 27 shows the rst level of the menu tree of the bacteria image analysis system.
6.2 Tool Design The direct manipulation model is a movement-oriented interaction model which allows a seemingly continuous movement of objects [17]. It is directly related to graphical applications and is possible only with a graphical representation. It has the following features:
Continuous representation of objects of interest;
71 Menu Tree
File
Edit
Process
Segment
Measure
Classify
Figure 27: The rst level of the menu tree of the bacteria image analysis system.
Physical actions such as selection, dragging, or clicking are used to a ect the objects; Operations are rapid, incremental, and reversible and their impact on the objects of interest is immediately visible. In direct manipulation, the interface object models are supported by the operator's knowledge of space, geometry, movement, and the characteristics of familiar objects. The operator formulates an action which is interpreted by the system; if the action is legitimate, the corresponding operation is performed, otherwise some direct feedback is sent back to the operator. This model is consistent with the requirements of interactive editing of an image analysis system. We have implemented all the interactive editing tools by using such an interactive model in the bacteria image analysis system. This provides operators with the most ecient method to interactively change input images and output images. For example, the implementation of an interactive object editor with direct manipulation interaction model enables operators to delete, undelete, draw, erase, rotate, move, and zoom objects directly, which is the most ecient way to operate on objects.
72
6.3 Implementation The human-machine interface of the bacteria image analysis system described above is implemented under X/Motif toolkit in C language. Figure 28 shows a snapshot of the graphical user interface of the bacteria image analysis system.
7 Summary In this report, we have introduced an application-oriented image analysis system. This image analysis system is proposed for automatic/semi-automatic analysis of bacteria images provided by the Center for Microbial Ecology, Michigan State University. In an application-oriented image analysis system, the algorithms must be both robust in performance and ecient in computation. On the other hand, because the currently available image processing algorithms can not always achieve good performance on di erent types of images, the design of an image analysis system has to be application-oriented and must include interactive editing tools for images that are dicult to process with automatic algorithms. In the implementation of the bacteria image analysis system, we have spent much of our e ort on the e ectiveness and eciency of the image processing algorithms used in the system. At the same time, an interactive tool is carefully designed to meet the requirements of interactive editing when automatic algorithms fail to produce expected results. The most important stage in an image analysis system is the segmentation module. More
73
Figure 28: A snapshot of Graphic User Interface.
74 than half of the total user interaction in an image processing cycle takes place here. The more ecient this module, the more ecient is the complete image analysis system. Reducing user interactions and making the segmentation module completely automatic is one of the ultimate goals of an image analysis system. We are currently focusing on designing more ecient and more e ective segmentation algorithms. Another diculty in image analysis system design is the implementation of the classi cation module. Given a set of object classes, selecting a set of features that are optimum in di erentiating the given classes is still a dicult problem in pattern recognition. We believe that we can nd a set of features that can eciently di erentiate di erent types of bacteria. We are exploring a variety of other shape features for this purpose.
Acknowledgements We greatlly acknowledge the supports of Professor James Tiedje and Professor Craig Criddle.
References [1] A. Aluzek, \Identi cation and Inspection of 2-D Objects Using New Moment-based Shape Descriptors," Pattern Recognition Letter, Vol. 16, 687-697, 1988. [2] A. A. Amini, S. Tehrani, and T.E. Weymouth, \Using Dynamic Programming for Minimizing the Energy of Active Contours in the Presence of Hard Constraints," IEEE Proc. Comput. Vision Patt. Recogn. Conf., 592-595, Ann Arbor, 1988.
75 [3] J. Canny, \A Computational Approach to Edge Detection," IEEE Trans. Patt. Anal. Machine Intell. Vol. 8, 679-698, Aug. 1986.
[4] C. H. Chen, L. F. Pau, and P. S. P. Wang, Handbook of Pattern Recognition and Computer Vision, World Scienti c Publishing, 1993. [5] M.-P. Dubuisson, A. K. Jain, and M. K. Jain \Segmentation and Classi cation of Bacteria Culture Images," Journal of Microbiological Methods, Vol. 19, 279-295, 1993. [6] R. Haralick and L. Shapiro, \Image segmentation techniques," Computer Vision, Graphics and Image Processing, Vol. 29, 100-132, 1985.
[7] D. Heller, Motif Programming Manual, O'Reilly & Associates, 1991. [8] M. K. Hu, \Visual Pattern Recognition by Moment Invariants," IRE Trans. Inform. Theory, Vol. 8, 179-187, 1962.
[9] L.-K. Huang and M.-J. J. Wang, \Image Thresholding by Minimizing the Measures of Fuzziness," Pattern Recognition, Vol. 28, 53-58, 1995. [10] A. K. Jain and M.-P. Dubuisson, \Segmentation o X-ray and C-scan Images of Fiber Reinforced Composite Materials," Pattern Recognition, Vol. 25, 257-270, 1992. [11] A. K. Jain, \Pattern Recognition," in International Encyclopedia of Robotics, R. C. Dorf Ed. John Wiley, 1052-1063, 1988. [12] A. K. Jain, S. P. Smith, and E. Backer, \Segmentation of Muscle Cell Pictures: A Preliminary Study," IEEE Trans. Patt. Anal. Machine Intell., Vol.2, 232-242, Feb. 1980.
76 [13] M. Kass, A. Witkin, and D. Terzopoulos, \Snakes: Active Contour Models," Proc. First IEEE Int. Conf. Comput. Vision, 259-268, London, 1987.
[14] F. Leymarie and Martin D. Levine, \Tracking Deformable Objects in the Plane Using an Active Contour Model", IEEE Trans. Pattern Anal. Machine Intell., Vol. 15, 617-634, June, 1993. [15] N. J. Naccache and Rajjan Shinghal, \An Investigation into the Skeletonization Approach of Hilditch", Pattern Recognition Vol. 17, 279-284, 1984. [16] W. Niblack, An Introduction to Digital Image Processing, Prentice Hall, 1986. [17] K. L. Norman, \Models of the Mind and Machine: Information Flow and Control between Humans and Computers," Advances in Computers, Vol. 32, M. C. Yovits (Ed.) Academic Press, 1994. [18] A. Rosenfeld and A. C. Kak, Digital Picture Processing, Academic Press, 1982. [19] P. K. Sahoo, S. Soltani, and A. K. C. Wong, \A survey of thresholding techniques," Comput. Vision, Graphics and Image Processing, Vol. 41, 233-260, 1988.
[20] T. Taxt, P. J. Flynn, and A. K. Jain, \Segmentation of document images," IEEE Trans. Patt. Anal. Machine Intell., Vol. 11, 1322-1329, Nov. 1989.
[21] Theo. Pavlidis, Algorithms for graphics and image processing, Computer Science Press, 1982.
77 [22] O. D. Trier and A. K. Jain, \Goal-Directed Evaluation of Binarization Methods," to appear in IEEE Trans. Patt. Anal. Machine Intell., 1995.
