MEDINFO 2004 M. Fieschi et al. (Eds) Amsterdam: IOS Press © 2004 IMIA. All rights reserved
Automatic segmentation of digital micrographs: A survey Tim W. Nattkemper Applied Neuroinformatics Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany
Tim W. Nattkemper
ter imaging approaches. In general, multiparameter imaging approaches are applied to cell samples from different specimens, in certain stress conditions, or with different pharmaceutical treatment to get comprehensive descriptions of spatial and temporal molecular dynamics in the cell [1].
Abstract Digital micrographs and play a crucial role in today’s biomedical research. Due to progresses in experiment standardization and automation large sets of digital microscopy images, so called micrographs are recorded and stored to databases. The subsequent analysis of the large number of digital images needs the image information to be transformed into quantitative data, which can be processed by statistical methods and datamining. This article summarizes applications of optical microscopy in biomedical research and describes the individual characteristics in micrograph segmentation and classification. An overview on past works based on image processing and artificial neural networks is given and the problem of segmentation evaluation. It concludes with recommendations for future works.
Today we observe a growing world-wide attention to biological cells as complex functional units that act in networks and the call for new multidisciplinary approaches for modeling and understanding cells and cellular networks, employing biology, biophysics and mathematics [9]. This development is fostered by the fact, that in pharmaceutical research the number of target structures, proposed by non-microscopy based, HTS approaches like microarrays and gels grows rapidly. Those need to be evaluated, e.g. in context of toxicology. Thus, microscopy based studies on the molecular level of living cells, are appreciated to be of substantial value. The basic aspect of optical microscopy approaches in biomedicine is that the information about spatial correlations between the mapped channels is preserved, a feature not owned by methods like e.g. gel-electrophoresis or microarrays. In parallel, the simultaneous local measurement of different cellular parameters shall culminate in the development of integrated models of cell functions. The simulation and the graphical display of such models in virtual cells are corner stones in the rapid evolving field of systems biology [10]. Without a doupt, model evaluation and verification strongly depends on the comprehensive analysis of large sets of microscopy images.
Keywords: Microscopy, Histological Techniques, Biomedical Engineering, Artificial Neural Networks, Image Processing, Reproducibility of Results
Introduction In the past decade, quantitative biomedical research on the molecular level has been extended from in-vivo studies to studies on living cells and tissue [1]. Due to advances in preparation techniques and design of antibody markers, the identification of molecules by selective visualization has improved, now enabling quantitave studies on a large scale. This has caused a new perspective on optical microscopy as a tool for quantitative studies and started a renaissance in this field [2-4]. Through the application of modern robotics and personal computers these visualization advances have been followed by additional ones in experiment automation and protocol standardization. Thus, the visualization-based molecular identification can now be applied for many specimens and makers on a large scale, i.e. for a large number of specimens with constant quality, which is frequently called high-throughput screening (HTS). In addition, advanced staining techniques and multiband imaging allow simultaneous visualization of macromolecules and compartments in the cells [5-7]. In multimodal imaging approaches, different imaging techniques are merged to link additional structural information to the cellular parameters [8].
These interests in multiparameter imaging on the cellular level results in a demand for algorithms for the (semi-)automatic analysis of digital microscopy data, so called micrographs. In HTS studies, the micrographs are stored in a database and are to be analyzed by (a) direct manual exploration (generally by visual inspection) and/or (b) statistical analysis, resp. datamining. To perform (b), quantitative data has to be extracted from the images. To this end, it is inevitable to perform image segmentation interactively or full automatically. Because large sets of micrographs are to be evaluated, an automation of the segmentation step to some extend is necessary to care for the reproducibility of results, barring the problem of heavy work load for expensive human experts. Since the segmentation of digital micrographs can thus be ranked as being substantial for microscopy approaches in the fields from pharmacy to molecular biology, this survey summarizes the fundamental problems in micrograph segmentation and gives an overview on the past works. It dis-
In this paper we summarize all the above approaches that are based applying microscopy with different parameters and/or in combination with different imaging techniques as multiparame-
847
TW. Nattkemper
cusses the problem of system evaluation and concludes with some recommendations regarding about the future of this field.
Also the imaging framework itself can be standardized only to a limited extend, because the visualization of microbiological structures is a field of permanent progress.
Segmentation of micrographs: fundamental problems
Consequently the task of automatic micrograph segmentation is generally ranked as such a demanding one, that even the development of interactive systems that support the human experts in the manual classification or segmentation of images are considered to be of substantial help (see for example [13-16]).
The segmentation task in micrographs is described as a pixel labeling problem. Each image pixel has to be assigned to a class label L(x,y). In the most simple application, the entire content of the image is classified to contain a biologically significant structure or not, i.e. the entire image is labeled by 1 (“contains object”) or 0 (“otherwise”). In the majority of more complex applications, n biological objects have to be separated from the background and from each other:
Image processing approaches Most of the work in biomedical image analysis is concerned with the segmentation and visualization of macrobiological structures, like tissue, organs and bones for example. Comprehensive overviews of this field are regularly published and discussed as in [17][18][19].
0, if (x,y) belongs to background L(x) = i, if (x,y) is covered by object i (i=1,…,n).
Compared to this, approaches for the analysis of digital micrographs from microbiological domains are less published. One reason for this is the large diversity in the image domains of microbiological studies, which is rooted in the complexity of cell functions, resulting from the overwhelming diversity of expressed molecules and their relationships. Also the publications are so wide-spread in literature, i.e. through the fields of microscopy, biomedical engineering, biomedical imaging, bioinformatics and pattern recognition that it is impossible to get a comprehensive overview about the works in this field. Altogether, microbiological signal processing has not made its way into a standard diagnostic practice like “classic” medical imaging that usually considers imaging sonography-, radiology- or magnet resonance imaging-based imaging applied to macrobiological structures. So the interests, efforts and funding into the developments of evaluation software for micrographs is still small compared to the above listed ``classic'' medical imaging domains.
Note, that the label L(x,y) can be •
the object class (0 = background, 1=covered by cell),
•
the index of different objects from the same class (which is for example the case, if the positions and bodies of objects from one class in the image is the basic biological quantitative parameter in the image),
•
the index of different objetcs from different classes (if for example two different kinds of cells have to be identified). In this case the class is subsequently assigned by a classifier or by expert user interaction.
The labeling of micrograph pixels is hampered by several problems, some are specific to this particular image domain [11] [12]. At least one of the following problems has to be accomplished: •
inhomogeneous illumination across the visual field
•
occlusion of objects
•
considerable variation of object shape and/or size and/or orientation
•
considerable variation of the signal intensity of objects from the same class, caused for example by inconsistent staining
One large fraction of algorithms for a micrograph segmentation can be summarized as model-based approaches. A large subset of those are based on circle or ellipsoid detection objective Hough-transforms [20] [21]. Hough-transforms are based on the assumption that the shape of the target objects has a limited number of descriptive parameters (for example radius, orientation). The problem in applying the Hough transform is that (a) the object shapes and sizes show such variation, that an efficient postprocessing of any Hough-accumulator appears to be infeasible, (b) an edge map of high quality needs to be computed in advance and (c) the shape model of the objects is hard to define. For the same reason, wave-propagation algorithms [22] seem not applicable to many micrograph segmentation problems. Recently, some works have been published to learn hough transform parameters for arbitrary shapes from a set of labeled data [23]. This interesting approach contains some work load, because the algorithms have to be provided with some hand labeled data set.
Nevertheless, even if these problems have special characteristics in the domain of biomedical imaging, they have been observed in related occurrence and discussed in the computer vision community as fundamental problems. In industrial computer vision applications, these can be compensated by employing certain heuristics (about size, shape and color of the objects) or by tuning the imaging framework (the light sources, for instance), that the images are of constant quality. In optical microscopy, the problems are often more difficult to solve, because much less heuristics can be made about the signal intensities or object shapes. This feature separates the field of microbioligical imaging from most of the classic fields of medical imaging, where a priori knowledge about anatomical structures can be used to improve segmentation and visualization of macrobioligical structures.
Another model-based approach for grouping image primitives of convex objects is presented in [24]. Step edges, roof edges and concave corners are detected by pre-processing and a set of initial hypotheses for convex groups is formulated. The search space of hypothesis is explored for such groups using dynamic
848
TW. Nattkemper
acter of ANNs. Which is on second thought unlikely, because in the field of single cell image classifiction, successful applications of ANNs are reported regularly [34-37].
programming and minimization of a cost function. The main disadvantage of this method is that again certain assumptions on the cell shape have to be made. For example, a concave corner in the image is strictly interpreted as an indication for a gap between two adjacent cells where the two convex shapes of the two cells meet. Therefore, small local concave distortions in the cell shape lead to undesired grouping results of segmenting one cell with a small concavity into two convex groups because all cells are assumed to be of full convex shape.
For the automatic segmentation of micrographs by ANNs the segmentation problem is transformed into a strict classification set-up. The classifier is provided by a training set T={(x,y)} of input features x and output values y. In a direct transformation, y is the pixel label i and the input x is a pixel value or a collection ones across a pixel neighborhood and the trained classifier realizes the labeling function L(x). If an adaptation of the system to changing experimental parameters is necessary a new training set has to be provided.
Another class of algorithms perform segmentation by combining morphological operators and grey value thresholding as proposed in the works by [25--28]. A grey value threshold is often found by the automated analysis of the grey value histogram. After the image is binarized, morphological operators are applied for closing gaps in the object or to separate two objects which stick together. In these approaches, the model is not explicitly described in geometrical terms but implicitly in the sequence of morphological operators applied. For an efficient application of these methods, the target objects are not clumped together to dense clusters. Hence, these methods are not applicable to invivo or in-situ imaging approaches.
In recent years, kernel based methods have been object of much research effort and gained remarkable popularity in the field supervised ML. The most prominent algorithm among these is the support vector machine (SVM) proposed by V.Vapnik [38] for binary classification. Nevertheless, SVMs have not been considered in the context of micrograph segmentation, since in the field of biomedical informatics, MLPs are still the most widely used supervised-learning architecture [39] and is reported to reach performances comparable to the SVM. Another reason might be the fact, that SVMs have a problem with output value normalization in case of unbalanced training sets, which is usually the case in object classification problems, where a restricted class of positive cases has to be distinguishes from a much larger class of negative cases.
Another group of algorithms are so called flooding schemes. In these works, a grey valued micrograph is regarded as mountains and valleys of grey values. Starting at the lowest valleys, the landscape is flooded and two lakes merge if they reach a common ridge, i.e. the edge of an object. These algorithms were successfully applied to the segmentation of electrophoresis gel images [29] and tissue segmentation [30]. A crucial parameter is the minimum ridge height at which a merging is accepted as a boundary of an object. This would be difficult to set for example in fluorescence micrographs because of the non-homogenous fluorescence signals across the cell.
In the application and evaluation of machine learning based approaches one needs to be supplied with the training set mentioned above, which often constrained by the limited time of human experts and the problem of error-prone hand labeling. Moreover, one has to define a cost function to assess the classification, i.e. segmentation accuracy, as a prerequisite to automatic parameter optimization.
Since the segmentation of micrographs is such a challenging task, that needs expert knowledge for tuning and optimizing, the question needs to be addressed how to incorporate the expert knowledge in the algorithm. As also pointed out in [31], the increase of automatically generated data of constant quality in biomedicine is overshadowed by a strong limitation of experts that could evaluate the data, using their experience, and thus contribute to tuning the algorithms. This motivates application of learning based methods from the field of artificial neural networks.
System evaluation and benchmark problems As well as for analyzing the evaluation performance of human experts, as for inventing and optimizing an efficient image processing system in biomedicine, the accuracy of an evaluation has to be defined, with respect to specific needs. So the first question is, how to define the index for the accuracy. In the literature many index terms like sensitivity, specificity, efficiency, accuracy, utility, value, worth, effectiveness, usefulness, positive/negative predictive value and likelihood ratio, to mention the most frequent, are listed. Those are based on the comparison of some evaluation with a referential correct evaluation, the gold standard. To this end, complementary pairs of terms (for example sensitivity against specificity) are plotted as so called Receiver-Operator-Characteristics (ROC). ROC are based on statistical decision theory and were introduced in the 1950's in the context of electronic signal detection and interpretation problems with radar [40]. Later, the ROC were applied also to medical imaging [41,42] and pap smear analysis [43]. The calculation of ROC for accuracy measurement is so widely used within the field of biomedical engineering, that its application has become a world wide standard. As it was also stated earlier in [44], the fields of Biomedical Imaging and Medical Informatics have both undergone a rapid growth in the past, but lacked a certain kind
In the past 20 years artificial neural networks (ANN) and machine learning (ML) have made their way from fundamental research to applicative engineering and they provide a well-suited framework for incorporating primary expert knowledge into the adaptation of algorithms. Nevertheless, in the field of micrograph analysis, image processing algorithms based on ANNs are still exceptional. In [32] a multi-layer perceptron (MLP) is used for the detection of cells encapsulated in a semipermeable and immunoprotective polymer. In [33] a system for the pre-selection of immunolabeled neurons is proposed. Even if the segmentation process in this work does not utilize an ANN, it is listed here because it incorporates a trainable classifier in a subsequent classification step. One reason for this low number of published works about ANNs in microbiological image processing might be an animosity of the users in this application field concerning the black box char-
849
TW. Nattkemper
of discussion in between. In fact, this is changing right now, and ROC measurements have been applied to analyze the classification performance notably (see for example [45]). For a review on ROC the reader may refer to [46,47]. The evaluation and discussion of new approaches is often based on the application on benchmark datasets. A comprehensive collection of such sets can be found at the Computer1 Vision Homepage. Most of these image data sets are recorded in an experimental set-up, showing faces and toys. A limited number of data sets show synthetical data for example to study texture, motion. In the field of biomedical image processing, benchmark data sets concentrate on macrobiological structur2es like mammorams, gastrointestin3al video endoscopy and magnet reso4nance imaging (MRI). The lack microbiological images is caused by several reasons: First, the imaging apparatus are not integrated standardized products like in the macrobiological domain. Thus, it is truly awkward to find one representative data set. Second, the biomedical researchers in this field can share their micrographs easily with others because of competition and proprietary rights problems. Above all the last one, if the images content is settled in the field of drug discovery. Additionally, the dataset must be labeled with a ground truth i.e. a numerical expression of the correct evaluation result.
References [1] Taylor DL, Woo ES, Giuliano KA. Real-time molecular and cellular analysis: the new frontier of drug discovery. Curr Op Biotech 2001: 12: 75-81. [2] Taylor DL, Nederlof M, Lanni F, Waggoner AS. The new vision of light microscopy. Am Sci 1992: 90(4): 322-335 [3] Lichtman JW. Confocal microscopy. Sci Am 1994: 272 (2): 40-45. [4] Smith RF. Microscopy and Photomicrography. CRC Press 1994. [5] Boland MV, Markey MK, Murphy RF. Automated recognition of patterns characteristic of subcellular structures in fluorescence microscopy images. Cytometry. 1998 Nov 1;33(3):366-75. [6] Oksvold MP, Skarpen E, Widerberg J, Huitfeldt HS. Fluorescent histochemical techniques for analysis of intracellular signaling. J of Histochem & Cytochem. 2002: 50 (30): 289303. [7] Schubert W. Topological Proteomics, Toponomics, MELKTechnology. In: Hecker M, Mullner S, eds. Proteomics of Microorganismus: Fundamental Aspects and Application, Advances in Biochemical Engineering/Biotechnology. Springer. 2003 (in press)
For example, in case of a segmentation task, this would consist of a label image. In case of an object detection task, it would be a list of correct referential object positions. In the common case, such a ground truth is not available and must be simulated by a manual data evaluation by a human expert. But it is a well known fact, that in biomedical imaging, such human evaluations are expensive and error-prone [13][32][48-51]. The processing of synthetical micrographs (or phantom images) may be a promising loophole.
[8] Wind RA, Minard KR, Holtom GR, Majors PD, Ackerman EJ, Colson SD, Cory DG, Daly DS, Ellis PD, Metting NF, Parkinson CI, Price JM, Tang XW. An integrated confocal and magnetic resonance microscope for cellular research. J Magn Reson. 2000 Dec;147(2):371-7. [9] Alberts B. The cell as a collection of protein machines: preparing the next generation of molecular biologists. Cell. 1998:92(3):291-294.
Future trends
[10]Kitano H. Computational systems biology. Nature. 2002 Nov 14;420(6912):206-10.
The field of micrograph processing is characterized by typical problems, which are (i) the diversity in imaging approaches, (ii) the frequent lack of a priori knowledge about the image content, (iii) the lack for common evaluation frameworks or benchmark problems and (iv) a large number of scientific disciplines involved. (i)-(iii) separates the problem of microbiological image processing from most of the works reported in the field of medical imaging. It seems apparent, that substantial progress in this field would be accelerated, if researchers of this field would build a community that formulates fundamental problems across single particular applications and propose terms for system performance and accuracy. This, together with an increased exchange of experience and (probably more important) data is considered to be the best framework for future progress in this field.
[11]Nazeran H, Rice F, Moran W, Skinner J. Biomedical image processing in pathology: a review. Australas Phys Eng Sci Med. 1995 Mar;18(1):26-38. [12]Ong SH, Jin XC, Jayasooriah, Sinniah R. Image analysis of tissue sections. Comput Biol Med. 1996 May;26(3):269-79. [13]Baumann I, Nenninger R, Harms H, Zwierzina H, Wilms K, Feller AC, Meulen VT, Muller-Hermelink HK. Image analysis detects lineage-specific morphologic markers in leukemic blast cells. Am J Clin Path. 1995; 105: 23-30. [14]Tagare HD, Jaffe CC, Duncan J. Medical Image Database: A content-based retrieval approach. J Am Med Inf Assoc. 1997;4:184-198. [15]Souchier C, French M, Benchaib M, Catallo R, Bryon PA. Methods for Cell Proliferation Analysis by Fluorescent Image. Cytometry. 1995; 20:203-9. [16]Comaniciu D, Meer, P.,Foran DJ. Image-guided decision support system for pathology. Mach Vis Appl. 1999; 11:213224.
1. http://www-2.cs.cmu.edu/~cil/v-images.html 2. http://marathon.csee.usf.edu/Mammography/Database.html 3. http://www.gastrointestinalatlas.com 4. http://www.nlm.nih.gov/research/visible/visible\_human.html
850
TW. Nattkemper
phocytes in tissue sections. IEEE Trans Inf Technol Biomed. 2001 Jun;5(2):138-49.
[17]Dhawan AP. A review on biomedical image processing and future trends. Comput Methods Programs Biomed. 1990;Mar-Apr;31(3-4):141-83. [18]Duncan JS, Ayache N. Medical Image Analysis: Progress over Two Decades and the Challenges Ahead. IEEE PAMI. 2000;22(1): 85-105.
[37]Boland MV, Murphy RF. A neural network classifier capable of recognizing the patterns of all major subcellular structures in fluorescence microscope images of HeLa cells. Bioinformatics. 2001 Dec;17(12):1213-23.
[19]Pham DL, Xu C, Prince JL. Current methods in medical image segmentation. Annu Rev Biomed Eng. 2000;2:315-37.
[38]Vapnik V. The Nature of Statistical Learning Theory. Springer. 1995.
[20]Ballard DH. Generalizing the Hough transform to detect arbitrary shapes. Patt Rec. 1981; 13:111—21.
[39]Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med. 2001 Aug;23(1):89-109.
[21]Gerig G, Klein F. Fast contour identification through efficient Hough transform and simplified interpretation strategy. Proc ICPR .1986;8:498-500.
[40]Metz CE, ROC methodology in radiologic imaging. Investigative Radiology. 1986;21:720-33.
[22]Hanahara K, Hiyane M. A Circle-detection algorithm simulating wave propagation. Mach Vis Appl. 1990;3:97-111.
[41]Lusted LB. Signal detectability and medical decision-making. Science. 1971 Mar 26;171(977):1217-9.
[23]Brejl M, Sonka M. Object localization and border detection criteria design in edge based image segmentation: automated learning from examples. IEEE TMI. 2000;19(10):973-98.
[42]Lusted LB. Decision-making studies in patient management. N Engl J Med. 1971 Feb 25;284(8):416-24.
[24]Cong, G. and Parvin, B. Model-based segmentation of cell nuclei. Patt Rec 2000;33(8):1383-93.
[44]Shiffman S, Shortliffe EH. Biomedical imaging and the evolution of medical informatics Comput Med Imaging Graph. 1996 Jul-Aug;20(4):189-92.
[43]Lusted LB. ROC recollected. Medical Decision Making.1984;4:131-135.
[25]Talbot, H. and Villalobos, I. Binary image segmentation using weighted skeletons. SPIE Im Alg Morph Im Proc. 1992;1769:393-403. [26] Sonka M, Hlavac V, Boyle R. Image Processing Analysis and Machine Vision. Chapman & Hall, London. 1995.
[45]Markey MK, Lo JY, Vargas-Voracek R, Tourassi GD, Floyd CE Jr. Perceptron error surface analysis: a case study in breast cancer diagnosis. Comput Biol Med. 2002 Mar;32(2):99-109.
[27] Demandolx D, Davoust J. Multiparameter image cytometry: from confocal micrographs to subcellular fluorograms. Bioimaging.1997;4:159-169.
[46]Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993 Apr;39(4):561-77.
[28]Nedzved A, Ablameyko S, Pitas I. Morphological segmentation of histology cell images ICPR. 2000; 1:500-3.
[47]Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med. 2000 May 30;45(1-2):2341.
[29]Meyer F, Beucher S. Morphological Segmentation. J. Vis Comm Im Repr. 1990;1(1): 21-46.
[48]Ackerman AB. Discordance among expert pathologists in diagnosis of melanocytic neoplasms. Hum Pathol. 1996 Nov;27(11):1115-6.
[30]Malpica N, de Solorzano CO, Vaquero JJ,Santos A, Vallcorba I,Garcia-Sagredo JM, del Pozo F. Applying watershed algorithms to the segmentation of clustered nuclei. Cytometry. 1997 Aug 1 ; 28(4): 289-297.
[49]Debeir O, Decaestecker C, Pasteels JL, Salmon I, Kiss R, Van Ham P. Computer-assisted analysis of epiluminescence microscopy images of pigmented skin lesions. Cytometry. 1999 Dec 1;37(4):255-66.
[31]Dybowski R. Neural computation in medicine: perspectives and prospects. Proc. ANNIMAB-1. 2000; pp. 27-36. [32]Sjostrom PJ, Frydel BR, Wahlberg LU. Artificial neural network-aided image analysis system for cell counting. Cytometry. 1999 May 1;36(1):18-26.
[50]Viedma Contreras JA. Leucocyte activation markers in clinical practice. Clin Chem Lab Med. 1999 Jun;37(6):60722.
[33]Hibbard LS, McCasland JS, Brunstrom JE, Pearlman AL. J Microsc. 1996 Sep;183 ( Pt 3):241-56.
[51]Nattkemper TW, Twellmann T, Ritter H, Schubert W. Human vs machine: evaluation of fluorescence micrographs. Comput Biol Med. 2003 Jan;33(1):31-43.
[34]Schnorrenberg F, Pattichis CS, Kyriacou KC, Schizas CN. Computer-aided detection of breast cancer nuclei. IEEE Trans Inf Technol Biomed. 1997 Jun;1(2):128-40.
Address for correspondence
[35]Shang C, Daly C,McGrath J. Neural network based classification of cell images via estimation of fractal dimensions. In: Proc. of ANNIMAB-1. 2000; pp. 111-116.
Tim W. Nattkemper, Applied Neuroinformatics Group - Faculty of Technoloy, Bielefeld University, PO Box 100131, D-33501 Bielefeld, Germany
[36]Nattkemper TW, Ritter HJ, Schubert W. A neural classifier enabling high-throughput topological analysis of lym-
[email protected] phone: ++49-(0)521-1066059 fax:++49-(0)521-1066011 email:
851
TW. Nattkemper
852
