Nonlinear filtering in improving the image quality of confocal ...

Machine Vision and Applications (1991) 4:243-253

Machine Vision and Applications 9 1991Springer-VerlagNew YorkInc.

Nonlinear Filtering in Improving the Image Quality of Confocal Fluorescent Images Pekka H~inninen and Ernst H.K. Stelzer European MoleculatBiologyLaboratory(EMBL), Heidelberg, FRG Juha Salo Wallac Oy, Turku, Finland

Abstract. A different way of processing confocally scanned fluorescence images is presented. Linear median hybrid methods and linear filtering methods are compared numerically with a conventionally processed artificial data set and with real confocal data. The use of linear median hybrid techniques reduces the time required for recording three-dimensional data sets with a confocal fluorescence microscope as well as the photo-damage to the biological sample. The implementation of a linear median algorithm on the hardware level of a confocal microscope is discussed.

Key Words: fluorescence imaging, confocal, median filtering, biomedical microscopy

1

Introduction

Fluorescence light microscopy is an efficient way of studying organelles in mammalian cells. With improved biochemical labeling techniquues and modern optical setups such as confocal microscopes, studies of biological objects have become easier. During the past few years the number of applications of fluorescence microscopy has grown rapidly (Shotton 1989). Fluorescence microscopy observes only a certain fluorescent dye that in cell biology is either a single molecule or a molecule attached to an antibody. In either case it will be specific for one target and label only this target. No other part in the cell is

Address reprint requests to: Pekka H~nninen, European Moleculat BiologyLaboratory(EMBL), Meyerhofstrasse 1, Postfach 10.2209, D-6900 Heidelberg, FRG.

usually labeled, and hence the contrast is very high (White et al. 1987); the segmentation problem encountered in every day life is solved biochemically. As observers we know that to look for and in most cases have sufficient a p r i o r i knowledge concerning the structure of the target. The advantage of confocal fluorescence microscopy over conventional fluorescence microscopy is (1) an improved lateral resolution and (2) a discrimination against out-of-focus contributions (Wilson et al. 1984, Pawley 1989). Confocal fluorescence micrographs have a higher contrast and are a natural step toward three-dimensional light microscopy. In most cases a stack of images can be recorded at different focal levels and used to reconstruct the object! Although fluorescence contrast imaging is an excellent tool for studying specific structures in biological organelles, it has some serious drawbacks in comparison to reflection or transmission microscopy: The low-level fluorescence emission is Poisson distributed, adding a signal dependent noise component, which becomes dominating in many applications of low-level excitation and emission. This phenomenon is especially disturbing when the digitally recorded images are to be processed further. Improved signal-to-noise ratios (SNR) are achieved by either higher illumination levels or long integration times in the detector device. Both these techniques have their drawbacks since (1) high illumination levels often damage the fluorescent dye and disturb the actual sample and (2) long integration times slow down the process of recording as well as distort the recorded signal of dynamic events. Also, long integration times even at low illumination levels cause photo-damage to the sample; the damage is a function of the illumination energy rather than a

244

P. H/inninen et al.: NonlinearFiltering

function of illumination intensity (Tsien and Waggoner 1989).

1.1 Fluorescence Imaging In a formal liiaear approach to light microscopy the spatial intensity distribution in the image plane I(i) depends on the distribution of the fluorophore in the sample F(is), a transfer function H(R, is), and a noise component N(R). I(i) = F(is) • H(i, i~) + N(i)

(1)

However, the noise component N(x) has terms that also depend on the object and on the circumstances under which the data are recorded. It is therefore appropriate to write I(~) =/~(xs) • H(~, ~) + N~(F, H, ~) + N0(~) (2) In this paper we assume operation in the low light limit so that N(F, H, x) is dominated by the Poisson noise. In our case the Poisson distribution defines the point probabilities of the detection of a photon count rate k with an expectation value ~: Pk=~.Te x

(3)

The expectation value ~ is at the same time the variance of the Poisson distributed signal. The noise power (rms) of a Poisson distributed signal can be expressed as N = X/S,

where S is the signal power.

(4)

Equations (3) and (4) show that with low photon count rates the signal dependent noise component dominates. When observing a fluorescent sample through the ocular, the Poisson distribution does not present such a large problem as it does in a digital analysis of the scene. During the observation the eye will not recognize the presence of noise since the noise frequency spectrum lies well beyond our perception. To a certain extent the human eye can also correct the noise in digitized images whereas it is difficult to do it digitally. The problem presented by Eq. (3) can be overcome by the same means as the human observer does--by integrating the signal in time and thus increasing the photon count rate expectation value X: S(x, y) =

s(x, y, t) dt

(5)

Increasing the expectation value ~ by means of increasing the excitation energy by either longer excitation times or higher excitation intensities is not always possible: In many cases the biological sample itself and the fluorescent dye in it will not last the higher illumination energies but will break down. This phenomenon, photo-bleaching, is in some applications so fast that the sample may not be observed directly through the ocular in fluorescence. Also, one must note that the fluorescent dye can be easily saturated with higher illumination levels, and thus only increasing the noise component (Tsien and Waggoner 1989).

1.2 Processing Fluorescence Images in the Spatial Domain In pursuit of a prediction of a picture element in fluorescence imaging spatial processing techniques may be used. The use of spatial processing techniques requires some consideration in contrast to time-domain integration since processing may introduce artifacts such as smearing small details and artificial "shapes." Standard linear spatial processing techniques, for example, low-pass filtering, can be used for image smoothing in cases where the noise and image spectra are distinguishable. The noise in fluorescence imaging can only be recognized in the frequency domain if the images are well oversampled. Usually, the instruments are pushed to their limits and compromises between oversampling and the field of view are hard to make (Pawley 1989). Nonlinear filters can be designed to attenuate noise that overlaps the signal spectrum without disturbing the signal. In this paper the use of median filtering techniques in reducing the Poisson noise in fluorescence signals is considered.

1.3 Median and Linear Median Hybrid Filtering Techniques Median filtering techniques have some properties that cannot be achieved by linear techniques: The step response of a median filter is a step and the impulse response is zero. The perfect step response of a median filter in image processing states that edges are preserved whereas impulsive noise can be removed without leaving any traces of the noise in the output signal. Also the median operator will not smear out piecewise constant intervals which are longer than one half of the median window length (Heinonen 1986). This property allows 'tuning' of a median filter to suppress structures below certain size. The use of the standard median (SM) filter in signal processing was first suggested by Tukey (1974) in smoothing statistical time series.

P. Hfinninen et al.: NonlinearFiltering The SM filter is defined as the center value in a series of 2k + 1 samples ( X l , X 2 . . . . . X2k+l) ordered increasingly (or decreasingly). Xme d =

MED[XI .

. . . .

Xk+l] = Xk

(6)

This can also be expressed as value (Xmea), which minimizes (Heinonen 1986, p. 4): 2k+ 1

Xmea = min ~ ]Xmed-- Xi]

(7)

i-0

The SM filter has been extensively used in signal and image processing. (Rabiner et al. 1975, Justusson 1979, Huang et al. 1979, and others). The SM filtering techniques provide very little flexibility in design and are more cumbersome to implement effectively than linear filters: Calculating a median of a set of samples is typically a N log Nproblem (Gallagher et al. 1981), where N is the number of samples in a filter window. Linear methods, in contrast, are often N-problems. Heinonen and Neuvo (1985, 1987) and Astola (1989) first suggested the use of linear FIR subelements in median filters to increase their flexibility and reduce the calculational cost of a nonlinear filter. With these FIR median hybrid filters (FMH) the well-understood linear filter design methods can be employed in the design of the linear subfilters. In FMH filtering a Ndimensional input signal X(i~ . . . . . iN) is filtered first with M FIR subfiiters (i = H/(zj . . . . . z,,), where j = 0 . . . . . M. The output of the FMH filter is then Y(Yl . . . . . y,) = MED(/I . . . . . l,,). The median operator is applied after the linear elements have been calculated, leading to edge-preserving and impulsive noise-removing characteristics. An important feature of the FMH filter class as well as the SM filter class is the ability to generate root signals. A root signal of such a filter is a signal that is invariant to filtering with that filter. In practice, this means that similar filter structures may be cascaded for a better performance. The root structures of the different linear median filtering (LMH) classes have been discussed in (Astola et al. 1987). FMH filters have been successfully employed in various one and two dimensional signal processing applications (Astola et al. 1989, Nieminen et al. 1987a, Nieminen et al. 1987b, Salo et al. 1988, Vainio et al. 1988 and others). Their properties and design has further been studied in (Yli-Harja 1988).

1.4 Some Other Nonlinear Filtering Techniques Already in 1975 (Rabiner et al. 1975) an effort was made to improve the characteristics of a SM filter

245

by cascading it with a linear smoothing flter. In (Nodes et al. 1980) the recursive median filter was introduced. In the recursive median filter part of the input samples to the median are replaced by outputs from the previous steps in the filtering process. Recursive median filter was then further developed by (Stein et al. 1985). They used the recursive median structure adaptively by using the earlier samples to determine whether the output of the filter should be the median or some other value inside the filter window. Adaptive recursive median structures have better edge response under noisy conditions than the simple recursive median filters. The generalization of the median filter was presented in (Bovik et al. 1983). They presented the order statistic filter in which the input is replaced by a linear combination of the ordered values in the neighborhood of a point. In 1986 (Wendt et al. 1986) the Stack filters class was introduced. Stack filters are lowpass type filters that can be used for image smoothing. Their operation is based on threshold decomposition. In a stack filter integer valued signal is composed to its binary signal components. Each of these binary components is then filtered independently with a boolean function. In fact SM filtering is a special case of the stack filter class. Stack filters are also closely connected to mathematical morphology (Maragos et al. 1987): The basic operations of morphology, erosion and dilation, are stack filter operations.

2

Materials and Methods

The cells shown in the pictures were prepared as described by Merdes et al. (1991) and recorded on the modular confocal microscope at the EMBL (Stelzer et al. 1988). The excitation Argon ion laser line was 488 nm at an incident power in a sample of 2.5/zW. The recording time per pixel was -700 ns (512 pixels per line, 900 lines per second and a duty cycle of 1/3). The one-dimensional simulations were performed with Matlab from Mathworks Inc., and the results were plotted using SigmaPlot from Jandel Scientific. The Poisson noise generator was taken from (Wetterling et al. 1988) and tested for periodicity and distribution. The two-dimensional operations were coded in " C . " All sources and our test data sets are available upon "written" request.

3

Improving Confocally Scanned Images

In beam scanning confocal microscope image planes are recorded digitally in pixel serial mode by scanning across the image field (Stelzer et al. 1988). In the confocal arrangement scanning across sev-

P. H/inninen et al.: Nonlinear Filtering

246

eral planes of focus generates three-dimensional images of the object. In confocal beam scanning microscopy the integration of the signal is done differently in comparison to video microscopy or the human eye: The speed of the line scanner is usually fixed while the scanner amplitude is variable and the integration is done by recording the same line or the full frame several times and by accumulating the results of the A/D conversion pixelwise as a function of the beam position in computer memory. Each pixel in an integrated image is a sum of discrete values recorded at different time points but with the same x, y, z coordinates. Thus, a single, three-dimensional confocal image actually consists of information from a four-dimensional observation, each recorded sample having a distinct x, y, z coordinate and Poisson noise contribution. A neighborhood algorithm to improve such images should therefore be designed in four dimensions. In the design phase of the different algorithms the problem was reduced to a two-dimensional problem, where the first dimension is one of the x, y, z axes of a confocal image and the second dimension is the statistical variation of the intensity at that point in time. The results from different algorithms were compared using (1) a quadratic error sum from the original source line (ref):

qe = ~

(li - refi) 2

test signal was in the order of 160 times the cost of standard pixel averaging and 20 times the cost of linear neighborhood filtering when the filter has been pushed to the level of finding the root signal. These results were achieved in a neighborhood of 5 x-direction points on three lines in time (15-point median) (see also Table 1). A less costly way to use the standard median is to employ it after the pixel averaging has been done. This simple linear median filter will give results equal to the standard median over the whole neighborhood. The problem with this simple linear median filter is that the shot noise of the detector systems (photomultipliers) will pass the linear elements whereas with pure median filters these impulsive shot components can be suppressed. The shot noise component increases as a function of amplification and should therefore be taken into account in cases of low emission levels (i.e., high amplification). Again, the design rules for such a filter are limited. In Figures 3 and 4 the quadratic error functions of the linear filtering, the standard median, and the linear standard median methods are plotted. Spatial linear median hybrid (LMH) algorithms can also be utilized to improve the image quality after the averaging has been done. The 2LH+ method presented in Nieminen et al. (1987) is a twolevel bidirectional FHM filter operating in a 5 x 5 window. The algorithm of the filter is

(8)

i=0

and (2) the preservation of edges. Some results are plotted in Figures 1 through 6.

3.1 Linear and Standard Median Neighborhood Methods in Spatial Domain The standard integration (summation) in confocal microscopy is a one-dimensional linear filter employed in the time (Poisson) domain. To improve the statistics of a single pixel, its spatial neighborhood can be used by employing a spatial linear filter. In the design of such a filter the spatial properties of the image must be considered: For the human observer the preservation of edges is important. Blurred edges will appear more disturbing than the noise that had been removed. The blurring of the edges can be avoided by the use of a nonlinear median operator in the pixel neighborhood. The median operation will be effective in suppressing the noise component and can be repeated to increase its performance. The use of a large neighborhood in the median operator causes the fine details to disappear. The problem with a standard median filter is its calculational cost, which, for example, on the

y(m,n) = MED (x(m,n), yl(m,n), y3(m,n))

(9)

where yl(m,n) = MED (x(m,n), (x(m-l,n-l) + x(m-2,n-2))/2, (x(m+l,n+l) + x(m+2,n+2))/2, (x(m+l,n-l) + x(m+2,n-2))/2, (x(m- 1,n+ 1) + x(m-2,n+2))/2) and y2(m,n) = MED (x(m,n), (x(m-1,n) + x(m-2,n))/2, (x(m+ l,nl) + x(m+2,n))/2, (x(m,n-1) + x(m,n-2))/2, (x(m,n+ 1) + x(m,n+2))/2) The algorithm is presented in Figure 1. This filter is known to preserve horizontal, vertical, and diagonal lines wider than 2 pixels but attenuates noise better than the 5 • 5 standard median. The problem of this algorithm in confocal image processing is that it is applied only in the two-dimensional spatial domain, whereas the data set also consists of data from the time domain and the zdirection. Applied to confocal data, the algorithm smooths the noise effectively but introduces larger

P. Hiinninen et al.:

Nonlinear Filtering

247

Table 1. Comparison of the different methods with relative performance calculated according to the ratio: (method quadratic error)/(single pixel averaging quadratic error) Achieved Relative Improvement Method/ Three Source Lines Single pixel Neighborhood averaging Averaging + neighbor median (5 pts.) Standard median (5 neighbor, 15 pts.) Method-I FMH Method-2 FMH

with a Test Signal of: 0-10 0-30 0-3000 Photons Photons Photons

Computational Cost/ Full Performance (0-30 photons)

Step Response in Sampling Units/ Full Performance

1 4 3

1 2 3

1 I 2

1 8 50

4 5 10

l 7 1

3

3

1

160

120

1

>5 7

5 5

2 2

120 120

40 20

1 1

Computation Cost/ >1 Performance

Two Source Lines Single pixel averaging Neighborhood averaging Averaging + neighbor median (5 pts.) Method-I/2 FMH Method-2/2 FMH

Computational Cost/ >2 Performance (0-30 photons)

0.8 3