Maximum-entropy data restoration using both real ... - IUCr Journals

686

M U L T I P L E R E F L E C T I O N S IN A P L A N E - P A R A L L E L M O S A I C C R Y S T A L

and then look for those values of A other than 0 or which cause the right-hand side of (25) to vanish. In practice it has been observed that there is only one such value, ,4 0, which is the root, when it exists, of a transcendental equation. As a consequence, there may be a dip at the centre of the reflection and a m a x i m u m at ,40. Notice that the existence of the dip is not confined to the a p p r o x i m a t e solution (22) in which case it could be a mathematical artefact. In conclusion, we can say that the present investigation has shown that, in the case of multiple scattering of electromagnetic radiation, in general there is coupling between the two states of polarization either through the amplitudes or through the intensities. Only in the coplanar case do the two X-ray components act i n d e p e n d e n t l y and can be decoupled. The numerical examples which have been reported are the first exact or almost exact calculations of the effect of multiple reflections in a mosaic crystal within the limits of validity of the transfer equations. These calculations show that application of the kinematical a p p r o x i m a t i o n to crystals having reflectivities comparable to those of copper can be grossly in error (up to a factor of two) if the crystal thickness is of the order of what is used in practice ( ~ o l - 1 for X-rays

and / x o l - 0 . 0 1 for neutrons). This discrepancy can be significantly reduced using the two-beam formulas; however, if one is interested in obtaining accurate structure factors, multiple reflections have to be taken into account. Equations (22), (23) and (24) can be very useful for this purpose, particularly in the case of neutrons. We thank V. Contini for m a n y helpful discussions. References

CATICHA-ELLIS, S. (1969). Acta Cryst. A25, 666-673. CHANG, S.-L. (1982). Acta Cryst. A38, 41-48. MAZZONE, G. (1981). Acta Cryst. A37, 391-397. MOON, R. M. & SHULL, C. G. (1964). Acta C~st. 17, 805-812. PRAGER, P. R. (1971). Acta Cryst. A27, 563-569. ROSSMAN1TH, E. (1985). Z. Kristallogr. 171,253-254. SHURCLIFF, W. A. (1962). Polarized Light. Cambridge: Harvard Univ. Press. UNANGST, D. & MELt,E, W. (1975). Acta Co'st. A31,234-235. WARREN, B. E. (1969). X-ray Diffraction. Reading, MA: Addison Wesley. WERNER, S. A. (1974). J. AppL Phys. 45, 3246-3254. ZACHAR1ASEN,W. H. (1945). Theory of X-ray Diffraction in Crystals. New York: John Wiley. ZACHARIASEN, W. H. (1963). Acta Cryst. 16, 1139-1144. ZACHAR1ASEN,W. H. (1965). Acta Crvst. 18, 705-710.

Acta Cryst. (1989). A45, 686-698

Maximum-Entropy Data Restoration Using Both Real- and Fourier-Space Analysis BY DAVID M. ANDERSON,* DAVID C. MARTINi AND EDWIN L. THOMAS~: Polymer Science and Engineering Department, University o f Massachusetts at Amherst, Amherst, M A 01003, U S A (Received 8 February 1988; accepted 15 May 1989)

Abstract

An extension of the m a x i m u m - e n t r o p y (ME) datarestoration m e t h o d is presented that is sensitive to periodic correlations in data. The method takes advantage of the higher signal-to-noise ratio for periodic i n f o r m a t i o n in Fourier space, thus e n h a n c i n g statistically significant frequencies in a m a n n e r which avoids the user bias inherent in conventional Fourier

* Current address: Physical Chemistry 1, Chemical Center, University of Lund, PO Box 124, S-221 00 Lund, Sweden. t Current address: Department of Materials Science and Engineering, The University of Michigan, H. H. Dow Building, Ann Arbor, MI 48109, USA. :~Current address: Department of Materials Science and Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA. 0108-7673/89/100686-13503.00

filtering. This procedure incorporates concepts underlying new approaches in q u a n t u m m e c h a n i c s that consider entropies in both position and m o m e n t u m spaces, although the emphasis here is on data restoration rather than q u a n t u m physics. After a fast Fourier transform of the image, the phases are saved and the array of Fourier moduli are restored using the m a x i m u m - e n t r o p y criterion. A first-order continuation method is introduced that speeds convergence of the M E computation. The restored moduli together with the original phases are then Fourier inverted to yield a new image; traditional real-space ME restoration is applied to this new image completing one stage in the restoration process. In test cases with various types of added noise and in examples of normal and high-resolution electron-microscopy images, dramatic i m p r o v e m e n t can be obtained from © 1989 International Union of Crystallography

DAVID M. ANDERSON, DAVID C. MARTIN AND EDWIN L. THOMAS two to four stages of iteration, even in cases where traditional ME restoration provides little improvement. It is shown that in traditional Fourier filtering spurious features can be induced by selection or elimination of Fourier components without regard to their statistical significance. With the present approach there is no such freedom for the user to exert personal bias, so that features present in the final image and power spectrum are those which have survived the tests of statistical significance in both real and Fourier space. However, it is still possible for periodicities to 'bleed' across sharp boundaries. An 'uncertainty' relation is derived describing the inverse relationship between the resolution of these boundaries and the level of noise that can be eliminated. Introduction

Image reconstruction using digital filtering techniques has become an important part of analysis and interpretation of images in general, and electron micrographs in particular. Because of the many difficulties inherent in high-resolution microscopy, including sample preparation, microscope tranfer aberrations and beam sensitivity, the images obtained are often not readily interpretable and so digital restoration techniques utilizing both direct image and Fouriertransform data have been developed. For the interpretation of data from periodic structures, Fourier-space data offer the advantage of condensing a great deal of structural detail into discrete Fourier-lattice spikes, often leading to a much higher signal-to-noise ratio than in the direct image. However, recent work has shown that conventional Fourier filtering may lead to spurious features (Forslund, 1969; Pradere, Revol, Nguyen & Manley, 1988). The user can create artifacts from noisy data by arbitrarily selecting certain Fourier components and masking others. The concept of the 'entropy' of a probability distribution was developed by Shannon & Weaver (1949), and the use of a 'maximum-entropy' principle was proposed by Jaynes (1957) as a means of choosing with minimum bias between several feasible outcomes of a given experiment. Maximum-entropy methods have been applied to many types of data including radio-astronomy images, X-ray tomography, NMR and X-ray diffraction (Gull & Skilling, 1984). The use of ME procedures has become very important in astronomy (Narayan & Nityananda, 1986), and the possibility of extending these techniques for use in electron microscopy has been suggested (Frieden, 1987). It has been proven in rigorous mathematical arguments that the maximum-entropy formalism is the only data restoration method that does not introduce correlations in data (Shore & Johnson, 1980, 1983). In cases where data are related to the observed object by a known linear transformation, such as convo-

687

Table 1. Comparison between sequential maxent and direct methods Sequential m a x e n t Raw data from real space All Fourier-space features tested for statistical significance Fourier moduli calculated from real-space data Entropy defined in both spaces Phases known from data Unique solution

Direct m e t h o d s Raw data from Fourier space Only data from Fourier "lattice' sites used Fourier moduli measured experimentally Entropy defined only in real space Phases calculated from model Non-unique solution

lution with a point-spread operator (or 'blurring' operator), there is guaranteed a unique solution to the maximum-entropy equations for any assigned value of the experimental variance (Gull & Skilling, 1984). The analogy between the maximum-entropy approach and the 'direct methods' of phase determination in X-ray crystallography has been presented in detail (Bricogne, 1984), and recently the constraint of atomicity due to Sayre (1952) has been shown to be an expansion to second order of the entropy operator in real space (Harrison, 1987). In direct methods of crystallography, the experimental data are available in Fourier space as integrated Bragg reflection intensities, which ideally means that estimated experimental variances are low owing to sampling from a very large number of periodic units in the real object. In the context of the present paper, direct methods of crystallography are viewed as belonging to a general class of maximum-entropy reconstruction methods in which the entropy functional is defined in real space while the error criterion is enforced in Fourier space. The goal of crystallographic reconstruction is to find that distribution of electron density in real space which maximizes the real-space entropy functional subject to the constraint that the sum of the squared deviations of the predicted Fourier moduli from the observed moduli be less than the estimated error. The similarities and contrast between our method, which we refer to as 'sequential maxent', and direct methods of phase determination in crystallography are summarized in Table 1. Maximum-entropy reconstructions perform well when the real-space object consists of point sources, as in atronomy or crystallography (Gull & Skilling, 1984). However, the method works poorly when the real-space data are all of nearly the same value. Furthermore, uniqueness of the solution is lost with the loss of phase information, because distinct realspace objects can have rigorously the same Fourier intensities. Loss of uniqueness occurs because without phase information the relation between the object and the observed data is nonlinear (Bryan & Skilling, 1986).

688

M A X I M U M ENTROPY DATA RESTORATION

Our interest is in the most general case of imagedata reconstruction where the both the real (or 'particle') and Fourier (or 'wave') aspects of our experimental data are tested for statistical significance. We are particularly interested in boundary and defect regions in low-dose high-resolution electron micrographs. Finally, although electron diffraction yields useful information, scattering techniques usually average over large areas of sample and therefore information from defects, which are typically dilute, is lost. Also, the size, shape and relative orientation between individual crystalline domains and defects is not known. Thus, our goal is to take full advantage of this higher degree of structural detail that can be obtained from electron micrographs. It should be mentioned that for simpler problems, such as unit-cell reconstructions for objects with periodic order which extends homogeneously throughout the sample, there exist other bias-free techniques combining electron diffraction data with electron micrographs. Examples are the iterative procedure of Gerchberg & Saxton (1971, 1972), and the extension of the direct method by Fan Hai-fu, Zhong Zi-yang, Zheng Chao-de & Li Fang-hua (1985). The basic philosphy of such methods is that the electron diffraction pattern provides the Fourier moduli, while the phase information is available from the electron micrograph. For the more general problem of restoring an image with information on defects, grain boundaries, or polycrystalline regions, it is not often experimentally possible to obtain an electron diffraction pattern and an electron micrograph that correspond to exactly the same state of the sample, particularly for radiation-sensitive materials. A further complication is that for certain samples (e.g. block copolymers), the features of interest are so large that diffraction experiments would have to be performed at unfeasibly small angles. Summarizing our requirements, we want the restoration method to: (1) enhance periodic correlations in the data; (2) use information only from the image itself; (3) be free from user bias. Traditional Fourier filtering satisfies (1) and (2) but not (3), as has been shown (Forslund, 1969; Pradere et al., 1988) and which we demonstrate in the Results section. However, if the Fourier moduli are restored not by filtering but rather by applying the principle of maximum entropy, the third requirement can also be satisfied. The Fourier moduli are particularly well suited for ME restoration, because the distribution of moduli is more localized than the original distribution of real-space densities for the cases of interest here, and the ME method is known to perform well for point sources (Gull & Skilling, 1984). In the next section we describe our method in detail. Our approach differs from previous M E formulations

which consider the Fourier representation of data (Bryan & Skilling, 1986) in that the entropy in the Fourier-space iteration step is calculated from the Fourier moduli of the original image. The second step in each cycle is an ordinary real-space ME restoration. In the Results section we apply the method to synthesized data and to actual electron-microscopy data. One ofthe latter examples is of a grain-boundary region in a radiation-sensitive polymer where techniques designed for unit-cell reconstruction do not apply and where radiation sensitivity precludes the possibility of combining electron microscopy and electron diffraction. The Discussion section focuses on the tendency we have found for frequencies which are enhanced in the image to 'bleed' across sharp boundaries. We show how to estimate the extent of the bleeding, but emphasize that this is at present a difficulty with the method which needs to be addressed. We are hopeful that an algorithm which maximizes the real- and Fourier-space entropies simultaneously, rather than iteratively, may be helpful in this regard. Method

The basic philosophy of maximum-entropy-data reconstruction is to find that image out of all the possible 'guess' images which maximizes the entropy functional

Q = - Y ~ [ p i i n p i ] - h Y~[(d,-f)2/o'2],

(1)

where {f} are the estimates to the original data {di}. The {p~} are the 'probabilities' associated with the pixels i, and are defined as p , = f / ~ { f } . Thus, Y~{p~} = 1.0. Q has two components, the first being the entropy of the probability distribution {p,} and the second being the deviation of the reconstruction from the measured data. The entropy term is the general form for a probability distribution {p,} suggested by Shannon (1948), and is similar to the Boltzmann entropy of statistical thermodynamics. The second term is a X2 constraint C imposed due to the experimental data {d~} and h represents a Lagrange multiplier whose final value is related to the quality of the data {d,}. h is chosen so that the X2 constraint C =Y. (d~-f)2/o2 is equal to N in the final solution. As the quality of the data improves and o- decreases, deviations of the estimates {f} from the original data {d~} become more costly and the {f} which finally maximizes Q is close to {d~}. For large values of o-, the entropy term dominates and the result is a flat featureless image. The entropy functional Q,,=S~-AC can be defined for any set of conditional probabilities {p~}. However, using this functional alone ignores any spatial relationship {x~} between the individual events p~. In many sets of data, in particular any image, there is an explicit spatial relationship between the

DAVID M. A N D E R S O N , DAVID C. M A R T I N A N D E D W I N L. T H O M A S intensities Pi. In other words, the values of {Pi} are not independent. In traditional ME reconstructions the known spatial relationship between events is incorporated by considering that in an experiment there is a point spread function (p.s.f.) b which causes a data point in the original 'blurred' object to be spread over several pixels in the final image (Gull & Skilling, 1984). The point spread function is usually assumed to be a linear convolution with every point in the original object. This is mathematically equivalent to Fourier transforming the object f~ to Fk, multiplying by B, the Fourier transform of b, and inverse transforming. If the p.s.f, is not known, it can be estimated as a function in the ME reconstruction (Gull & Skilling, 1984). We have found that it is useful to assess the statistical significance of the frequency characteristics of an image by applying the principle of maximum entropy in Fourier space. Since the Fourier transform of the object gives an array of complex numbers Fk which is itself an 'image', it is possible to consider a 'Fourier-space entropy' by taking the normalized magnitudes of the complex numbers Fk as probabilities themselves. This new entropy term takes into account the spatial relationships in the real-space data {di}. Recent work in quantum mechanics using entropies defined in both spaces has helped in refining estimates of wave functions (Gadre, Bendale & Gejji, 1985; Gadre & Bendale, 1985a). Gadre & Bendale (1985b) have suggested a new maximum-entropy principle which states that it is the sum of the real-space and Fourier-space entropy which should be maximized. Here, we investigate the application of this principle to observable data. We make a first approximation to the 'maximum-sum' idea by applying maximum entropy iteratively in Fourier space and in real space. The relationship between our method and that proposed by Gadre & Bendale (1985b) is shown schematically in Fig. 1. Consider a real-space wave function ~ ( x ) and its Fourier transform in momentum space qb(k). The wave function gr(x) gives rise to a real-space probability distribution p(x) upon

~/(x) ~*(x) - - >

p(x) ~

I

P(k) ~-~ p'(x)

~b(k) qb*(k) - - >

P'(k)

Fig. 1. Schematicof the relationship between probability functions in real and Fourier space, gt(x) and 4~(k) are complex Fouriertransform pairs, giving rise to probabilities p(x) and P(k). The Fourier transform ofp(x) is P'(k),which is not the same as P(k).

689

multiplication with its complex conjugate gt*(x); likewise, crp(k)crp*(k) generates a momentum-space probability P(k). The functions p(x) and P(k) are not Fourier-transform pairs; the Fourier transform of p(x) is P'(k), while the Fourier transform of P(k) is p'(x). Since p(x) and P(k) represent probability distributions they are always real and positive, whereas P'(k) and p'(x) are in general complex. The transformation from qt(x) or qb(k) to p(x) or P(k) respectively is irreversible; having complete information about p(x) does not mean that ~ ( x ) can be determined. Gadre & Bendale (1985b) calculate the entropy functional using the integral form with p(x) and P(k). This is possible because the starting point is ~ ( x ) , obtained from quantum calculations. In our case, we have information only about the real-space probability distribution p(x). Therefore, in order to calculate the entropy in Fourier space we Fourier transform p(x) to P'(k) and then define an entropy S~, S ~ , = - ~ P~, In P~,,

(2)

where P~, are the normalized moduli of P'(k). By the convolution theorem, P'(k) is the autocorrelation function of ~ ( k ) . The numerator of the constraint term is the sum over the squared differences between {f} and {di} in real space or {Fk} and {Dk} in Fourier space. By Parsevars theorem it can be shown that these two terms are equivalent as long as the phase of each Fk is left equal to that of the corresponding Dk. Since in our method we mandate that the phases for Fk remain the same as for Dk, it does not matter whether we choose to calculate the constraint term in real or Fourier space. On the other hand, the entropies Sx and S~, are not the same for a given image. To address the full information content of a signal, it seems necessary to consider both its 'particle' and its 'wave' characteristics. Indeed, it has been noted by Bialynicki-Birula & Mycielski (1975) that it is possible to recast the Heisenberg uncertainty principle in the more general form

Sx+ Sk >-3Np[l +ln ('n')].

(3)

This says that the sum of the real-space entropy and Fourier-space entropy is always greater than a constant which depends on the sumber of particles Np in the system. Our algorithm is represented schematically in Fig. 2. First, the original data set {di} is Fourier transformed to form a complex data set with moduli {Fk} and phases {Ok}.Then, the phases {Ok}are stored and a maximum-entropy reconstruction to a new set of Fourier moduli {F~'} is performed. The moduli {F~'} are then recombined with the phases and inverse Fourier transformed to form a real-space image {f}. Finally, a real-space maximum-entropy restoration is

690

M A X I M U M E N T R O P Y DATA R E S T O R A T I O N

performed to give a new estimate in real space {f~}. The process can now be repeated using the new set of {fT'}'s. Our computational scheme for the maximumentropy reconstruction follows that used by MoriartySchieven, Snell, Strom, Schloerb, Strom & Grasdelen (1987) which is itself an extension of the analysis by Willingdale (1981). A maximum in the entropy functional is found when the transcendental equation

r"=jexp[A(b*d"-b*b*r")]

(4)

is solved. Here, r" is the reconstructed image ({F~'} or {f~'} in our case), dr, the original data ({Dk} or {di}), b the point spread function, j a normalizing constant to preserve flux, and * the convolution operator. The value of A is related to the error o- in the data and is the only parameter in the reconstruction. With information about the average error in the data tr it is possible to choose A so that the g 2 constraint C is equal to N. By using original data and adding noise, Moriarty-Schieven et al. (1987) showed that there is an optimum choice for A which minimizes the error between the reconstruction and the 'noise-free' data. As A increases the error between the reconstruction solution and d a t a first decreases and then increases as the reconstruction begins to fit the noise. At present, we are unsure of the error cr in our data and therefore use the reconstruction as a means of its estimation. Once tr is more well known it will be possible to constrain the problem further so that there are no adjustable parameters. In order to improve the rate of convergence of the method, a first-order continuation was performed which takes into account the expected variation in r,, due to a change in A. These first-order continuation procedures have proven very effective, for example, Data

I MaxEnt

fi

1

] FFT

MaxEnt

F~

I

' I

.....

;

Fig. 2. Schematic of the iterative algorithm used in this paper. First, the original data d, are Fourier transformed to create a set of moduli Fk and their associated phases. Maximum-entropy techniques are applied to Fk to create a new set of amplitudes F~', these are then recombined with the original phases and inverse Fourier transformed to create a new real-space array f,. Maximum entropy is then performed on f, in real space to create f~". The new real-space estimatef~ may be Fourier transformed to restart the cycle.

in finite element algorithms (Silliman, 1979). If we approximate the Jacobian as diagonal,

(OE,,IOr,,)(Or"IOA)+(OE,,laA)=O

(S)

when the solution is found. Thus, {1 + NA exp[A(dm-r")]}(Arm/AA ) = - r m ( d " - r , , ) (6) or

A r , , = - ( A A ) r , , ( d " - r , , ) / { l + NA exp [A (d" - r,,)]}.

(7) The idea behind this approach is to keep the residuals dE,, close to zero when A is incremented. During the ME reconstruction, A was slowly increased until the correct value of A was reached. This first-order continuation scheme allowed us to estimate the expected change in r,, due to a small change in A and thus speed convergence. We have found that best results are obtained if the average value of the Fourier transform is first set to zero. This is done simply by setting the intensity at the origin pixel to zero. In the FFT algorithm, the intensity in this single pixel transforms to a uniform signal over all of Fourier space, thus artificially raising the entropy due to a decrease in relative varihtions. Furthermore, the average pixel intensity was subtracted before evaluating the Fourier transform. This eliminated the large spike at the origin which otherwise tended to dominate the Fourier-space maximumentropy step. In using this sequential ME algorithm there is no freedom for the user to enhance frequencies of interest. The frequencies in the image which survive are those which pass the test of statistical significance imposed by the Fourier-space maximum-entropy step. The following simple example shows clearly how features can be induced in a noisy image by choosing frequencies without concern for their statistical significance. Fig. 3(a) shows a noisy image that was created by placing 10 000 spots of intensity with radii of mean zero and standard deviation 5 pixels randomly on a 128 x 128 array. Fig. 3(b) shows the result of conventional Fourier filtering, where a randomly chosen frequency was extracted. The scaling of pixel intensities was kept the same as in Fig. 3(a) so a direct comparison could be made. Note that the chosen frequency is indeed 'supported' by the noisy data, giving rise to a reconstruction that could possibly be misinterpreted as representative of some type of periodic structure. Results

In this section we apply the new method to several fabricated images, and to actual TEM micrographs from two polymer materials of particular interest in

DAVID M. A N D E R S O N , DAVID C. MARTIN A N D EDWIN L. THOMAS our group. The quality of the final reconstructed images will be assessed on two grounds: first, the clarity and symmetry of those statistically significant periodicities which are restored, as evidenced in both the final image and its transform; and second, the faithful reproduction of sharp boundaries for these periodicities, with bleeding kept to a minimum. We also use a fabricated image in a critical assessment of the singular-value-decomposition (SVD) imagerestoration technique.

(a)

(b) Fig. 3. (a) 128 x 128 pixel noisy image created from 10 000 Gaussian spots with radius of mean zero and standard deviation of five pixels. (b) Reconstructed image formed by choosing an arbitrary frequency and inverse transforming. The image has a set of bands with apparent 'dislocations' and other anomalous features for which there is no statistical support in the data.

691

Fig. 4(a) shows a simulated image of a simple sinusoidal periodicity with noise added. The noise consists of 40 sine waves each of random wavelength and direction, with intensities randomly.distributed between 0 and 15% of the primary periodicity. Fig. 4(c) shows the final result after applying a complete full Fourier-space and real-space restoration cycle. The results of a single restoration cycle in this example show remarkable image enhancement in real space; it is now an easy matter to distinguish the primary periodicity. Figs. 4(b) and (d) show the power spectra of Figs. 4(a) and (c), respectively, and it is here that the reason for the dramatic improvement during the Fourier-space restoration step is most evident. These and all subsequent power spectra are presented on a logarithmic scale to allow weaker components to be easily discerned. The primary periodicity is circled in Fig. 4(b); in Fig. 4(d) it stands clearly above the now supressed 'noisy' frequencies. Note that the strong horizontal and vertical streaks in Fig. 4(b) due to aliasing at the edges of the figure have been suppressed. The influence of the effective point spread function of the simulation is seen in Fig. 4(d) as a damping of high-frequency information. The discrete nature of both the primary periodicity and the noise in Fourier space makes the new restoration method particularly effective and transparent in this simple example. In the next two examples, we impose a sharp boundary on the primary periodicity in order to assess the extent of bleeding. In Fig. 5(a) the same periodicity as in the previous example has been given unit intensity on the left half of the image, and noise added to the entire image. The noise consists of 40 sine waves as in the previous example, but in this example the intensity of a wave is randomly distributed between 0 and 30% of the primary-wave intensity. Fig. 5(b) shows the image resulting from one cycle of restoration. While the primary periodicity appears clearer on the left side of the image, it can be seen that this periodicity has bled onto the right side. This can be most easily seen by viewing the figure at a low angle parallel to the direction of the fringes. Thus the new method has created an artifact. At this noise level, some of the added sine waves have sufficient intensity - close to 30% of that of the primary periodicity - that the maximum-entropy technique evaluates them to be statistically significant in both real and Fourier space, and thus regardless of the number of restoration cycles the image cannot be reduced to the primary periodicity alone. In the next example (Fig. 6), the same primary periodicity as in the previous example has been given unit intensity on the left half of the vertical center line, and then noise has been added to the entire image. The noise in this case consists of the superposi-

692

M A X I M U M E N T R O P Y DATA R E S T O R A T I O N

tion of 250 aperiodic waves of the functional form: a,, sin [ b,,(iG + C,]) I/2]

(8)

G = ax, +/3x2

(9)

where the intensity is randomly distributed between 0 and 25%. The parameters a and /3 are the components of a vector which determines the direction of the wave. Contrary to the previous example, this noise does not have a discrete Fourier transform. The high level and large number of noise components in this example makes the primary periodicity very difficult to see in the unrestored image, even when the observer knows its exact location. Fig. 6(b) shows the image resulting from two cycles of restoration. The primary periodicity on the left is now clearly visible, and the vertical boundary appears

to be sharp. However, owing to the high noise level, it is difficult to judge from the real-space image whether or not bleeding has occurred on the right side. This determination must be made using the Fourier transform. Fig. 6(c) is the power spectrum of Fig. 6(a); the peak corresponding to the primary periodicity is circled. Figs. 6(d) and (e) are the power spectra of the right side of Fig. 6(a) before and after restoration. Note that in Fig. 6(d) there is no support for the primary periodicity, whereas in the final image there is indication of the primary periodicity on the right side of the figure. Therefore bleeding has occurred. However, the intensity of the circled peak is on the order of the noise level in the restored power spectrum, thus explaining why the bleeding cannot be distinguished above the background noise in Fig. 6(b).

(a)

(b)

(c)

(d)

Fig. 4. (a) 128 x 128 pixel image ofa sinusoidal periodicity with added noise of 40 sine waves each of random wavelength and direction, with intensity randomly distributed between 0 and 15% of the primary periodicity. (b) Log of the power spectrum of (a). The sine waves generate strong peaks with associated horizontal and vertical 'streaking' due to aliasing when the sine waves meet the edge of the frame. (c) Reconstructed image after one cycle of the iterative maximum-entropy scheme. (d) Log of the power spectrum of (c). The noisy waves have been considerably damped but are still visible on a log scale.

DAVID M. A N D E R S O N , DAVID C. MARTIN A N D EDWIN L. THOMAS The next two examples are actual electron micrographs of polymeric samples with interesting periodic structures. Fig. 7(a) is a digitized micrograph of a triply periodic morphology that has recently been discovered in block copolymers called the ordered bicontinuous double-diamond (OBDD) structure (Thomas, Alward, Kinning, Martin, Handlin & Fetters, 1986). This structure is modeled well by two imbedded periodic surfaces of constant mean cur-

'

,

(a)

(b) Fig. 5. (a) 128 x 128 pixei image with a strong sinusoidal periodicity on the left half of the image and 40 sine waves with intensities randomly distributed between 0 and 30% of the primary periodicity. (b) Image after one cycle of the restoration. The left side of the image is clearer, but the periodicity can also be seen on the right when viewed carefully at a glancing angle.

693

vature which divide space into three regions. Two of these regions are congruent and have diamond symmetry (Anderson & Thomas, 1988). Fig. 7(a) was taken at a magnification of 33 000x and digitized directly using a 50 ~m spot size with an Optronics P-1000 scanning microdensitometer. An 8-bit intensity resolution for an optical density range of 0-2 was used. The image shown is a 128 x 128 pixel region. This structure is formed in a microphase separated sample of a star diblock copolymer of poly(styrene) and poly(isoprene). The regions of poly(isoprene) have been stained dark by exposure to OsO4 vapor. Fig. 7(c) is the result after two complete cycles of restoration. Considerable symmetrization has occurred, and the image appears smoothed and clarified as evidenced by the uniformity from unit cell to unit cell. This uniformity does not extend to the edges of the image, and in fact a general rule is that the borders of an image restored with the present method are susceptible to artifacts. In unit-cell reconstructions, this does not pose a serious problem, and neither does bleeding, because the primary concern is to obtain a single symmetrized unit cell. Figs. 7(b) and (d) show the power spectra before and after restoration respectively. As is evident, the main source of the dramatic improvement from Fig. 7(a) to Fig. 7(c) is due to a clarification of the power spectrum. Fig. 7(e), which is a restoration of Fig. 7(a) using only real-space maximum entropy, does not show much improvement in clarity, nor any enhancement in symmetry, as compared with the improvement evident in Fig. 7(b). Note that the structure in the power spectrum Fig. 7 ( f ) is not nearly as developed as in Fig. 7(d). Fig. 8(a) is a low-dose high-resolution electron micrograph of a fiber sample of poly{(benzo[1,2-d: 4,5-d']bisthiazol-2,6-diyl)-l,4-phenylene} (PBZT), a rigid-rod polymer which crystallizes in a monoclinic space group (Roche, Takahashi & Thomas, 1980; Odell, Keller, Atkins & Miles, 1981). The strong equatorial reflections which can be imaged by HREM in this material are the 0.36 nm (010) and 0.59 nm (100) planes which represent the lateral packing between adjacent chains. We have selected a micrograph that captures an apparent grain-boundary region, with 0.59 nm fringes on the lower left meeting 0.36 nm fringes on the upper right. This image was formed by digitization of an intermediate negative produced by enlarging the original low-dose HREM image. The final sampling rate was 0.072 nm pixel -~ Whereas all of the images so far in this section have been 128 x 128 pixels, this example is 256 x 256. Fig. 8(b) shows the resulting image after three cycles of ME restoration. The periodicities have been brought out and clarified dramatically. Fig. 8(c) is the same region restored using traditional Fourier filtering. This image was formed by calculating the

694

MAXIMUM

ENTROPY DATA RESTORATION

(a) (b)

(¢)

(d)

(e)

Fig. 6. (a) 128 x 128 pixel image with the same periodicity as in Fig. 5, but now obscured by 250 waves of aperiodic noise randomly distributed from 0 to 25% of the primary periodicity. (b) Image after two restoration cycles of the sequential maximum-entropy algorithm. (c) Log of the power spectrum of (a). The primary periodicity is circled. (d) Log of the power spectrum of (a) with the left side set equal to zero. The primary periodicity is absent. (e) Log of the power spectrum of (b) with the left sicle set equal to zero. The fact that the primary periodicity can be detected (circle) confirms that bleeding has occurred.

DAVID M. A N D E R S O N , DAVID C. MARTIN A N D EDWIN L. THOMAS FFT, selecting only those frequency components whose moduli were above an arbitrary background, and inverse transforming. Although the images in Figs. 8(b) and (c) are similar, the ME image retains features which are statistically significant yet are

(a)

695

eliminated by the Fourier approach. It would be tempting to make more substantial statements about the structure of the grain boundary from these data, however until the bleeding problem is resolved we are hesitant to do so.

(b)

'qF

•

D

D

~

~

#b

•

o

-

& O D ~ O 4 D • Q °

e ' ~ te •e

p 8~ q D Q Q D ' QQQ D Do

4J

•

Ij

e

°

qb Q 0 9

• Q

(c)

(d)

Q Q

9

qlP

I

~

Q ~

~

4

b ID,

t

o

"

4

IP

,. 'e.

• o d

•

Q

di a

6

•

D

• 1~

I

9

qb

LD (e)

(f)

Fig. 7. (a) 128 x 128 pixel digitized micrograph of the ordered bicontinuous double-diamond structure in a block copolymer. Sampling rate 1.5 nm pixel -~. (b) Log of the power spectrum of (a). (c) (a) after two complete cycles of sequential maximum-entropy restoration. (d) Log of the power spectrum of (c). (e) (a) restored using only real-space maximum-entropy reconstruction. ( f ) Log of the power spectrum of (d).

696

M A X I M U M E N T R O P Y DATA R E S T O R A T I O N

The final example in this section is intended to test claims in the literature (Andrews, 1976) that singularvalue decomposition (SVD) can be used in image processing without the dangers of bleeding and ringing inherent in Fourier processing. SVD is a technique of decomposing an image into a set of 'eigenimages', each of which is associated with a particular eigenvalue of the matrices G G r and G ' G where G T is the transpose of the original image G. The original image G can be reconstructed from the sum over the number of rows or colums M as

G=

~ (E~2U.,V,,,),

(10)

i=l,M

where E,, is t h e ruth eigenvalue and U,,, are the column eigenvectors of G G r, V,,, the row eigenvectors of GTG. Fig. 9(a) shows a fabricated test image, in which a pentagon has been filled with a sinusoidal wave, and the aperiodic noise shown in Fig. 5(a) super-

III

".L

'~ "~

ql~

•

~a,.-

.'

r~.

. .~

6,.6

•

"

~

,

f

."

.

,

.

.

"'P

-'~"

•

o

9

. .:. "

"

""

-

•

This section is devoted to a discussion of the bleeding problem, and possible means for its solution. We have seen that for unit-cell reconstruction bleeding is not a problem. However, in our studies we routinely deal with micrographs in which many crystallites with many orientations are imaged in the field of view, and at a level which is not dramatically above the

•

-

"

,. - -" ~-' "O"

~

"

°d

-...,1

~-..

~'"

Discussion

~oG

"""

..? . : - ' ' . . 7 -

,,".

.1" •

"

"

"I.o, "",, .'- ~

,.....'"q.o

I

.~,s"

.. ,~ ,,*

.,,4b d l ~ p ' "

';41,

~

posed over the entire image at the 30% level. Fig. 9(c) is the weighted sum of the five eigenimages having the largest magnitudes of the eigenvalues. Clearly bleeding has occurred. In fact, if a box is drawn with horizontal and vertical edges, this being the smallest such box containing the entire pentagon, then the primary periodicity seems to fill out this entire box. We conclude that, contrary to claims in the literature, SVD processing can cause significant bleeding.

;'"

t :"'

""'"

.-..-...::: • "

.,

~

w. •-

--

"41 b "

....

• --..

~

.,"'-

. . . .

~-

,-.

o

,

dlb

--"-"

~..'V.

~ .....

...-~.

(a)

(c)

_

.-. , "

,_1,~,.,. (b)

Fig. 8. ( a ) 256 x 256 pixel digitized low-dose high-resolution electron m i c r o g r a p h o f PBZT. Sampling rate 0.072 nm pixel -~. (b) ( a ) after three cycles o f sequential m a x i m u m - e n t r o p y restoration. (c) ( a ) restored using Fourier filtering where all o f the frequencies with moduli below some arbitary level were set equal to zero.

DAVID M. A N D E R S O N , DAVID C. MARTIN A N D EDWIN L. THOMAS noise level. In such instances artifacts resulting from Fourier filtering are known to be a serious problem. Let us now consider a specific example which will help us illustrate the expected extent of the bleeding problem in a general image-reconstruction effort. Although a one-dimensional example is used for simplicity, the extension to two (or more) dimensions is straightforward. Assume that the form of the image is a set of well defined fringes modulated by a 'shape factor' T which is equal to either 1 or 0 depending on whether the fringes are visible or not. Ix = A cos ( kox + O)Tx.

(11)

The Fourier transform of this object is a delta function at frequencies of +ko, convoluted with the transform

697

of the object function Tk:

Ik=A[8(k-ko)+6(k+ko)]* Tk.

(12)

If Tx is a rectangular function, then Tk takes the form of (sin Zk)/k, where 2Z is the dimension of T,,. In this formalism, Tk is the 'form factor' of the crystallites and the delta functions are the 'lattice'. Note that the form factor Tk depends on the shape of the crystailite region and will be different for different areas of the image. Bleeding occurs because the components of Tk which are low in magnitude cannot be separated from noise in Fourier space, and thus Tx cannot be reproduced faithfully. Losing parts of Tk thus make the resulting T~, sharper, making T" broader than Tx, causing bleeding of the frequency.

q •

i

r

t., (a)

(b)

t

(c)

(d)

Fig. 9. (a) 128 × 128 pixel test image of pentagon containing sinusoidal wave superimposed on noise similar to Fig. 5(a) at a level of 30%. (b) Log of the power spectrum of (a). (c) Sum of the five eigenimages with the largest eigenvalues. (d) Log of the power spectrum of (c).

698

M A X I M U M ENTROPY DATA RESTORATION

Let us define an error criterion E which measures the squared error between the unbled image and the bled image. By Parseval's theorem, this error can be defined in either real or Fourier space:

IBM Institute for Interface Science, AFOSR and URI for financial support. The algorithms used in this work were written in Fortran 77 and are available by request from the authors.

E = ~ (Fk,- Fk2)2 dk, dk2 =~J (Bd cos k~x) 2 dx, dx2,

(13)

where Bd is the average strength of the bleeding periodicity with frequency kb in real space. Evaluation of the second integral for an interface of length L with a width of bleeding W gives

E>_(B2LW)/2.

(14)

A protocol for using this criterion is as follows. Let us assume we are interested in estimating the extent of bleeding W in an image. First, estimate the length of the boundary L. Then calculate the error integral E in Fourier space near the frequency of interest kb. Formula (14) then provides an estimate of the strength Ba of the bleeding. It has recently been shown that a localized entropy operator can be successful in edge detection (Shiozaki, 1986; Martin, 1988). Our approach suggests that a similar operator which includes the Fourier-space entropy term may be useful in detecting features which only significantly affect the Fourierspace entropy, such as the edges of faint periodicities on a noisy background. Such an algorithm would be quite useful for determining the boundaries of crystallites in low-dose HREM lattice images. The failure of SVD processing in preventing bleeding, as demonstrated in the last section, is particularly disappointing in that of all outer-product expansions of an image - which include as special cases Fourier, SVD and Hadamard expansions - the SVD expansion is considered to be the expansion in which the component images are most closely matched to the image. Therefore we are lead to believe that any outerproduct expansion will be subject to bleeding artifacts. Resolution of the bleeding problem may await an algorithm which finds the maximum of the sum of entropy functionals Sx and S~ subject to the data constraint AC. However, this approach will require a memory-intensive and speedy algorithm since the FFT of the image will need to be part of the iteration step. Thanks are due to Dr Alain Boudet for obtaining the PBZT HREM data, and to Dr David Kinning for obtaining the (111) projection of the O B D D structure. DCM thanks the Shell Company and the University of Massachusetts for fellowships. ELT thanks the

References ANDERSON, D. M. & THOMAS, E. L. (1988). Macromolecules, 21, 3221-3230. ANDREWS, H. C. (1976). Appl. Opt. 15, 495. BIALYNXCK1-BIRULA, I. & MYC1ELSK1, J. (1975). Commun. Math. Phys. 44, 129-132. BR1COGNE, G. (1984). Acta Cryst. A40, 410-455. BRYAN, R. K. & SK1LUNG, J. (1986). Opt. Acta, 33(3), 287-299. FAN HAI-FU, ZHONG ZI-YANG, ZHENG CHAO-DE & L1 FANGHUA (1985). Acta Crysr A41, 163-165. FORSLUND, B. (1969). Svensk Naturventenskap 1969, pp. 93-96. Statens Naturvetenskapliga Forskningsrad, Stockholm, Sweden. FRIEDEN, B. R. (1987). Proc. 45th Annual Meeting of the Electron Microscopy Society of America, pp. 14-17. San Fransisco Press. GADRE, S. R. & BENDALE, R. D. (1985a). Curr. Sci. 54(19), 970. GADRE, S. R. & BENDALE, R. D. (1985b). Int. J. Quantum Chem. 27, 311-314. GADRE, S. R., BENDALE, R. D. & GEJJI, S. P. (1985). Chem. Phys. Letr 117(2), 138. GERCHBERG, R. W. & SAXTON, W. O. (1971). Optik (Stuttgart), 34, 275-283. GERCHBERG, R. W. & SAXTON, W. O. (1972). Optik (Stuttgart), 35, 237-246. GULL, S. F. & SKILL1NG, J. (1984). lEE Proc. 131, Part F., No. 6, 646-657. HARRISON, R. W. (1987). Acta Crysr A43, 428-430. JAYNES, E. T. (1957). Phys. Rev. 106, 620-630. MARTIN, D. C. (1988). Proc. 46th Annual Meeting o.f the Electron Microscopy Society of America, pp. 840-841. San Fransisco Press. MORIARTY-SCH1EVEN, G. H., SNELL, R. L., STROM, S. E., SCHLOERB, F. P., STROM, K. M. & GRASDELEN, G. L. (1987). Astrophys. J. 319, 742-753. NARAYAN, R. & NITYANANDA, R. (1986). Annu. Rev. Astron. Astrophys. 24, 127-170. ODELL, J. A., KELLER, A., ATKINS, E. D. T. & MILES, M. (1981). J. Mater. Sci. 16, 3309-3318. PRADERE, P., REVOL, J.-F., NGUYEN, L. & MANLEY, R. ST J. (1988). Ultramicroscopy, 25, 69. ROCHE, E. J., TAKAHASHI, T. & THOMAS, E. L. (1980). In Fiber Diffraction Methods, edited by A. D. FRENCH & K. H. GARDNER. Am. Chem. Soc. Symp. Ser., Vol. 141, pp. 303-313. SAYRE, D. (1952). Acta Crysr 5, 60-65. SHANNON, C. E. (1948). Bell Sysr Tech. J. 27, 379, 623. SHANNON, C. E. & WEAVER,W. (1949). The Mathematical Theory of Communication. Univ. of Illinois Press. SHIOZAKI, A. (1986). Comput. Vision, Graphics Image Process. 36, 1-9. SHORE, J. E. & JOHNSON, R. W. (1980). IEEE Trans. Inf. Theor).', 26, 26-37. SHORE, J. E. & JOHNSON, R. W. (1983). IEEE Trans. Inf. Theoo', 29, 942-943. SILLIMAN, W. J. (1979). Viscous Film Flows with Contact Lines. PhD thesis, Univ. of Minnesota, Minneapolis, MN, USA. THOMAS, E. L., ALWARD, D. B., KINNING, D. J., MARTIN, D. C., HANDLIN, D. L. JR & FEqq'ERS, L. J. (1986). Macromolecules, 19, 2197-2202. WILL1NGDALE, R. (1981). Mon. Not. R. Astron. Soc. 194, 359.