Journal of Electron Microscopy 60(1): 39–46 (2011) doi: 10.1093/jmicro/dfq076 ........................................................................................................................................................................................................................................................
Physical: Technical Report
An automatic method of detecting and tracking fiducial markers for alignment in electron tomography Meng Cao 1, Akio Takaoka 2, Hai-Bo Zhang 1, * and Ryuji Nishi 2 1
..............................................................................................................................................................................................
Abstract
We presented an automatic method for detecting and tracking colloidal gold fiducial markers for alignment in electron tomography (ET). The second-order derivative of direction was used to detect a fiducial marker accurately. The detection was optimized to be selective to the size of fiducial markers. A preliminary tracking result from the normalized correlation coefficient was refined using the detector. A constraint model considering the relationship among the fiducial markers on different images was developed for removing outlier. The three-dimensional positions of the detected fiducial markers and the projection parameters of tilt images were calculated for post process. The accuracy of detection and tracking results was evaluated from the residues by the software IMOD. Application on transmission electron microscopic images also indicated that the presented method could provide a useful approach to automatic alignment in ET.
..............................................................................................................................................................................................
Keywords
fiducial marker, detection and tracking, automatic alignment, electron tomography
..............................................................................................................................................................................................
Received
10 June 2010, accepted 16 October 2010, online 12 November 2010
..............................................................................................................................................................................................
Electron tomography (ET) is a powerful microscopic imaging tool to obtain three-dimensional (3D) internal structure of samples from a tilt series of projection images collected by the transmission electron microscope (TEM). To obtain an accurate tomographic reconstruction, the projection images should be aligned accurately before performing the reconstruction. Methods of image alignment in ET can be classified into two categories: marker-based alignment [1,2] and marker-free alignment [3–5]. The former guarantees better accuracy and consistency among the whole image set and is therefore widely used if the interference of markers is acceptable [6,7]. Compared with tedious manual selection and identification of fiducial markers, an automatic
technique is much more convenient and is therefore highly desired in automatic ET [8,9]. Some work has been reported [10–13] for the automatic detection and tracking of fiducial markers. However, fiducial models from automatic tracking often have to be refined manually, and research in the automatic and accurate realization of detecting and tracking fiducial markers is still in progress [14,15]. Pattern matching between a template and an original image was widely used for detecting fiducial markers. To suppress background noise, mask smoothing and low-pass filtering were often introduced, which would make the response of the detector be unsharp and thus decrease the detection accuracy. On the other hand, on tilt images corresponding to different tilt angles of a sample, fiducial markers
........................................................................................................................................................................................................................................................
© The Author 2010. Published by Oxford University Press [on behalf of Japanese Society of Microscopy]. All rights reserved. For permissions, please e-mail:
[email protected]
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
Key Laboratory for Physical Electronics and Devices of the Ministry of Education, Department of Electronic Science and Technology, Xi’an Jiaotong University, Xi’an 710049, People’s Republic of China and 2Research Center for Ultrahigh Voltage Electron Microscopy, Osaka University, 7-1 Mihogaoka, Ibaraki, Osaka 567-0047, Japan *To whom correspondence should be addressed. E-mail:
[email protected]
40
J O U R N A L O F E L E C T R O N M I C R O S C O P Y , Vol. 60, No. 1, 2011
@f ddðxÞ ¼f ; @x dx
ð1Þ
where the asterisk denotes the convoluting integral. The Dirac function δ(x) indicates that the derivative is calculated on a point and a detector based on normal derivatives is therefore suitable for an infinitive small point. To construct the detector that is sensitive to a feature with a certain size, derivatives in scale space should be used. The Dirac function in Eq. (1) is then replaced by a smooth kernel. The smooth kernel ensures that the derivative is carried out on a certain scale level rather than on a point. A suitable parameter for the smooth kernel can make the detector have a high response for a feature with a certain size. The flowchart of our method is shown in Fig. 1. The detected fiducial markers are preliminarily tracked using the normalized correlation coefficient (NCC) between consecutive images. Refinement with an improved accuracy is then achieved based on the detection operator. By considering the relationship among the fiducial markers on different tilt images, a constraint model is further developed to eliminate the misplacements in tracking results. The robust random sample consensus (RANSAC) algorithm is used to find the model parameters. Finally, projection parameters for all tilt images are calculated to refine the results and the accuracy is evaluated using the software package IMOD [13] especially for the ET processing of electron microscopic images.
Detection of fiducial markers on a reference image is the first step for marker-based alignment. In ET, colloidal gold particles often serve as a kind of standard fiducials. Projections of the gold particles are dark round spots with almost constant size on all tilt images. At a dark spot center, two important properties can be used for detection. First, a dark spot center is the trough of the gray level, which means that the second-order directional derivative of the gray level reaches the local maximum in any direction. Second, the spot is round so that the second-order directional derivatives are nearly constant in all directions. The second-order directional derivative Iθθ of an image I (x,y) in direction θ is Iuu
I ¼ ½cos u sin u xx Iyx
Ixy Iyy
cos u ¼ VTu MVu : ð2Þ sin u
Here, a subscript denotes a derivative with respect
Fig. 1. The flowchart of the detection and tracking process.
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
are projections of the same group of colloidal gold particles. There are some constraint conditions on the arrangements of fiducial markers. The epipolar geometry constraint involving two tilt images was introduced for tracking fiducial markers [16]. Here, a constraint model that involves more tilt images should be more effective. In this note, therefore, we present an automatic method to accurately detect and to robustly track fiducial markers and describe its application on tilt TEM images. An accurate detection of the marker with a certain size requires the detector to be maximum for the given size. Here, derivatives in scale space can be used to optimize the detection. Normally, a derivative can be seen as
M. Cao et al. Automatic detection and tracking of fiducial markers
to the corresponding variable. For a given point, the maximum and minimum values of Iθθ for all θ are just two eigenvalues, λmax and λmin, of the matrix M. The value of λmin reaches the local maximum at the center of the dark spot and therefore can be used as an evaluation of darkness. The difference between the two eigenvalues reflects the roundness of the spot. Thus, we detect the projections of the gold particles by the detector D ¼ lmin expðlmin lmax Þ:
ð3Þ
s Ixx ¼ Iðx; yÞ Gsxx ðx; yÞ s Iyy ¼ Iðx; yÞ Gsyy ðx; yÞ s Ixy
¼ Iðx; yÞ
ð4Þ
Gsxy ðx; yÞ;
where 2 x2 s2 x þ y2 exp 2ps4 2s2 2 y2 s2 x þ y2 ¼ exp 2ps4 2s2 2 xy x þ y2 : ¼ exp 2ps4 2s2
Gsxx ¼ Gsyy Gsxy
ð5Þ
For a dark spot, whose gray value is zero inside and one outside, the value of detector D is r2 r2 D ¼ 2 exp 2 ; ð6Þ 2s 2s where r is the radius of thepspot. Evidently, D will ffiffiffi reach its maximum if s ¼ r= 2. Figure 2a shows a TEM image with fiducial markers. Here we use the sample dataset from http://bio3d.colorado.edu/imod as an example. The map of D is shown in Fig. 2b. High response values for D appear at centers of fiducial markers and the structures of the background are suppressed because the detector is optimized for round fiducial markers. By using a non-maximum suppression method [17], the fiducial markers are detected accurately as shown in Fig. 2a. Furthermore, we look at the robustness of the detection with respect to
noise. The same TEM image, but with additional random noise, shown in Fig. 2c, is used as an input and the map of detector D is given in Fig. 2d. It can be seen that the detector can still work effectively for the noised image. The detected fiducial markers are tracked preliminarily using the NCC between consecutive images. However, there are often small deviations in the results of NCC. These deviations may accumulate and consequently lead to unacceptable errors. Figure 3 shows an example of this kind of deviations. For the fiducial marker circled in Fig. 3a, the tracking results on the selected images at tilt angles 20°, 40° and 60° are shown in Fig. 3b–d, respectively. There is an obvious deviation in Fig. 3b. The deviation becomes stronger in Fig. 3c and the position of the marker is completely wrong in Fig. 3d. A method to avoid the accumulation of deviations is to refine the position of fiducial marker by searching the local maximum of D in Eq. (3) near the preliminary result. Figure 3e–h demonstrates the effect of the refining strategy. We see that the refinement adjusted markers to the spot centers and thus avoided the accumulation of errors. The refinement makes the positions of the fiducial markers be spot centers. However, misplacement might occur if high-contrast features perturb a spot. To find out the misplacements, we developed a constraint model for the positions of fiducial markers on different images. Assume that rj = [Xj, Yj, Zi]T is the 3D coordinate of the jth fiducial point at the 0° tilt angle and pij = [xij, yij ]T is its projection on the ith tilt image. The projection model, which considers both the geometry of projections and changes in the specimen during the recording, holds [18] pij ¼
ai di
bi ei
¼ A i r j þ di ;
2 3 Xj ci 4 5 Dxi Yj þ fi Dyi Zj ð7Þ
where Ai is the projection matrix for the image i and di is the translation after projection into the xoy plane. Vector rj can be arbitrarily shifted and di can be adjusted to have the same pij. This degeneracy can be eliminated by just putting d0 = 0. Furthermore, combining the projection models of
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
To detect fiducial markers effectively, the value of D is required to be maximum for a certain size of fiducial markers. It can be derived from the Gaussian derivatives. The second-order Gaussian derivatives with a scale level σ are given by
41
42
J O U R N A L O F E L E C T R O N M I C R O S C O P Y , Vol. 60, No. 1, 2011
Fig. 3. Selected tracking results using the NCC method and their refinements using the value of detection operator D. The tilt angle for (a), (b), (c) and (d) is 0°, 20°, 40° and 60°, respectively. The derivations can be reduced by the refinement, as shown in (e)–(h).
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
Fig. 2. (a) A TEM image with fiducial markers and (b) the map for the value of detection operator D in Eq. (3). High response values for D appeared at centers of fiducial markers and the position of the fiducial were detected successfully, as shown in (a). (c) The same TEM image with random noise and (d) the map of the value of detection D. It can be seen that the detection can still be effective for the noised image.
M. Cao et al. Automatic detection and tracking of fiducial markers
images 0 and i together, we have 2
3 2 x0j a0 6 y0j 7 6 d0 6 7 6 4 xij 5 ¼ 4 ai yij di
b0 e0 bi ei
3 2 3 c0 2 3 0 Xj 6 7 f0 7 74 Yj 5 þ 6 0 7: 4 Dxi 5 ci 5 Zj fi Dyi
ð8Þ
Eliminating Xi, Yi and Zi leads to ui1 x0j þ vi1 y0j þ ui2 xij þ vi2 yij ¼ wi ;
ð9Þ
x2j y2j
¼
r1 r5
r2 r6
r3 r7
2 3 x0j 7 r4 6 6 y0j 7 þ r9 : r8 4 x1j 5 r10 y1j
ð10Þ
The constraint model in Eq. (10) can be used to pick out the displacements in tracking results. The parameters r1 to r10 can be calculated from more than five groups of matched fiducial markers. Here, we use the RANSAC algorithm that is robust to outliers to calculate the parameters [19]. In this method, an estimated model is calculated from randomly selected five groups of matched fiducial markers. All remaining markers are then tested by this model. Markers that fit the model with errors bigger than a given threshold are considered as outliers. A number of estimated candidate models were calculated using this method and one with the smallest error was selected as the final constraint model. The constraint model enhances the robustness of the tracking method. A projection of a gold particle is difficult to be distinguished if it is covered by other high-contrast features. At the same time, the markers on the subsequent images with a stained reference are difficult to be tracked accurately. However, the constraint model can correct an incorrect position of a marker with the value p2j calculated from p0j and p1j, using Eq. (10). The correct positions of markers that fit the constraint model are consistent, but the incorrect ones are not. Therefore, the outliers of the constraint model expose the incorrect position and make the tracking method robust.
After obtaining the whole set of positions of the fiducial markers, we perform a post process by checking the projection models in Eq. (7) for all tilt images. First, we assume that there are M images totally and Ni fiducial markers were obtained on image i. Putting the obtained values of pij back into P Eq. (7), we have 2 M i¼1 Ni equations. The unknown variables in these equations are the elements of matrix Ai (i = 0, … ,M − 1), vectors di (i = 1, … ,M) and the 3D positions [Xj,Yj,Zj ], ( j = 0, … ,N0 − 1). The number of the unknown variables is 8M + 3N0 − 2, which is normally less than the number of equations in practice. Consequently, the overdetermined equations can be solved using a least squares method. A fiducial marker with a big error is considered as an outlier and its correction is then calculated from the corresponding projection model. This fiducial marker is further refined using the value of D if there is a local maximum nearby, or discarded if not. The position data of fidcuial markers can be exported to an IMOD model file for further alignment process. For the former used dataset, we calculate errors using the software IMOD. The tracked fiducial markers on ±60° tilt images are shown in Fig. 4a and b, respectively. The mean residues given by the software IMOD are plotted versus the tilt angles in Fig. 4c. It can be seen that the fiducial markers are accurately tracked. We have also tested our method on another dataset of an integrated circuit (IC) sample [20]. The 0° tilt TEM image of a rod-shaped IC sample and the map of its detector D are shown in Fig. 5a and b, respectively. Again, high response values for D occur at centers of fiducial markers. We selected some fiducial markers as references and tracked their positions on the tilt series images which were recorded with an interval of 1° over the range from −90° to 89°. The mean residues calculated using the software IMOD are plotted in Fig. 5c, and a successful tracking is obtained. Note that the residues in Fig. 5c are bigger than those in Fig. 4c. Here, since colloidal gold particles on the surface of the rod-shaped sample have a complex spatial arrangement, it may be more difficult to track their projections than those in a simple planar sample. The accuracy of our method should depend on some properties of input tilt images. The detector D
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
determining the geometric relations between fiducial markers on image 0 and i. Putting i = 1 and 2, we have two equations as Eq. (9). Solving x2j and y2j from these two equations, we finally obtain the constraint model as
43
44
J O U R N A L O F E L E C T R O N M I C R O S C O P Y , Vol. 60, No. 1, 2011
has a high response value for a round dark spot with smooth background. However, if the background contains too much high-contrast disturbance, it will be difficult to detect fiducial markers accurately, especially if structures of background have the scale levels same with the markers. Improvement is still required to detect a ‘camouflaged target’ in cluttered backgrounds. In addition, the constraint model in Eq. (10) is based on the projection model in Eq. (7) that is only a linear model. A more general model including non-linear distortion should be considered in further investigation. In summary, the presented novel method can accurately detect positions of markers. In addition, refinement with the detection operator enhances the accuracy of tracking results. Furthermore, the constraint model can find out and correct incorrect
positions of markers. Therefore, our method may provide a technique of accurate detection and robust tracking for fiducial markers. We believe that it will be applicable for the automatic alignment in ET.
Funding This work was supported by ‘Nanotechnology Network Project of the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan’ at the Research Center for Ultrahigh Voltage Electron Microscopy, Osaka University (Handai Multi-Functional Nanofoundry), the Natural Science Basic Research Plan in Shaanxi Province of China (No. 2009JQ8001) and Specialized Research Fund for the Doctoral Program of Higher Education of China (No. 20070698013).
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
Fig. 4. The tracking results on (a) −60° tilt and (b) 60° tilt TEM images. (c) Mean residues calculated with the software IMOD under different tilt angles.
M. Cao et al. Automatic detection and tracking of fiducial markers
45
.......................................................................................................................
References .......................................................................................................................
1 Lawrence M C (1992) Least-squares methods of alignment using markers. In: Frank J (ed.), Electron Tomography, pp 197–204 (Plenum, New York). .......................................................................................................................
2 Levine Z H, Volkovitsky A, and Hung H K (2007) Alignment of fiducial marks in a tomographic tilt series with an unknown rotation axis. Comput. Phys. Commun. 176: 694–700.
.......................................................................................................................
3 Sorzano C O S, Messaoudi C, Eibauer M, Bilbao-Castro J R, Hegerl R, Nickell S, Marco S, and Carazo J M (2009) Marker-free image registration of electron tomography tilt-series. BMC Bioinformatics 10(124): 1–11. .......................................................................................................................
4 Hayashida M, Terauchi S, and Toshiyuki F (2010) Automatic coarse-alignment for TEM tilt series of rod-shaped specimens collected with a full angular range. Micron 41: 540–545. .......................................................................................................................
5 Brandt S S and Ziese U (2006) Automatic TEM image alignment by trifocal geometry. J. Microsc. 222: 1–14. .......................................................................................................................
6 Huang Z W, Chen M K, Li K P, Dong X D, Han J D, and Zhang Q F (2010) Cryo-electron tomography of Chlamydia trachomatis gives a clue to the mechanism of outer membrane changes. J. Electron Microsc. 59: 237–241.
7 Andersson B V, Herland A, Masich S, and Inganas O (2009) Imaging of the 3D nanostructure of a polymer solar cell by electron tomography. Nano Lett. 9: 853–855. .......................................................................................................................
8 Dierksen K, Typke D, Hegerl R, Koster A J, and Baumeister W (1992) Towards automatic electron tomography. Ultramicroscopy 40: 71–87. .......................................................................................................................
9 Kobayashi A, Fujigaya T, Itoh M, Taguchi T, and Takano H (2009) Technical note: a tool for determining rotational tilt axis with or without fiducial markers. Ultramicroscopy 110: 1–6.
.......................................................................................................................
10 Ress D, Harlow M L, Schwarz M, Marshall R M, and Mcmahan U J (1999) Automatic acquisition of fiducial markers and alignment of images in tilt series for electron tomography. J. Electron Microsc. 48: 277–287. .......................................................................................................................
11 Brandt S S and Heikkonen J (2000) Automatic alignment of electron tomography images using markers. Proc. SPIE 4197: 277–287. .......................................................................................................................
12 Amat F, Moussavi F, Comolli L R, Elidan G, Downing K H, and Horowitz M (2008) Markov random field based automatic image alignment for electron tomography. J. Struct. Biol. 161: 260–275.
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
Fig. 5. (a) The 0° tilt TEM image of an IC sample with fiducial markers and (b) the map for the value of detection operator D in Eq. (3). (c) Mean residues calculated with the software IMOD under different tilt angles.
46
J O U R N A L O F E L E C T R O N M I C R O S C O P Y , Vol. 60, No. 1, 2011
.......................................................................................................................
.......................................................................................................................
13 Kremer J R, Mastronarde D N, and Mcintosh J R (1996) Computer visualization of three-dimensional image data using IMOD. J. Struct. Biol. 116: 71–76.
17 Neubeck A and Van Gool L (2006) Efficient non-maximum suppression. In: Proceedings of 18th International Conference on Pattern Recognition, pp 850–855 (Hong Kong).
.......................................................................................................................
.......................................................................................................................
14 Friedrich H, de Jongh P E, Verkleij A J, and de Jong K P (2009) Electron tomography for heterogeneous catalysts and related nanostructured materials. Chem. Rev. 109: 1613–1629.
18 Mastronarde D N (2006) Fiducial marker and hybrid alignment methods for single- and double-axis tomography. In: Frank J (ed.), Electron Tomography: Methods for Three-Dimensional Visualization of Structures in the Cell, pp 163–185 (Springer, Berlin).
.......................................................................................................................
.......................................................................................................................
15 Cantele F, Paccagnini E, Pigino G, Lupetti P, and Lanzavecchia S (2010) Simultaneous alignment of dual-axis tilt series. J. Struct. Biol. 169: 192–199.
19 Fischler M A and Bolles R C (1981) Random sample consensus – a paradigm for model-fitting with applications to imageanalysis and automated cartography. Commun. ACM 24: 381–395.
.......................................................................................................................
.......................................................................................................................
16 Brandt S, Heikkonen J, and Engelhardt P (2001) Multiphase method for automatic alignment of transmission electron microscope images using markers. J. Struct. Biol. 133: 10–22.
20 Cao M, Zhang H B, Li C, and Nishi R (2009) Effect of sample structure on reconstruction quality in computed tomography. Rev. Sci. Instrum. 80(026104): 1–3.
Downloaded from http://jmicro.oxfordjournals.org/ at Xi'an Jiaotong University on November 19, 2011
