Camera Motion Recovery without Correspondence from Microrotation Sets in Wide-Field Light Microscopy Sami S. Brandt and Markos Mevorah Laboratory of Computational Engineering Helsinki University of Technology
[email protected],
[email protected]
Abstract. This paper considers 3D camera motion recovery from tomographic projections, where no correspondence information is required. Instead of using feature matching to find the motion, relative to the object, we will show for the first time how the 3D orientation of the camera, can be directly recovered from the image intensity measurements, by the Bayesian inversion theory. In this study, we assume calibrated affine camera geometry, and solve the translation parameter separately in the preprocessing step. The method is applied for micro-rotation sets of cells, taken by a conventional wide-field light microscope supplied with a cell rotation cage.
1
Introduction
Computerized tomography refers to the imaging of an object from transmission data, collected by illuminating the object from many different directions. The application of tomography or similar techniques in micro-rotation sets of biological objects, is a novel methodology for the biologists to inspect the interior structure of the object, such as non-adherent live cells [5],[6]. The final goal is to reconstruct the 3D object from projections, obtained by a wide-field light microscope, but to reach the objective the projection directions must be solved. We refer to the latter problem, as the motion estimation problem, and consider it in this paper. Usually, something is known about the image acquisition setting of the microscope [1], giving some background knowledge for the motion recovery. Standard approaches for solving the motion estimation problem in electron tomography, referred to as alignment of the images, are proposed in [2],[3], where either fiducial or non-fiducial interest points on the images are used. In wide-field light microscopy, like in the application considered in this paper, it is difficult to extract enough interest points, in order to construct reliable matching method. Therefore, we follow an alternative approach, proposed in [4], where the imaging geometry is solved, simultaneously with the volume reconstruction problem, by Bayesian inversion theory. In theory, no reference feature points or prior knowledge of the motion are needed. This approach however is relatively complex
computationally, compared to the previous approaches, so only 2D reconstruction examples were reported in [4]. In this paper, we pose the motion estimation problem in the 3D space. In other words, we consider markerless, correspondance free method for camera motion recovery. The method is especially suitable, for wide-field light microscope and micro-rotation sets, as the one we use in our experiments. To obtain an initial guess for the motion parameters, the approximate tilt angles are first estimated from the auto-correlation function, after which we segment the object and extract the center-of-mass, which is used as an estimate for the translation parameters. After the preprocessing steps, we optimize the motion parameters by minimizing the statistical criterion [4], which will lead us to minimising the error between the modeled images, and the actual microscopic measurements. The final stage for our algorithm, involves selecting the gauge freedom, so that we finally obtain meaningful uncertainty estimates for the parameters. This paper is organized as follows. In Section 2, we propose the pre-processing steps for the micro-rotation sets, in order to obtain rough estimate for the 3D motion of the object. In Section 3, the actual motion estimation problem is set and the methodology is introduced. In Section 4, we propose how the gauge freedom should be set in this application, so that uncertainty can be reliably characterized. In Section 5, we report our experiments with a real test image sequence representing a mitosis of a cell. The conclusions are in Section 5, together with some insights for future directions.
2
Preprocessing
In this section, we will show how we solve the initial guesses for the tilt angles and estimates for the translation parameters. In addition, we propose how the object can be segmented from the micro-rotation sets. 2.1
Fundamental Period
To compute the initial estimates to the tilt angles, we first compute the fundamental period of the rotation, by inspecting the intensity values in the microrotation image set. We store the images in a 3D array I, where the third dimension of I displays the progression in time. By taking one pixel location in all the images, we can treat the related time series as a 1D signal. Hence, for the pixel k, we can compute the auto-correlation function
wk (n) =
1 nmax − n
2nmax X−1
sk (i)sk (n + i) ,
(1)
i=0
where s is the mean-corrected 1D signal, over the time-image index n, i.e. sk (n) = I(xk , y k , n), where xk , y k are the coordinates for the P pixel k. From 1 k (1), we obtain the mean auto-correlation function w(n) = kmax k w (n), see Fig.1(a).
60
1950
5th rotation 1900
40
extracted location 20
derivative
autocorrelation score
1850
4th rotation
1800
3rd rotation 1st rotation
1750
2nd rotation
0
−20 1700
−40 1650
1600
0
10
20
30
40
50
60
70
80
90
100
−60
0
10
20
30
40
50
60
70
80
90
100
time lag
time lag
(a)
(b)
Fig. 1. (a) The mean auto-correlation function for the mitosis set; (b) the derivative of the mean autocorrelation function
The fundamental period N0 , corresponds to the location of the first maximum in the mean auto-correlation function. The maximum location is computed, by extracting the location where the derivative of w(n) has its first zero crossing from positive to negative (Fig.1(b)). The found integer location is interpolated by fitting a parabola to the maximum and its two neighbouring points. 2.2
Segmentation of the Object
In the micro-rotation sets considered in this paper, the object is located in a central area of each image, while there is relatively large empty surrounding space, as Fig.2(a) illustrates. As we will see later, the computational complexity of our method crucially depends on the image size, so we segment, crop and finally downsample them. As far as the segmentation is concerned, we propose to inspect the differences in intensity values over time, since we are studying a bounded rotated object. Let the absolute difference image S be S(x, y) = |I(x, y, 2) − I(x, y, 1)| + |I(x, y, 3) − I(x, y, 2)| + . . . . . . + |I(x, y, n) − I(x, y, n − 1)| ,
(2)
where the background values are expected to be close to zero (Fig.2(b)), whereas the high values indicate the object structure. The problem here is how to find the threshold value, so that it would divide the image into the background and the object values. Since the background values are concentrated to the values near to zero, we select the threshold by finding the maximum bin value and fit a line to the 15% of the number of bins located right after the maximum, where the threshold is taken as the position where the line crosses the x-axis in the histogram.
The threshold divides S into a binary mask image, one indicating the object and zero the background. To remove noise from the object boundary, we filter the mask image by the dilation and erosion operations [11]; we denote the filtered mask image by M (Fig.2(c)). 2.3
Center Of Mass in the Rotation Origin
In this paper, we assume that the object rotates around its 3D center-of-mass. Assuming that we have uniformly distributed tilt angles, the projections of the center-of-mass of the 3D object, should coincide with the mean center-of-mass, computed from the corresponding 2D images over a complete multiple of revolutions. The identification of the projection centre is therefore equivalent to solving the translation parameters. Only the rotation refinement is hence needed in Section 3. The coordinates for the center-of-mass are obtained from P
k,n xCOM = P
M(xk , y k )I(xk , y k , n)xk
k,n
M(xk , y k )I(xk , y k , n)
and P
k,n yCOM = P
M(xk , y k )I(xk , y k , n)y k
k,n
M(xk , y k )I(xk , y k , n)
.
After the center-of-mass is computed, we crop and resample the images, so that the center-of-mass becomes the center of the cropped images. The cropped images are finally downsampled by a selected factor, which defines the resolution and computational complexity of the problem. The steps of the procedure described in this Section, are illustrated in Fig.2.
3
Solving Motion Estimates
The method described in this section is based on the work [4], but the method is here extended to the 3D case. 3.1
Projection Model
The projection model is described by the formula m = Af + n ,
(3)
where m represents projection measurements (all the images stacked into the vector), f represents the object densities (voxel values) and n is the noise vector. The transformation matrix A projects the 3D object on the 2D image planes.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2. The micro-rotation set preprocessing illustrated. (a) Original image from the set; (b) the absolute difference image S; (c) the binary filtered mask image M; (d) segmented image; (e) center-of-mass coordinates superimposed; (f) the final cropped image, where the center-of-mass is in the centre
We model the projection as attenuation over rays in space, i.e., for the pixel k, we can write the projection rule as
ak f =
Z
f (x)dL = Lk
X l
ξl
Z
g(x − xl )dL = Lk
X
ξl akl ,
(4)
l
where akl =
Z
g(x − xl )dL = gσ (d(xl , Lk )) ,
(5)
Lk
and gσ is the 1D gaussian density function, d(xl , Lk ) is the distance between the point l and the projection ray k, and f = (ξ1 ξ2 . . . ξL )T . In other words, the element akl describes the contribution of the voxel l to the attenuation measurement (pixel value) k.
3.2
Motion Parameterisation
We will now describe, how A depends on the motion parameters θ. To see the dependence, we only need to form d(xl , Lk ) in (5). Hence, using the geometric projection formula ul = si PRiα,β,γ xl + ti ,
(6)
the voxel centre xl projects onto the point ul , where we assume that si = 1 and ti = const. Let the pixel k have the 2D coordinates vk . Now, if we assume the orthographic projection model, the distance d(xl , Lk ) = d(ul , vk ). 3.3
Least Squares Error Minimization
In the projection model (Section 3.1), we assume that the noise is normally distributed, hence, our goal is to minimize the least squares error c(f , θ) = kAθ f − mk2 =
X
2i = (Aθ f − m)T (Aθ f − m) = T ,
(7)
i
where = (Aθ f − m). For simplicity, we use the diagonal Tikhonov regularized solution for f , i.e., −1
AT θm ,
−1
2 AT θ m − mk ,
ˆ f = AT θ Aθ + λI
(8)
where λ is the regularization parameter. By substituting ˆ f into Eq.7, the objective function obtains the form c(θ) = kAθ AT θ Aθ + λI which depends only on θ. 3.4
(9)
Minimization Details
We perform the least squares error minimization by the trust-region iterative method, as it is implemented in Matlab Optimization toolbox [7]. Since the optimization method is a second-order method, we need to compute the Jacobian of , with respect to the parameters. The construction is relatively complicated, since we need to solve several large linear systems of the form (AT θ Aθ +λI)w = b, on each iteration. There are practically two different approaches to solve the linear systems: direct methods and iterative Krylov-subspace methods. We selected the direct approach, since they are easy to implement, fast and suitable for relatively large sparse systems. We solved the systems by the Cholesky decomposition, which has
the advantage that we obtain the Cholesky factor, which can be used for solving all the linear systems at the current iteration, by simple back substitutions. Furthermore, we used the reverse-Cuthill ordering algorithm, to obtain more efficient performance, as it clusters non-zero entries close to the diagonal, and hence makes the computation of the corresponding Cholesky factor lighter. As initial guess for the parameter optimization, we assumed ideal tilts around the z-axis, that is, we initialize the parameter α to zero, the parameter β to π2 2π and the tilt parameter γ to the ideal tilt angles with the increment N , starting 0 from zero. In the optimization, we used the first image as the reference (trivial gauge), i.e., its rotation matrix was fixed to identity. For each image, we have 3 parameters to solve, thus, in total we have 3 × (Nimages − 1) parameters.
4
Gauge Projection
In the optimization process, we fix the coordinate system by setting one view as the reference, i.e., use a trivial gauge as discussed in [8]. After the optimization, in order to obtain meaningful interpretation for the uncertainty estimates, we will project the gauge, so that no single view has a special role. The selection of the better gauge is proposed as follows. First, we transform the trivial gauge to another one, by searching for the plane to which the viewing vectors lie closest in the least squares sense, and transforming that plane into the xy-plane. Let di be the distance between the ith viewing vector xi and the tilt plane, thus, we mimimize
c=
X i
d2i
=
P
i
nT n T xi xT i n = nT n
T i xi xi nT n
P
where n is the normal vector to the plane and W = it holds
n
P
i
=
nT Wn , nT n
(10)
xi xT i . For the minimum,
∂c 2Wn 2nT Wn =0⇔ T − n = 0 ⇔ Wn = λn . ∂n n n nT n
(11)
As we want to minimise c, the searched plane normal vector is the eigenvector corresponding to the minimal eigenvalue. The rotation matrix which transforms the found plane normal onto the z-axis, is obtained as R = UT , where U contains the eigenvectors of W, sorted into descending order of the eigenvalues and their signs are selected so that det(U)=1. The transformed parameters are obtained by decomposing the new rotation matrix Rz (α)Ry (β)Rz (γ)RT = Rz0 y0 z0 (α0 , β 0 , γ 0 ) = Rz0 (α0 )Ry0 (β 0 )Rz0 (γ 0 ) ,
(12)
where α0 , β 0 , γ 0 and z 0 and y 0 belong to the transformed coordinate system. Until now, we have put constrains on two parameters describing the orientation of the
reference plane. The third constraint is set, by the requirement that the mean of the initial tilts must be equal to the mean of the parameter γ 00 , in the final coordinate system, which describes the rotation (tilt) on the reference plane. We i i hence identify the constant tilt δγ 0 for the transformation γ 00 = γ 0 + δγ 0 , so that γ 00 = γ0 , where γ0 represents the initial guess for the tilt parameter. We get γ 00 = γ 0 + δγ 0 = γ 0 + δγ 0 = γ0 ⇒ δγ 0 = γ00 − γ 0 .
(13)
Furthermore, (12) and (13) yield Rz00 y00 z00 (α00 , β 00 , γ 00 ) = Rz0 (α0 )Ry0 (β 0 )Rz0 (γ 0 )Rz0 (δγ 0 ) = Rz (α)Ry (β)Rz (γ)RT Rz0 (δγ 0 ) . (14) From (14) and the fact that Rz00 y00 z00 (α00 , β 00 , γ 00 ) = Rz00 (α00 )Ry00 (β 00 )Rz00 (γ 00 ) ,
(15)
we get the final gauged parameters α00 , β 00 and γ 00 . The uncertainty on the estimated parameters α00 , β 00 and γ 00 , is characterized by the gauged covariance matrix. We approximate it by computing the Jacobian J, of the cost function evaluated at α00 , β 00 and γ 00 , and set Cθ ' 2σ 2 H† ' σ 2 (JT J)† , where ·† refers to the Moore-Penrose pseudoinverse, and the variance is approximated as σ 2 '
5
ˆ c(θ) N −dim(θ)−dim(f ) .
Results
A micro-rotation set representing a cell mitosis was used for testing. This set consisted of 90 images of 195×212 pixels obtained by wide-field light microscope, but they were cropped and finally downsampled into the size of 20×20 pixels, because of memory limitations. By applying the autocorrelation method, we automatically found that there were five complete revolutions in the image set; and the fundamental period was found to be 18.1 images. The details of the optimization, are shown in Table 1. In Fig.3, we show the final optimized parameters, where the gauge has been updated (Section 4), superimposed with the initial guess values for each of the three angle parameters. The error bars correspond to 95% confidence intervals. The results show that the camera motion, can be relatively accurately recovered by the proposed method, as the uncertainty is normally less than a degree.
6
Conclusions
In this paper, we proposed a novel approach for solving the motion recovery problem in the cell rotating application, where we used a tomographic projection assumption. Instead of using feature correspondences, the 3D cell rotation
Table 1. Optimization details based upon the microscopic image set. Optimization Details
Mitosis
Number of Images Number of Parameters Number of Iterations Number of Linear Systems Total Time
90 267 34 9112 27.2 hours
information is directly recovered from the image intensity information, using Bayesian inversion theory. The approach is relatively complex computationally, where the main computational load is due to the large linear systems that have to be solved. The results show that the 3D cell rotation is recovered with an accuracy level less than a degree for the most parameter estimates, though the images were currently downsampled by the factor of six, due to computational limitations. In the future work we are therefore going to investigate the possibilities of increasing the efficiency of the algorithm, to obtain even higher level of accuracy.
References 1. P. Engelhardt: Electron tomography of chromosome structure. In Encyclopedia of analytical chemistry, Vol. 6:4948-4984, 2000. 2. S. Brandt, J. Heikkonen, and P. Engelhardt: Multiphase method for automatic alignment of transmission electron microscope images using markers. In J. Struct. Biol., 133(1):10-22, 2001. 3. S. Brandt, J. Heikkonen, and P. Engelhardt: Automatic alignment of transmission electron microscope tilt series without fiducial markers. In J. Struct. Biol., 136(3):201-213, 2001. 4. S. Brandt, V. Kolehmainen: Motion without correspondence from tomographic projections by bayesian inversion theory. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recogntion (CVPR 2004), Vol I, pp. 582-587, Washington DC, June. 5. Lizundia R., Sengmanivong L., Guergnon J., Wang J., Mueller T., Schnelle T., Chalmond B., Langsley G. and Shorte S.L.: Using micro-rotation imaging to study JNK-mediated cell survival inTheileria parva-infected B-lymphocytes, ISSN 00311820, p629-636, 2005. 6. Korlach J., Reichle C., Mueller T., Schnelle T. and Web W.W.: Trapping, deformation,and rotation of giant unilamellar vesicles in octode dielectrophoretic field cage, Biophysical Journal, vol 89, July 2005, 554-562. 7. Coleman, T.F. and Y. Li, ”An Interior, Trust Region Approach for Nonlinear Minimization Subject to Bounds,” SIAM Journal on Optimization, Vol. 6, pp. 418-445, 1996. 8. B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon: Bundle adjustment a modern synthesis. In Vision Algorithms: Theory and Practice, Springler-Verlag LNCS 1883, 2000.
parameter β
parameter α 10
initial guess uncertainty estimated
93
initial guess uncertainty estimated
92
5
degrees
degrees
91
90
89
0
88
87 −5 10
20
30
40
50
60
70
80
10
90
20
30
40
50
60
70
80
90
image index
image index
(a)
(b) parameter γ 350
300
degrees
250
200
150
100
initial guess uncertainty estimated
50
0
10
20
30
40
50
60
70
80
90
image index
(c) Fig. 3. Estimates for the parameters α, β and γ in the mitosis set. The initial parameter values are shown by crosses, and the optimized parameters after the gauge projection, with dots; the errorbars correspond to the 95% confidence interval
9. R. Snay: Reducing the profile of sparse symmetric matrices. In Bulletin G´ eod´ esique, 50:341-352, 1976. 10. G. Golub, and C. Van Loan: Matrix Computations, The Johns Hopkins University Press, 1996. 11. R. Gonzalez, and R. Woods: Digital Image Processing, Addison-Wesley, 1993.
