TR-CS-97-20
Finite Element Thin Plate Splines for Surface Fitting M. Hegland, S. Roberts and I. Altas November 1997
Joint Computer Science Technical Report Series Department of Computer Science Faculty of Engineering and Information Technology Computer Sciences Laboratory Research School of Information Sciences and Engineering
This technical report series is published jointly by the Department of Computer Science, Faculty of Engineering and Information Technology, and the Computer Sciences Laboratory, Research School of Information Sciences and Engineering, The Australian National University. Please direct correspondence regarding this series to: Technical Reports Department of Computer Science Faculty of Engineering and Information Technology The Australian National University Canberra ACT 0200 Australia or send email to:
[email protected] A list of technical reports, including some abstracts and copies of some full reports may be found at: http://cs.anu.edu.au/techreports/
Recent reports in this series: TR-CS-97-19 Xun Qu, Jeffrey Xu Yu, and Richard P. Brent. Implementation of a portable-IP system for mobile TCP/IP. November 1997. TR-CS-97-18 Richard P. Brent. Stability of fast algorithms for structured linear systems. September 1997. TR-CS-97-17 Brian Murphy and Richard P. Brent. On quadratic polynomials for the number field sieve. August 1997. TR-CS-97-16 M. Manzur Murshed and Richard P. Brent. Algorithms for optimal self-simulation of some restricted reconfigurable meshes. July 1997. TR-CS-97-15 Peter Strazdins. Reducing software overheads in parallel linear algebra libraries. July 1997. TR-CS-97-14 Michael K. Ng and William F. Trench. Numerical solution of the eigenvalue problem for Hermitian Toeplitz-like matrices. July 1997.
1
Finite Element Thin Plate Splines for Surface Fitting M. Hegland
Computer Sciences Laboratory, RSISE, Australian National University, Canberra, ACT 0200, Australia.
S. Roberts
School of Mathematical Sciences, Australian National University, Canberra, ACT 0200, Australia.
I. Altas
School of Information Studies, Charles Sturt University, Wagga Wagga, NSW 2678, Australia.
1. Introduction
Surface tting and smoothing splines techniques are widely used in practice to t data arising from many di erent application areas such as meteorology, insurance and stock exchange. A common problem is the approximation of functions of many variables for given values of the function at various points. What makes the problem even more complicated is that the given or observed values may contain noise. We are particularly interested in data mining applications which deal with very large databases. These problems arise in many di erent areas [4]. One basic aim in data mining is to model functional relationships of high dimensional data sets which introduce the \curse of dimensionality". In order to overcome this curse, additive and interaction splines have been used [9]. Generalised additive models, if interaction terms are limited to order two interactions, lead to the determination of coupled surfaces and curves. Thus, an important part of any data analysis algorithm for these problems is the determination of an approximating surface for extremely large data sets. Thin plate splines are frequently used for the estimation of smooth surfaces of the form y = f (1; 2) from observed data. A variational characterisation of the thin plate smoothing splines was proposed by Duchon [5]. He also proposed to use a radial basis function approach to solve this problem. However, this leads to symmetric inde nite dense linear system of equations. It was seen that this system can be reduced to a positive de nite system of equations which can be solved by a conjugate gradient method [11]. Further improvements using ideas from multipole expansions and Lagrange functions [1, 2, 3] lead to methods which are of O(n) or O(n log(n)) in complexity where n is the number of observations. In this work, a new smoothing method is proposed which can be viewed as a discrete thin plate spline. This new approach combines the favourable properties of nite element surface tting with the ones of thin plate splines. In Section 2, the method is introduced which is based on rst order techniques similar to mixed nite element techniques for the biharmonic equation. The numerical solution of the linear system of equations is discussed in Section 3. In Section 4, the technique is illustrated with an example. Conclusions are given in Section 5.
2
M. Hegland, S. Roberts & I. Altas
2. Thin Plate Splines and the Mixed Finite Element Method
The thin plate spline considered here is the function f for which all the partial derivatives of total order 2 are in L2 ( ) (where R 2 ) and which minimises the functional Z n X , 1 2 M (f ) := n (f (xi) , yi) + f1 1 (x)2 + 2f12 (x)2 + f2 2 (x)2 dx
i=1
where x = (1; 2)T and xi = (i;1; i;2)T 2 R 2 ; yi 2 R ; i = 1; : : : ; n denote the measurements. The smoothing parameter depends on the size of the data errors and can be determined, e.g., by generalised cross validation. In order to reduce the smoothness requirements this problem is transformed into a problem using rst order derivatives only. In the following it is assumed that ui; i = 0; 1; 2 are H 1 functions with zero mean such that (rv; ru0) = (rv; u); for all v 2 H 1( ) with zero mean.
(2.1)
An operator P is de ned by
P u := (u0(x1 ); : : : ; u0(xn))T : The functions u1;2 are obtained as the minimum of
J (u; c) = n1 (P u + X c , y)T (P u + X c , y) + (ju1j21 + ju2j21 )
where c = (c0 ; c1; c2)T are constants and where the matrix X is de ned by
X := x1 x1 1 n
T
2 R n;3 :
It follows now that the functional has exactly one minimum if the measurement points xi are not collinear and moreover, there is also exactly one minimum in the set of all u with zero curl, i.e., where @u2 , @u1 = 0: @1 @2 As the curl is zero for this function it follows that u = ru0 with zero mean and it can be seen that the function de ned by f (x) = u0(x)+ c0 + c11 + c2 2 minimises the original functional M (f ) and is thus the thin-plate smoothing spline. Formal proofs of these results can be found in [10]. In practical tests, it was seen that the solution which does not enforce the \curl zero" condition does give solutions which are very similar to the original Duchon splines and it is suggested that these \modi ed" thin plate splines can be used especially as they are much cheaper to compute. In the following the computation of these modi ed thin plate splines will be discussed.
Finite Element Thin Plate Splines
3
First, the nite element approximation is established. This computes the minimum of J h(u; c) which has been obtained from J (u; c) by replacing the operator P with Ph which is de ned by Phu := (uh0 (x1); : : : ; uh0 (xn))T where uh0 is the nite element solution of the boundary value problem (2.1) in a nite element space Xh of H 1-functions with zero mean. The minimum of J h is taken over Xh2 R 3 . The existence and uniqueness of this minimum has been established in [10]. As P is approximated by Ph in the discrete problem and thus the functional J is modi ed one obtains a non-conforming nite element method even though the basis chosen is continuous piecewise bilinear and H 1. If b1(x); : : : ; bm (x) denotes a basis of Xh, one gets the representation
uhj(x) =
m X i=1
uhj;ibi(x)
and with uhj = (uhj;1; : : : ; uhj;m) the discretised boundary value problem (2.1) is reduced to a system of linear equations of the form
Auh0 = B1uh1 + B2uh2 where [A]i;j = (rbi ; rbj ) and [Bk ]i;j = (@bi =@k ; bj ). These equations form linear constraints and the function J h takes the value J h = n1 kN uh0 + X ck2 + (uh1 T Auh1 + uh2 T Auh2 ) where k k denotes the Euclidean norm in R n and [N ]i;j = bj (xi ). This is a quadratic optimisation problem with linear constraints and one gets the equivalent linear system of equations [7]: 2
32
3
2
3
A 0 0 0 ,B1T uh1 0 T h 6 0 7 6 7 6 A 0 0 ,B2 7 6 u2 7 6 0 77 6 6 0 0 M F A 77 66 uh0 77 = 66n,1N T y77 2 R 4m+3 6 4 0 0 F T E 0 5 4 c 5 4n,1X T y5 ,B1 ,B2 A 0 0 wh 0
(2.2)
where c = [c0 ; c1; c2]T as before and w is the Lagrange multiplier vector. Here M = n,1 N T N , F = n,1 N T X and E = n,1 X T X . It can be seen that the system matrix is symmetric inde nite and sparse. In the following the index h denoting the discretisation level will be omitted in order to simplify the notation.
3. Solution of the Equations
The size of the system (2.2) does not depend on the number of observations n and the only thing which is a ected by the data size is the number of rows of the matrices
4
M. Hegland, S. Roberts & I. Altas
N and X . This provides a desirable scalability property as the observations need to be visited once only when the equations are constructed and so the memory requirement is independent of n. The amount of work required is O(n) plus something which only depends on the grid resolution m. As is seen from Equation (2.2) the linear system is sparse and symmetric but inde nite. For such systems the Uzawa algorithm and its variants have been used successfully [6, 8, 12]. If several ajacent elements contain no observation points xi, the Uzawa algorithm can break down as the matrix M becomes singular. This problem is addressed by applying a block row interchange of the third and the last block row of Equations (2.2). For this interchanged system, block Gauss-Seidel iteration leads to 32
2
3
2
3
2
3
0 B1T A 0 0 0 0 uk1+1 k +1 T7 6 6 7 6 7 6 0 0 77 B A 0 0 0 u 76 2 7 6 2 7 k 6 6 7 6,B1 ,B2 A 0 0 7 6 uk+1 7 = 6 0 7 w + 6 6 ,1 0 T 7 : 76 0 7 6 7 6 T k +1 4n X y 5 4 0 0 F E 05 4 c 5 4 0 5 k +1 n,1N T y 0 0 0 M F A w
(3.1)
We will call this \Uzawa algorithm with (row) interchange". As wk appears only on the right hand side this iteration is easily rewritten as an iteration for wk as
Awk+1 = S wk + d
(3.2)
where
S = ,1(FE ,1F T , M )A,1 H; and H = B1 A,1B1T + B2A,1 B2T , and d = n,1 N T y , n,1 FE ,1X T y. Like the classical Uzawa algorithm [8], the Uzawa algorithm with interchange can be interpreted as a Richardson iteration method for wk . As is seen from equation (3.1) one requires to solve four equations with matrix A for each iteration step. As A is a nite element approximation of the 2 dimensional Laplacian these equations can eciently be solved with a conjugate gradient method. This \inexact" Uzawa algorithm with interchange then performs the following iteration: 1. Compute uk1+1 and uk2+1 such that
Auki +1 = BiT wk + rki ; i = 1; 2: 2. Compute uk0+1 such that
Auk0+1 = B1uk1+1 + B2uk2+1 + rk0 : 3. Solve for ck+1 directly (up to rounding errors). Note that this is a 3 by 3 system:
E ck+1 = ,F T uk0+1 + n,1X T y:
Finite Element Thin Plate Splines
5
4. Compute wk+1 such that Awk+1 = ,M uk0+1 , F ck+1 + n,1N T y + rk3 : Here, in a similar manner to [8], the rki are the residuals of the iterative solvers for the uki +1 and the rk3 is the residual of the last step. This method is very similar to the inexact Uzawa algorithm [8]. As stated in [8], if solutions obtained from inner iterations are crude, the resulting solution may not be accurate. By controlling the size of the residuals of the inner iterations one can obtain reasonably accurate solutions from inner iterations. In order to control the size of the residuals we use krki k 1 kn,1 N T y , M uk0 , F ck , Awk k = 1 krk3,1k; i = 0; 1; 2 where rk3,1 has already been evaluated to nd wk at the fourth step of the algorithm. For the residual rk3 the stopping criteria krk3 k 2 kAuk0+1 , B1 uk1+1 , B2 uk2+1k = 2 krk0 k has been used. Convergence Analysis
0
10
−1
10
−2
e
10
−3
10
−4
10
−5
10
−6
10
0
2
4
6
8
10 12 outer iterations
14
16
18
20
Figure 1. Convergence of the Inexact Uzawa Algorithm for = 0:01
The convergence is investigated by comparing the error against the \exact solution" obtained from a direct solver of Equation (2.2) using matlab. The error term has been de ned as ek = ku0 , uk0 k1 + kw , wk k1
6
M. Hegland, S. Roberts & I. Altas
where u0 and w denote the results returned from the direct solver. The outer iteration was stopped when all the changes kwk+1 , wk k1, kck+1 , ck k1 and kuk0+1 , uk0 k1 are less than 10,7. The results of this test are displayed in Figure 1 for the case of uniformly distributed random 2500 data points in [0; 1]2 and the function
f (x; y) = exp(,8[(x , 1=2)2 + (y , 1=2)2): The smoothing parameter was chosen to be relatively large, namely 0.01. The grid for this test problem is 16 by 16. The constants have been chosen as 1 = 0:1 and 2 = 1:6. Convergence Analysis
1
10
0
10
−1
10
−2
10
−3
e
10
−4
10
−5
10
−6
10
−7
10
−8
10
0
5
10
15
20 25 outer iterations
30
35
40
45
Figure 2. Convergence of the Inexact Uzawa Algorithm for = 0:0001. Equation (3.3) solved by direct solver: |, by SOR: - - -.
For values of < 0:001 it was seen that this method did not converge very well. In this case, the equation for the Lagrange multiplier (A , S )w = d is solved by a di erent method. First an approximation R of A , S is obtained using incomplete LU decomposition for the approximation of the A,1 occurring in the formula for S . The iterative method obtained from this is de ned by
Rwk+1 = (R , A + S )wk + d + rk
(3.3)
Finite Element Thin Plate Splines
7
where the inner iteration for R is solved using SOR until a similar stopping criteria as the one above for the residual rk is satis ed. If this equation can be solved exactly, (rk = 0) this iteration was seen to converge very quickly. Further research is in progress to improve the convergence when an iterative method is used to solve the system with matrix R. Convergence results, both for the exact and iterative solution of Equation (3.3) are displayed in Figure 2. The example chosen is the same as before except that = 0:0001 this time. Further investigations are in progress to obtain better approximations of A , S for small . exact
duchon
tps fem
10
10
10
0
0
0
−10 1
−10 1
−10 1
1
0.5
1
0.5
0.5 0 0
0 0
data points for tps
data points for tpsfem 1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0.5
0.5 0 0
1
0
1
0.5
0.5
1
0
0
0.5
1
Figure 3. A nite element discrete thin plate splines example. The
Duchon technique uses 458 points whereas the nite element one uses 45,533 data points.
4. Comparison of the Finite Element Thin Plate Splines with the Ordinary Thin Plate Splines The data model is
yi = f (xi) + i; i = 1; : : : ; n where the random variables i are normally distributed with expectation 0 and standard deviation 5. The components of the data points xi are independently normally distributed with expectation 0:5 and standard deviation 0:25. As an example for f the \peaks" function from matlab has been chosen. The peaks function is formed as a linear combination of several scaled and translated Gaussian distributions. The smoothing parameter is = 10,5. While for the ordinary thin plate splines the data size of 458 takes a reasonable time for processing, the nite element thin plate spline method did not have any diculty with a set 100 times larger. Of course the result produced was better.
8
M. Hegland, S. Roberts & I. Altas
5. Conclusions
In this work, we introduced a non-conforming nite element method which provides
exible and practical tool for smoothing as well as surface tting. It is a scalable method in the sense that the size of data does not e ect the overall size of the system obtained from the discretisation of the equations. On the other hand the assembly of some nite element matrices depend on the data size. The method introduced here forms an alternative to the Duchon thin plate splines technique.
References
[1] Beatson, R.K., Goodsell, G. and Powell, M.J.D., On multigrid techniques for thin plate spline interpolation in two dimensions, The mathematics of numerical analysis (Park City, UT, 1995) (Providence, RI), Lectures in Appl. Math., vol. 32, Amer. Math. Soc., Providence, RI, 1996, pp. 77{97. [2] Beatson, R.K., and Newsam, G.N., Fast evaluation of radial basis functions. I, Comput. Math. Appl. 24 (1992), 7{19. [3] Beatson, R.K. and Powell, M.J.D., An iterative method for thin plate spline interpolation that employs approximations to Lagrange functions, Numerical analysis 1993 (Dundee, 1993) (Harlow), Pitman Res. Notes Math. Ser., vol. 303, Longman Sci. Tech., Harlow, 1994, pp. 17{39. [4] Berry, M.J.A., Lino , G.S., Data Mining Techniques for Marketing, Sales and Customer Support, John Wiley and Sons, 1997. [5] Duchon, J., Splines minimizing rotation-invariant semi-norms in Sobolev spaces, (1977), 85{100. Lecture Notes in Math., Vol. 571. [6] Fortin, M. and Glowinski, R., Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, North-Holland, 1983. [7] Golub,G. H. and Van Loan, C. F., Matrix computations, 2 Johns Hopkins University Press, 1989. [8] Elman, H. C. and Golub, H. G., Inexact and Preconditioned Uzawa Algorithms for Saddle Point Problems, SIAM J. Numer. Anal., 31, 1994, 1645{1661. [9] Hastie,T.J. and Tibshirani,R.J., Generalized additive models, Monographs Chapman and Hall, 1990. [10] Hegland, M., Roberts, S. and Altas, I., Finite Element Thin Plate Splines for Data Mining Applications, submitted to Mathematical Methods for Curves and Surfaces II editors M. Daehlen, T. Lyche and L. S. Schumaker. [11] Sibson R. and Stone, G., Computation of Thin-Plate Splines, SIAM J. Sci. Statist. Comput. 12 (1991), 1304{1313. [12] Verfurth, R., A Combined Conjugate Gradient Multi-Grid Algorithm for the Numerical Solution of the Stokes Problem, IMA J. Numer. Anal., 4, 1984, 441{455.
