SCAN 2008 Sep. 29 - Oct. 3, 2008, at El Paso, TX, USA
Accurate Solution of Triangular Linear System Philippe Langlois DALI at University of Perpignan France
Nicolas Louvet INRIA and LIP at ENS Lyon France
[email protected] [email protected]
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
1 / 19
Motivation How to improve and validate the accuracy of a floating point computation, without large computing time overheads ?
Our main tool to improve the accuracy: compensation of the rounding errors. Existing results: I
I
summation and dot product algorithms from T. Ogita, S. Oishi and S. Rump, Horner algorithm for polynomial evaluation from the authors.
Today’s study: triangular system solving which is one of the basic block for numerical linear algebra.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
2 / 19
Outline
1
Context
2
The substitution algorithm and its accuracy
3
A first compensated algorithm CompTRSV
4
Compensated algorithm CompTRSV2 improves CompTRSV
5
Conclusion
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
3 / 19
Floating point arithmetic Let a, b ∈ F and op ∈ {+, −, ×, /} an arithmetic operation. fl(x op y ) = the exact x op y rounded to the nearest floating point value. x+y
x⊕y
Every arithmetic operation may suffer from a rounding error. Standard model of floating point arithmetic : fl(a op b) = (1 + ε)(a op b),
with
|ε| ≤ u.
Working precision u = 2−p (in rounding to the nearest rounding mode). In this talk: IEEE-754 binary fp arithmetic, rounding to the nearest, no underflow nor overflow. Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
4 / 19
Why and how to improve the accuracy? The general “rule of thumb” for backward stable algorithms: result accuracy . condition number of the problem × computing precision.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
5 / 19
Why and how to improve the accuracy? The general “rule of thumb” for backward stable algorithms: result accuracy . condition number of the problem × computing precision. Classic solution to improve the accuracy: increase the working precision u. – Hardware solution: I
extended precision available in x87 fpu units.
– Software solutions: I
I
arbitrary precision library (the programmer choses its working precision): MP, MPFUN/ARPREC, MPFR. fixed length expansions libraries: double-double, quad-double (Bailey et al.) ,→ XBLAS library = BLAS + double-double (precision u2 )
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
5 / 19
Why and how to improve the accuracy? The general “rule of thumb” for backward stable algorithms: result accuracy . condition number of the problem × computing precision. Classic solution to improve the accuracy: increase the working precision u. – Hardware solution: I
extended precision available in x87 fpu units.
– Software solutions: I
I
arbitrary precision library (the programmer choses its working precision): MP, MPFUN/ARPREC, MPFR. fixed length expansions libraries: double-double, quad-double (Bailey et al.) ,→ XBLAS library = BLAS + double-double (precision u2 )
Alternative solution: compensated algorithms use corrections of the generated rounding errors.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
5 / 19
Error-free transformations (EFT) Error-Free Transformations are algorithms to compute the rounding errors at the current working precision.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
6 / 19
Error-free transformations (EFT) Error-Free Transformations are algorithms to compute the rounding errors at the current working precision. +
(x, y ) = 2Sum(a, b)
×
(x, y ) = 2Prod(a, b)
/
(q, r ) = DivRem(a, b)
such that such that such that
6 flop
Knuth (74)
17 flop
Dekker (71)
20 flop
Pichat &
x = a ⊕ b and a + b = x + y x = a ⊗ b and a × b = x + y q = a b, and a = b × q + r .
Vignes (93)
with a, b, x, y , q, r ∈ F.
Algorithm (Knuth)
Algorithm (EFT for the division)
function [x,y] = 2Sum(a,b) x =a⊕b z =x a y = (a (x z)) ⊕ (b z)
function [q, r ] = DivRem(a, b) q =a b [x, y ] = 2Prod(q, b) r = (a x) y
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
6 / 19
Substitution algorithm for Tx = b Consider the triangular linear system t1,1 . .. . . . tn,1 · · · tn,n
Tx = b, with T ∈ Fn×n and vector b ∈ Fn , x1 b1 . . . = . . . . xn bn
The substitution algorithm computes x1 , x2 , . . . , xn , the solution of Tx = b, as ! k−1 X 1 bk − tk,i xi , xk = tk,k i=1
Ph. Langlois (Univ. Perpignan, France)
Algorithm (Substitution) function b x = TRSV(T , b) for k = 1 : n b s k,0 = bk for i = 1 : k − 1 b p k,i = tk,i ⊗ b xi b s k,i = b s k,i−1 b p k,i end b xk = b s k,k−1 tk,k end
Accurate Solution of Triangular Linear System
October 2, 2008
7 / 19
Accuracy of the substitution algorithm
Skeel’s condition number for Tx = b is cond(T , x) :=
k|T −1 ||T ||x|k∞ . kxk∞
The accuracy of b x = TRSV(T , x) satisfies kb x − xk∞ ≤ nu cond(T , x) + O(u2 ), kb x k∞ assuming cond(T ) := k|T −1 ||T |k∞
1 nu .
The computed b x can be arbitrarily less accurate than the w.p. u.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
8 / 19
Substitution : accuracy w.r.t. cond(T , x) Erreur relative en fonction de cond(T,x) [n=40] 1 TRSV 10-2 10-4 -6
rT rn
ep
ou
10-8 Bo
erreur relative
RS
V
10
10-10 10-12 -14
10
u
10-16 10-18
u-1 1
105
1010
1015 1020 cond(T, x)
u-2 1025
1030
1035
How to compensate the rounding errors generated by the substitution algorithm? Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
9 / 19
Compensated triangular system solving b x = TRSV(T , x) is the solution to Tx = b computed by substitution. Our goal: to compute an approximate b c of the correcting term c =x− b x,
with
c ∈ Rn×1 ,
and then compute the compensated solution x = b x⊕b c.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
10 / 19
Compensated triangular system solving b x = TRSV(T , x) is the solution to Tx = b computed by substitution. Our goal: to compute an approximate b c of the correcting term c =x− b x,
with
c ∈ Rn×1 ,
and then compute the compensated solution x = b x⊕b c. Since Tc = b − T b x , c is the exact solution of the triangular system Tc = r ,
with
r =b−T b x ∈ Rn×1 .
In the classic iterative refinement frame, I I
r is the residual associated with computed b x the useful approach to compute an approximate residual b r: evaluate b − T b x using twice the working precision (u2 )
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
10 / 19
Computing the residual thanks to EFT function b x = TRSV(T , b) for k = 1 : n b s k,0 = bk for i = 1 : k − 1 b p k,i = tk,i ⊗ b xi b s k,i = b s k,i−1 b p k,i end b xk = b s k,k−1 tk,k end
{rounding error πk,i ∈ F } {rounding error σk,i ∈ F } {rounding error ρk /tk,k , with ρk ∈ F }
Proposition Given b x = TRSV(T , b) ∈ Fn , let r = (r1 , . . . , rn )T ∈ Rn be the residual r =b−T b x associated with b x . Then we have exactly rk = ρk +
k−1 X
σk,i − πk,i ,
for k = 1 : n.
i=1 Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
11 / 19
Compensated substitution algorithm This algorithm computes
Algorithm function x = CompTRSV(T , b) for k = 1 : n b s k,0 = bk ; b r k,0 = b c k,0 = 0 for i = 1 : k − 1 xi ) [b p k,i , πk,i ] = 2Prod(tk,i , b [b s k,i , σk,i ] = 2Sum( b s k,i−1 , − b p k,i ) b r k,i = b r k,i−1 ⊕ (σk,i πk,i ) b c k,i = b c k,i−1 ⊕ tk,i ⊗ b ci end [b x k , ρk ] = DivRem( b s k−1 , tk,k ) b r k = ρk ⊕ b r k,k−1 b ck = ( b rk b c k,k−1 ) tk,k end x= b x⊕b c Ph. Langlois (Univ. Perpignan, France)
the approximate solution b x = (b x 1, . . . , b x n )T = TRSV(T , b) by the substitution algorithm; the rounding errors πk,i , σk,i and ρk ; the residual vector b r = (b r 1, . . . , b r n )T ≈ b − T b x as a function of πk,i , σk,i and ρk ; the correcting term Tc = b b c = (b c 1, . . . , b c n )T = TRSV(T , b r ); the compensated solution x= b x⊕b c.
Accurate Solution of Triangular Linear System
October 2, 2008
12 / 19
An a priori error bound for CompTRSV The approximate residual vector b r computed in CompTRSV is as accurate as if it was computed in doubled working precision (u2 ).
Theorem The accuracy of the compensated solution b x = CompTRSV(T , b) satisfies k x − xk∞ kxk∞
. u + fn u2 K(T , x) + O(u3 ).
This error bound is more pessimistic than the “expected” error bound u + gn u2 cond(T , x) + O(u3 ), since k(|T −1 ||T |)2 |x|k∞ k|T −1 ||T ||x|k∞ =: cond(T , x) ≤ K(T , x) := . kxk∞ kxk∞ But we often observe K(T , x) ≤ α cond(T , x) with α “small” in practice. . . Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
13 / 19
Practical accuracy of CompTRSV w.r.t. cond(T , x) 1040 1035 1030
103 cond(T,x) K(T,x) cond(T, x)
1025 20
10
1015 1010 5
10
1
5
1
10
10
10
15
20
10
25
10
10
30
35
10
10
1 TRSV
-2
10
XBlasTRSV CompTRSV
n=40, systems generated by “method 1”: K(T , x) ≤ 103 cond(T , x)
SV
10-4
XB las po ne
po
Bo r
rn e
10-10
In these experiments, CompTRSV is as accurate as XBlasTRSV
ur
TR ur
10-8 Bo
erreur relative
SV
TR
10-6
10-12 -14
10
u
10-16 10-18
u-1 1
5
10
10
10
Ph. Langlois (Univ. Perpignan, France)
15
10
u-2 20
10 cond(T, x)
25
10
30
10
35
10
Accurate Solution of Triangular Linear System
October 2, 2008
14 / 19
Practical accuracy of CompTRSV w.r.t. cond(T , x) 1030 106 cond(T,x) K(T,x) cond(T, x)
1020
1010
1
5
1
10
10
10
15
20
10
10
1 TRSV
10-2
XBlasTRSV CompTRSV
10-4
n=100, systems generated by “method 2”: K(T , x) ≤ 106 cond(T , x)
erreur relative
10-6 10-8 10-10 ur
10-12
e
rn
Bo
TR
The accuracy of CompTRSV is not as good as expected. . .
SV TR
as
SV
ur
po
ne
l XB
po
r Bo
-14
10
u
-16
10
10-18
u-1 1
5
10
Ph. Langlois (Univ. Perpignan, France)
10
10 cond(T, x)
15
10
20
10
Accurate Solution of Triangular Linear System
October 2, 2008
14 / 19
From CompTRSV to CompTRSV2 Algorithm function x = CompTRSV(T , b) for k = 1 : n b s k,0 = bk ; b r k,0 = b c k,0 = 0 for i = 1 : k − 1 [b p k,i , πk,i ] = 2Prod(tk,i , b xi ) bk,i ) [b s k,i , σk,i ] = 2Sum( b s k,i−1 , − p b r k,i = b r k,i−1 ⊕ (σk,i πk,i ) b c k,i = b c k,i−1 ⊕ tk,i ⊗ b ci end [b x k , ρk ] = DivRem( b s k−1 , tk,k ) b r k = ρk ⊕ b r k,k−1 b ck = ( b rk b c k,k−1 ) tk,k end x= b x⊕b c Ph. Langlois (Univ. Perpignan, France)
x k and b c k as a Iteration k produces b function of b x 1, . . . , b x k−1 b c 1, . . . , b c k−1 with b c k ≈ xk − b xk Vector compensation: corrected x is computed only after whole b x and b c have been computed Issue: component compensation, i.e. , compute x k at iteration k
Accurate Solution of Triangular Linear System
October 2, 2008
15 / 19
From CompTRSV to CompTRSV2 Algorithm
Iteration k produces
function x = CompTRSV2(T , b) for k = 1 : n b s k,0 = bk ; b r k,0 = b c k,0 = 0 for i = 1 : k − 1 [b p k,i , πk,i ] = 2Prod(tk,i , x i ) bk,i ) [b s k,i , σk,i ] = 2Sum( b s k,i−1 , − p b r k,i = b r k,i−1 ⊕ (σk,i πk,i ) b c k,i = b c k,i−1 ⊕ tk,i ⊗ y i end [b x k , ρk ] = DivRem( b s k−1 , tk,k ) b r k = ρk ⊕ b r k,k−1 b ck = ( b rk b c k,k−1 ) tk,k [ x k , y k ] = 2Sum( b xk, b ck) end Ph. Langlois (Univ. Perpignan, France)
b x k w.r.t. x 1 , . . . , x k−1 b c k s.t. b c k ≈ xk − b xk xk, b c k ), Since [ x k , y k ] = 2Sum( b xk ⊕ b ck xk = b xk + b ck xk + yk = b and y k ≈ xk − x k Iteration k computes x k and y k w.r.t. x 1 , . . . , x k−1 y 1 , . . . , y k−1 with y k ≈ xk − x k
Accurate Solution of Triangular Linear System
October 2, 2008
15 / 19
Accuracy of CompTRSV2 In algorithm CompTRSV, x1 . n . x = . ∈F , y = xn
y1 .. n . ∈F yn
x1 + y1 .. ∈ Rn . x+ y = . xn + yn
and
The vector x + y is an approximate solution of the system Tx = b, and the exact solution of a slightly perturbed system, (T + ∆T )( x + y ) = (b + ∆b),
with
2 2 |∆T | ≤ γ6n |T | and |∆b| ≤ γ6n |b|,
where γ6n ≈ 6nu.
Theorem The accuracy of the compensated solution x = CompTRSV2(T , b) satisfies k x − xk∞ kxk∞ Ph. Langlois (Univ. Perpignan, France)
. u + 72n2 u2 cond(T , x) + O(u3 ). Accurate Solution of Triangular Linear System
October 2, 2008
16 / 19
Practical accuracy of CompTRSV2 w.r.t. cond(T , x) n=100, systems generated by “method 2” 1 TRSV
-2
10
XBlasTRSV CompTRSV2
10-4
erreur relative
10-6 10-8 10-10 ur
10-12
n or
e
as
V RS
T
po
e
rn
Bo
B
po
Bl rX
SV TR
u
10-14 u
10-16 10-18
u-1 1
105
1010 cond(T, x)
1015
1020
CompTRSV2 is as accurate as the classic substitution algorithm performed in twice the working precision (u2 ). Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
17 / 19
Running time comparisons Top: overhead ratios to double the accuracy while increasing dimension n Down: CompTRSV and CompTRSV2 run at least twice as fast as XBlasTRSV Surcouts de XBLAS_dtrsv et CompTRSV par rapport a TRSV [P4, gcc, sse]
Surcouts de XBLAS_dtrsv et CompTRSV par rapport a TRSV [Ath64, gcc, sse]
XBlasTRSV / TRSV XBlasTRSV / TRSV
20
XBlasTRSV / TRSV XBlasTRSV / TRSV
20
Ratio
15
Ratio
15
10
10
5
5
0
0 0
200
400
600 800 Dimension du systeme n
1000
1200
1400
0
4 3.5
3 XBlasTRSV / CompTRSV
2.5
400
600 800 Dimension du systeme n
1000
1200
1400
Ratio du temps d’execution de XBlasTRSV sur celui de CompTRSV [Ath64, gcc, sse]
4 3.5 Ratio
Ratio
Ratio du temps d’execution de XBlasTRSV sur celui de CompTRSV [P4, gcc, sse]
200
2
3 XBlasTRSV / CompTRSV
2.5 2
1.5
1.5 0
200
400
600 800 Dimension du systeme n
Ph. Langlois (Univ. Perpignan, France)
1000
1200
1400
0
200
400
Accurate Solution of Triangular Linear System
600 800 Dimension du systeme n
1000
1200
October 2, 2008
1400
18 / 19
Conclusion
We have presented a compensated substitution algorithm, CompTRSV: I I
a priori error bound, and practical numerical behavior
not entirely satisfying. . . We have also presented an improvement of this method, CompTRSV2: the solution computed by CompTRSV2 is as accurate as if it was computed by the substitution algorithm in twice the working precision (u2 ). CompTRSV and CompTRSV2 runs at least twice as fast as XBlasTRSV.
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
19 / 19
Componentwise condition numbers for linear systems Let Ax = b be a linear system, with A ∈ Rn×n non singular and b ∈ Rn×1 . Given E ∈ Rn×n and b ∈ Rn×1 , with |E | ≥ 0 and |f | ≥ 0, Higham defines the following componentwise condition number, condE ,f (A, x) = lim sup ε→0
k∆xk∞ , (A + ∆A)(x + ∆x) = b + ∆b, εkxk∞ |∆A| ≤ εE , |∆b| ≤ εf ,
and proves that condE ,f (A, x) =
k|A−1 |(E |x| + f )k∞ . kxk∞
For the special case E = |A| and f = |b|, we use Skeel’s condition number, cond(A, x) =
k|A−1 ||A||x|k∞ , kxk∞
which differs from cond|A|,|b| (A, x) by at most a factor 2. Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
20 / 19
Floating-point operations counts
TRSV: n2 CompTRSV and CompTRSV2: 27n2 /2 + O(n) XBlasTRSV: 45n2 /2 + O(n)
Ph. Langlois (Univ. Perpignan, France)
Accurate Solution of Triangular Linear System
October 2, 2008
21 / 19
