The Optimal Computation of Hessians in Optimization

The Optimal Computation of Hessians in Optimization Assefaw H. Gebremedhin, Alex Pothen, Arijit Tarafdar and Andrea Walther Old Dominion University, Norfolk VA USA and Technical University of Dresden, Germany

June 16th 2006

Overview of Talk

1

Computing derivatives using FD and AD

2

Exploiting sparsity

3

Direct and Substitution recovery methods and Star and Acyclic coloring formulations

4

New algorithms and results on Colors and Runtimes

5

Results for Hessian computation in ADOL-C

6

Conclusions

One approach for computing derivatives: Approximation via Finite Differencing F : Rn → Rm , A = F 0 (x) Aej =

1 ∂ F (x) ≈ [F (x + hej ) − F (x)], 1 ≤ j ≤ n ∗ ∂xj h

(ej is the jth coordinate vector) f : Rn → R, A = ∇2 f can be estimated by applying * on ∇f (if available). Note that computing the entire Jacobian/Hessian in this fashion requires n + 1 function evaluations.

A better approach for computing derivatives: Automatic Differentiation

I

I

AD is a technology for transforming source code for computing a function into source code for computing its derivative Relies on systematic application of the chain rule I

I

a function is decomposed into a sequence of arithmetic operations and intrinsic functions (variables classified as independent, intermediate and dependent)

Unlike Finite Differencing, incurs no truncation error

Automatic Differentiation (cont’d) I

Two basic modes I

Forward idea: choose an input variable, calculate sensitivity of every intermediate wrt that input. computes the Jacobian-vector product F 0 (x) · s at a cost of at most 5 · OPS(F (x)).

I

Reverse idea: choose an output variable, calculate sensitivity of that output wrt each intermediate. computes the vector-Jacobian product s · F 0 (x) at a cost of at most 5 · OPS(F (x)), independent of number of inputs.

I

Using the second-order adjoint mode, which is a combination of the forward and the reverse modes, the Hessian-vector product ∇2 f (x) · s can be computed at a cost of at most 10 · OPS(f (x)).

Sparse derivative matrix computation Given: a vector function F (or a scalar function f ) Goal: compute the derivative matrix A = F 0 (or ∇2 f ) efficiently Approach: employ the following four-step strategy 1 2 3 4

Compute the sparsity pattern of A ∈ Rm×n Compute a seed matrix S ∈ {0, 1}n×p with the smallest p Compute the compressed matrix B = AS Recover the nonzeros of A from B

The matrix S partitions the columns of A: ( 1 if column aj belongs to group k, sjk = 0 otherwise. p X k=1

sjk = 1

∀j : 1 ≤ j ≤ n.

A matrix and its compressed representation

Computation of seed matrix: problem variations

Matrix types

Partition types

I

nonsymmetric

I

unidirectional

I

symmetric

I

bidirectional

Recovery schemes

Required matrix entries

I

direct

I

all

I

substitution

I

some

Overview of coloring formulations

Jacobian Hessian Jacobian Hessian

1d partition distance-2 coloring star coloring — acyclic coloring

Nonsym A Sym A

2d partition star bicoloring — acyclic bicoloring —

Gb (A) = (V1 , V2 , E ) G (A) = (V , E )

Direct Direct Substitution Substitution

Compressing the Hessian for direct recovery Symmetrically orthogonal partition whenever aij 6= 0 I

I

1 2 3 4 5 6

aj only col in a group with nonzero at row i, or

X X X X X

ai only col in a group with nonzero at row j.

Formulated by Powell and Toint.

X X

X X X X X X X

a1

Star coloring

X X

X

a2

X

a5

A vertex coloring φ of G (A) s.t. I

φ is a distance-1 coloring, and

I

every P4 uses ≥ 3 colors.

Equivalence established by Coleman and Mor´e (84).

a6

a3 a4

Compressing the Hessian for indirect recovery Substitutable partition whenever aij 6= 0 I

I

1 2 3 4 5 6 7 8 9 10 X XX XXX X XXX X XX X XX X XX X X X X X X

aj in a group where all nonzeros in row i are ordered before aij , or ai in a group where all nonzeros in row j are ordered before aij .

Formulated by Powell and Toint.

a1

Acyclic coloring φ is a distance-1 coloring, and

I

every cycle uses ≥ 3 colors.

Equivalence established by Coleman and Cai (86).

X X

X XX XXX XX

a3

a5

A vertex coloring φ of G (A) s.t. I

a2

X

a7

a4

a6

a8

a9

a 10

Experimental results: Overview I

Algorithms compared I greedy algorithms for distance-k coloring (D1 and D2) I two earlier algorithms for star coloring (NS, GMP; RS, Powell and Toint) I an earlier algorithm for triangular coloring (T-sl, Coleman and More) I the new star (S) and acyclic (A) coloring algorithms

range sum

|V | in 1000 10 – 150 1,500

|E | in 1000 50 – 17,000 88,000

MaxDeg 8 – 860 6,400

MinDeg 0 – 230 800

AvgDeg 3 – 600 4,200

Table: Summary of size and density of test graphs (total: 29).

colors time (min)

D2 9,240 28.2

RS 8,749 34.4

NS 7,636 930

S 7,558 162

T-sl 5,065 12.4

A 4,110 32.5

D1 1,757 0.04

Table: Total number of colors and runtime, summed over all test cases.

Experimental results in more detail: Colors

6

4.5

D2 RS NS S MaxDeg

5.5

MaxDeg T−sl A 4

5

Colors normalized wrt D1

Colors normalized wrt D1

3.5 4.5

4

3.5

3

3

2.5

2 2.5 1.5

2

1.5

0

5

10

15 Graph id

20

25

30

1

0

5

10

15 Graph id

20

25

30

Experimental results in more detail: Runtimes 2

10

NS S RS T−sl A

1

Runtime normalized wrt D2

10

0

10

−1

10

−2

10

−3

10

0

5

10

15 Graph id

20

25

30

Experiments using ADOL-C Test function: with f (x) = x t Cx + at x,

min f (x)

x∈Rn

C ∈ Rn×n , a = (10, . . . , 10)t ∈ Rn Structure and size of the Hessian C : bd

0

bd2

0

10

10

10

20

20

20

30 0

bd1 bd2 rd

10

nz = 198

20

30

30 0

rd, ρ=6.87

0

10

nz = 186

20

30

30 0

10

nz = 206

20

30

banded matrix of bandwidth ρ + 1 (nnz/row = ρ + 1) banded matrix of bandwidth 2ρ + 1 (nnz/row = ρ + 1) random matrix with average nnz/row ρ ≈ ρ + 1 ρ ∈ {10, 20}, ρ ∈ {10.98, 20.99} n/1000 ∈ {1, 5, 10, 20, 40, 60, 80, 100}

ADOL-C Times for 100K problem: Compr Hess and Tot bd

−3

x 10

runtime/n

4 3

dir, ρ=10 dir, ρ=20 sub, ρ=10 sub, ρ=20

3 2 1

1 0 0 5 10 20

40 60 n/1000

80

0 0 5 10 20

100

rd

x 10−3

12

10

10

8

8

runtime/n

runtime/n

x 10

4

2

12

bd

−3

5

dir, ρ=10 dir, ρ=20 sub, ρ=10 sub, ρ=20

runtime/n

5

6 4 2 0 0 5 10 20

40 60 n/1000

80

100

80

100

rd

x 10−3

6 4 2

40

60

80

100

0 0 5 10 20

40

60

ADOL-C: Absolute Runtimes

Problem Banded Banded Random Random

Method Direct Subs. Direct Subs.

Color 3 12 37 24

Compr. Hess. 274 168 612 207

Recovery 0.4 3 0.5 45

Total 278 183 650 277

Table: Times for various steps in Hessian computation.

Conclusions

I

First practical algorithm and implementation for acyclic coloring (Hessian computation with a substitution method)

I

New efficient algorithm for star coloring (Hessian computation with a direct method)

I

Codes for computing seed matrix (coloring), compressed Hessian, and recovery of Hessian elements incorporated into ADOL-C.

I

A substitution method enables faster computation of Hessians than with a direct method.

References

Gebremedhin, Manne and Pothen. Graph Coloring for Computing Derivatives. SIAM Review, Vol 47, No 4, pp. 629–705, 2005. Gebremedhin, Tarafdar, Manne and Pothen. New Acyclic and Star Coloring Algorithms. Submitted to SIAM Journal on Scientific Computing. Gebremedhin, Pothen, Tarafdar and Walther. Efficient Computation of Sparse Hessians using ADOL-C. In Prep.