Aug 28, 2012 - NET and Python APIs. s AMPL, AIMMS, GAMS, MATLAB, and R ... x Practice: Slow. ... s If too many dense columns are located then assume no.
On recent improvements in MOSEK Erling D. Andersen MOSEK ApS, Fruebjergvej 3, Box 16, 2100 Copenhagen, Denmark. WWW: http://www.mosek.com August 28, 2012 http://www.mosek.com
Introduction
2 / 23
MOSEK Introduction MOSEK New in MOSEK version 7 The factorization step Dense columns Computational results
■
MOSEK is software package for large scale optimization.
■
Version 1 released April 1999.
■
Version 6 released March 2009.
■
Linear and conic quadratic (+ mixed-integer).
■
Conic quadratic optimization (+ mixed-integer).
■
Convex(functional) optimization.
■
C, JAVA, .NET and Python APIs.
■
AMPL, AIMMS, GAMS, MATLAB, and R interfaces.
■
Free for academic use. See http://www.mosek.com. 3 / 23
New in MOSEK version 7 Introduction MOSEK New in MOSEK version 7 The factorization step
■
Interior-point optimizer ◆ Rewritten factor routines. ◆ Handling of intersection cones in conic quadratic
Dense columns
optimization.
Computational results
◆ Handling of semi-definite optimization problems. ■
New mixed integer optimizer for conic problems.
■
Fusion, a modeling tool for conic problems.
Note: ■
Semi-definite talk: J. Dahl, Thursday at 11, Room H0110.
■
Version 7 is in early beta mode.
■
Only limited results.
4 / 23
The factorization step Introduction MOSEK New in MOSEK version 7 The factorization step
MOSEK solves: cT x Ax = b, x ≥ 0.
(P ) min st
Dense columns Computational results
where A is large and sparse using a homogenous interior-point algorithm. Each iterations requires solution of multiple: "
−H −1 AT A 0
#"
f1 f2
#
=
"
r1 r2
#
(1)
where H is a diagonal matrix. ■
Known as the augmented system.
5 / 23
Normal equation system Introduction MOSEK New in MOSEK version 7 The factorization step Dense columns Computational results
Is reduced to: AHAT f2 =
(2)
■
The normal equation system!
■
Identical to the augmented system approach using a particular pivot order.
■
System is reordered to preserve sparsity using approximative min degree (AMD) or graph partitioning (GP).
■
No numerical pivoting and the pivot order is kept FIXED over the iterations.
■
Just one dense columns in A cause a lot of fill-in.
■
Known since the early days of interior-point methods. 6 / 23
Dense columns
7 / 23
Dense columns handling Introduction Dense columns Other improvements to factor routines Computational results
Methods for dealing with dense columns: ■
Pre interior-point. Tearing. (Duff, Erisman and Reid [2])
■
Augmented system. (Fourer and Mehrotra [3]) ◆ Pivoting for stability and sparsity. Not fixed. ◆ Stable and low flops. ◆ Practice: Slow.
■
Sherman-Morrison-Woodbury. (Cheap but unstable). (Karmarkar + many more)
■
Stabilized SMW. (Cheaper and OK stability). (Andersen [1])
■
Product form Cholesky. (Costly but stable). (Goldfarb and Scheinberg [4]]. 8 / 23
Idea of SMW Introduction Dense columns Other improvements to factor routines Computational results
■
Let S¯ be the index set of the sparse columns in A. And ¯ the index set of the dense ones. N
■
Solve a system with the matrix "
AS¯HS¯ATS¯ ATN¯
AN¯ −1 −HN ¯
■
Pivot order is fixed.
■
Requires AS¯HS¯ATS¯ to be of full rank.
#
◆ Leads numerical instability. ■
Consequence: Minimize number of dense columns. 9 / 23
Dense columns detection Introduction Dense columns Other improvements to factor routines Computational results
■
Density is only a sufficient condition for problems.
■
Troublesome columns may not be that dense at all.
■
Most published methods uses a simple threshold of around 30. Fairly naive.
■
Meszaros [5] discuss a sophisticated method. ◆ Hybrid approach. ◆ Augmented but with a fixed pivot order. ◆ Similar to Vanderbei’s [6] quasi definite approach
with a sophisticated dense column detection.
10 / 23
Example: karted Introduction Dense columns Other improvements to factor routines Computational results
Dens. 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Num. 8 81 544 3782 17227 48321 62561 1 3 15 27 127 224 193
■
Has dense columns!
■
Which cutoff to use?
11 / 23
A new graph partitioning based approach Introduction Dense columns Other improvements to factor routines
■
Idea. ◆ Try to emulate the optimal ordering for the
Computational results
augmented system. ◆ Fixed pivot order. ◆ Keep detection cost down. ■
Solve a linear system of the form: "
AS¯HS¯ATS¯ ATN¯
AN¯ −1 −HN ¯
#
12 / 23
Introduction Dense columns Other improvements to factor routines
¯N ¯ ) be an initial guess for the partition. And choose a Let (S, reordering P so
Computational results
P
"
AS¯HS¯ATS¯ AN¯ −1 ATN¯ −HN ¯
#
PT
=
M11 0 M13 0 M22 M23 M31 M32 M33
■
M11 and M22 should be of about identical size.
■
M33 should be as small a possible.
■
Ordering can be locate using graph partitioning i.e. use MeTIS or the like.
■
¯ and M33 are the dense Nodes that appears in both N columns.
■
¯ N ¯ ) is obtained. A refined (S,
13 / 23
Refinements of the basic idea Introduction Dense columns Other improvements to factor routines Computational results
■
If too many dense columns are located then assume no dense columns.
■
Can be applied recursively on M11 and M22 blocks.
■
Graph partitioning is potentially expensive. ◆ Skip dense detection when safe. ◆ Do not recurse too much.
14 / 23
Other improvements to factor routines Introduction Dense columns Other improvements to factor routines
■
New graph partitioning based ordering.
■
Remove dependency on OpenMP for parallelization.
Computational results
◆ Use native threads. ◆ OpenMP is a pain in a redistribution setting. ■
Made it possible to use external sequential BLAS. ◆ Exploit hardware advances such as Intel AVX easily.
15 / 23
Computational results
16 / 23
Dense column handling Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
■
Comparison of MOSEK v6 and v7.
■
Hardware: Linux 64 bit, Intel(R) Xeon(R) CPU E31270 @ 3.40GHz.
■
Using 1 thread unless otherwise stated.
■
Problems ◆ Private and public test problems
17 / 23
Results for linear problems Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
Name bas1 difns8t4 L1 nine12 bienstock-310809-1 net12 gonnew16 GON8IO ind3 15dec2008 pointlogic-210911-1 time horizon minimiser lt dray17 ind2 zhao4 neos3 c3 avq1 TestA5 ml2010-rmine14 rusltplan tp-6 dray5 stormG2 1000 degme karted friedlander-6 ts-palko scipmsk1 160910-2 gamshuge G. avg
Time (s) v6 8.27 8.28 8.54 10.39 11.14 13.42 13.85 15.05 19.06 20.82 23.89 29.26 31.07 46.26 48.32 60.93 84.82 110.40 117.18 139.17 140.50 142.91 171.21 188.52 229.51 242.50 266.44 320.57 325.59 326.86 1266.58
R. time v7/v6 0.66 1.62 1.62 1.06 0.37 2.69 0.65 1.16 0.39 1.14 0.37 0.52 0.43 1.45 0.26 1.14 1.81 0.72 0.72 1.64 1.01 2.06 0.44 0.44 1.01 1.21 0.11 0.65 1.01 0.72 0.77 0.79
Iter. v6 10 27 15 20 42 37 27 12 21 45 15 46 77 12 31 9 9 12 14 24 41 49 53 107 62 20 24 212 17 80 98
R. iter. v7/v6 0.82 1.07 1.00 2.19 0.63 1.00 0.96 1.00 0.95 0.83 1.00 0.53 1.67 1.23 0.91 1.80 1.10 1.15 1.00 2.40 1.02 0.98 0.69 0.49 0.54 0.95 1.04 0.11 1.28 4.95 1.15 0.99
Num. v6 5 74 29 400 544 246 73 3 175 451 23 292 55 318 680 1 57 538 1 28 718 776 0 119 883 193 0 210 749 291 270
dense v7 40 89 0 625 545 329 278 185 287 175 0 506 447 970 0 2 0 1 1 28 2094 742 1203 119 890 590 721 210 1 1893 44
18 / 23
Comments Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
■
Other changes contributes to difference. ◆ New dualization heuristic. ◆ Better programming, new compiler etc.
■
Many dense columns in v7. ◆ Does not affect stability much.
■
New method seems to work well. ◆ Can be relatively expensive for smallish problems.
19 / 23
Results for conic quadratic problems Introduction Name ramsey3 050508-1 msprob3 041208-1 230608-1 280108-1 pcqo-250112-1 211107-1 autooc msci-p1to bleyer-200312-1 201107-3 G. avg
Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
Time (s) v6 0.78 1.02 1.06 1.80 3.87 4.92 5.67 12.68 15.43 16.41 34.10 35.67
R. time v7/v6 1.47 1.80 0.90 0.61 0.92 1.37 0.82 1.08 2.21 0.11 0.23 0.46 0.77
Iter. v6 12 25 32 25 62 42 17 50 28 24 11 50
R. iter. v7/v6 1.00 1.15 1.21 1.19 0.84 1.09 1.11 0.88 1.03 1.16 1.42 1.14 1.09
Num. dense v6 v7 199 212 72 198 73 31 5 93 7 750 67 15 1176 1260 0 1725 41 201 42 433 0 1828 0 199
Comment: ■
Results is similar to the linear case.
20 / 23
Factor speed improvements Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
■
Very limited and selective test.
■
Only Mittlemans conic quadratic benchmark.
Name firL2L1alph firL1Linfeps firL2Linfeps firL1 firL2L1eps firL2Linfalph firLinf firL2a firL1Linfalph G. avg
■
Large speed-up.
■
Dense problems!
Time (s) v6 1.08 4.85 7.12 15.02 15.67 21.92 32.93 53.68 63.79
R. time v7/v6 0.87 0.56 0.56 0.55 0.56 0.25 0.38 0.14 0.31 0.41
Iter. v6 11 57 17 18 18 20 21 9 22
R. iter. v7/v6 1.09 0.91 1.00 0.78 0.94 0.70 1.00 0.56 1.05 0.87
21 / 23
Summary and conclusions Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
■
A new method for detecting dense columns has been presented.
■
Seems to work well.
■
MOSEK v7 performs much better than v6 on Mittlemans CQO benchmark.
■
Slide are at http://mosek.com/resources/presentations/.
22 / 23
References Introduction Dense columns Computational results Dense column handling Factor speed improvements Summary and conclusions References
[1] K. D. Andersen. A Modified Schur Complement Method for Handling Dense Columns in Interior-Point Methods for Linear Programming. ACM Trans. Math. Software, 22(3):348–356, 1996. [2] I. S. Duff, A. M. Erisman, and J. K. Reid. Direct methods for sparse matrices. Oxford University Press, New York, 1989. [3] R. Fourer and S. Mehrotra. Solving symmetric indefinite systems in an interior point method for linear programming. Math. Programming, 62:15–40, 1993. [4] D. Goldfarb and K. Scheinberg. A product-form cholesky factorization method for handling dense columns in interior point methods for linear programming. Mathematical Programming, 99:1–34, 2004. 10.1007/s10107-003-0377-7. [5] Cs. M´esz´aros. Detecting “dense” columns in interior
23 / 23
