## Interior Point Algorithm - Princeton University

Mar 30, 2007 - 4. Update: x := x + tâx. Advantages of Newton method: Fast, Robust, Scalable ..... Ellipsoid method, cutting plane method, simplex method .

ELE539A: Optimization of Communication Systems Lecture 18: Interior Point Algorithm

Professor M. Chiang Electrical Engineering Department, Princeton University

March 30, 2007

Lecture Outline • Newton’s method • Equality constrained convex optimization • Inequality constrained convex optimization • Barrier function and central path • Barrier method

• Summary of topics covered in the course • Final projects • Conclusions

Newton Method Newton step: ∆xnt = −∇2 f (x)−1 ∇f (x)

Positive definiteness of ∇2 f (x) implies that ∆xnt is a descent direction

Interpretation: linearize optimality condition ∇f (x∗ ) = 0 near x, ∇f (x + v) ≈ ∇f (x) + ∇2 f (x)v = 0 Solving this linear equation in v, obtain v = ∆xnt . Newton step is the addition needed to x to satisfy linearized optimality condition

Main Properties • Affine invariance: given nonsingular T ∈ Rn×n and let f¯(y) = f (T x). Then Newton step for f¯ at y: ∆ynt = T −1 ∆xnt and x + ∆xnt = T (y + ∆ynt ) • Newton decrement: “ ”1/2 “ ”1/2 T 2 −1 T 2 λ(x) = ∇f (x) ∇ f (x) ∇f (x) = ∆xnt ∇ f (x)∆xnt Let fˆ be second order approximation of f at x. Then 1 f (x) − p∗ ≈ f (x) − inf fˆ(y) = f (x) − fˆ(x + ∆xnt ) = λ(x)2 y 2

Newton Method GIVEN a starting point x ∈ dom f and tolerance ǫ > 0 REPEAT 1. Compute Newton step and decrement: ∆xnt = −∇2 f (x)−1 ∇f (x) and ` ´1/2 T 2 −1 λ = ∇f (x) ∇ f (x) ∇f (x) 2. Stopping criterion: QUIT if

λ2 2

≤ǫ

3. Line search: choose a step size t > 0 4. Update: x := x + t∆x

Advantages of Newton method: Fast, Robust, Scalable

Error vs. Iteration for Problems in R100 and R10000

T

minimize c x −

M X

100 log(bi − aT and ∈ R10000 i x), x ∈ R

i=1

5

5

10

f (x(k) ) − p∗

f (x(k) ) − p∗

10

0

10

0

10

−5

10

b

x

−10

10

−5

10

−15

10

0

2

4

k

6

8

10

0

5

10

k

15

20

Equality Constrained Problems Solve a convex optimization with equality constraints: minimize

f (x)

subject to

Ax = b

f : Rn → R is twice differentiable A ∈ Rp×n with rank p < n Optimality condition: KKT equations with n + p equations in n + p variables x∗ , ν ∗ : Ax∗ = b,

∇f (x∗ ) + AT ν ∗ = 0

Approach 1: Can be turned into an unconstrained optimization, after eliminating the equality constraints

Approach 2: Dual Solution Dual function: g(ν) = −bT ν − f ∗ (−AT ν) Dual problem: maximize − bT ν − f ∗ (−AT ν)

Example: solve Pn

minimize

subject to

Ax = b

i=1

log xi

Dual problem: maximize − bT ν +

n X

log(AT ν)i

i=1

Recover primal variable from dual variable: xi (ν) = 1/(AT ν)i

Example With Analytic Solution Convex quadratic minimization over equality constraints: minimize

(1/2)xT P x + q T x + r

subject to

Ax = b

Optimality condition: 2 4

P A

AT 0

32 54

x∗ ν∗

3

2

5=4

−q b

3 5

If KKT matrix is nonsingular, there is a unique optimal primal-dual pair x∗ , ν ∗ If KKT matrix is singular but solvable, any solution gives optimal x∗ , ν ∗ If KKT matrix has no solution, primal problem is unbounded below

Approach 3: Direct Derivation of Newton Method Make sure initial point is feasible and A∆xnt = 0 Replace objective with second order Tayler approximation near x: minimize

fˆ(x + v) = f (x) + ∇f (x)T v + (1/2)v T ∇2 f (x)v

subject to

A(x + v) = b

Find Newton step ∆xnt by solving: 2 32 3 2 3 ∇2 f (x) AT ∆xnt −∇f (x) 4 54 5=4 5 A 0 w 0 where w is associated optimal dual variable of Ax = b

Newton’s method (Newton decrement, affine invariance, and stopping criterion) stay the same

Inequality Constrained Minimization Let f0 , . . . , fm : Rn → R be convex and twice continuously differentiable and A ∈ Rp×n with rank p < n: minimize

f0 (x)

subject to

fi (x) ≤ 0, i = 1, . . . , m Ax = b

Assume the problem is strictly feasible and an optimal x∗ exists

Idea: reduce it to a sequence of linear equality constrained problems and apply Newton’s method First, need to approximately formulate inequality constrained problem as an equality constrained problem

Barrier Function Make inequality constraints implicit in the objective: minimize

f0 (x) +

subject to

Ax = b

Pm

i=1 I− (fi (x))

where I− is indicator function: 8 < 0 I− (u) = : ∞

u≤0 u>0

No inequality constraints, but objective function not differentiable Approximate indicator function by a differentiable, closed, and convex function: Iˆ− (u) = −(1/t) log(−u), dom Iˆ− = −R++ where a larger parameter t gives more accurate approximation Iˆ− increases to ∞ as u increases to 0

Log Barrier 10

y

5

0

−5 −3

−2

−1

0

u

Use Newton method to solve approximation: P ˆ minimize f0 (x) + m i=1 I− (fi (x)) subject to

Ax = b

1

Log Barrier Log barrier function: φ(x) = −

m X

log(−fi (x)), dom φ = {x ∈ Rn |fi (x) < 0, i = 1, . . . , m}

i=1

Approximation better if t is large, but then Hessian of f0 + (1/t)φ varies rapidly near boundary of feasible set. Accuracy Stability tradeoff Solve a sequence of approximation with larger t, using Newton method for each step of the sequence Gradient and Hessian of log barrier function: ∇φ(x) 2

∇ φ(x)

=

=

m X

1 ∇fi (x) −fi (x)

i=1

X 1 1 T 2 ∇f (x)∇f (x) + ∇ fi (x) i i fi (x)2 −f (x) i i=1

i=1 m X

m

Central Path Consider the family of optimization problems parameterized by t > 0: minimize

tf0 (x) + φ(x)

subject to

Ax = b

Central path: solutions to above problem x∗ (t), characterized by: 1. Strict feasibility: Ax∗ (t) = b,

fi (x∗ (t)) < 0, i = 1, . . . , m

2. Centrality condition: there exists νˆ ∈ Rp such that t∇f0 (x∗ (t)) +

m X i=1

1 ∇fi (x∗ (t)) + AT νˆ = 0 ∗ −fi (x (t))

Every central point gives a dual feasible point. Let λ∗i (t) = −

1 , i = 1, . . . , m tfi (x∗ (t))

ν ∗ (t) =

νˆ t

Central Path Dual function g(λ∗ (t), ν ∗ (t)) = f0 (x∗ (t)) − m/t which implies duality gap is m/t. Therefore, suboptimality gap f0 (x∗ (t)) − p∗ ≤ m/t

Interpretation as modified KKT condition: x is a central point x∗ (t) iff there exits λ, ν such that Ax = b, fi (x) ≤ 0,

i = 1, . . . , m

λ0 ∇f0 (x) +

m X

λi ∇fi (x) + AT ν = 0

i=1

−λi fi (x) = 1/t,

i = 1, . . . , m

Complementary slackness is relaxed from 0 to 1/t

Example Inequality form LP: minimize

cT x

subject to

Ax  b

Log barrier function: φ(x) = −

m X

log(bi − aT i x)

i=1

with gradient and Hessian: ∇φ(x) 2

∇ φ(x)

=

=

m X i=1 m X i=1

ai T d = A bi − aT x i ai aT T 2 i = A diag(d )A T (bi − ai x)2

where di = 1/(bi − aT i x) Centrality condition becomes: tc + AT d = 0

c is parallel to ∇φ(x). Therefore, hyperplane cT x∗ (t) is tangent to level set of φ

c

xo

xs

Barrier Method GIVEN a strictly feasible point x ∈ dom f, t := t(0) > 0, µ > 1 and tolerance ǫ > 0 REPEAT 1. Centering step: compute x∗ (t) by minimizing tf0 + φ subject to Ax = b, starting at x 2. Update: x := x∗ (t) 3. Stopping criterion: QUIT if

m t

≤ǫ

4. Increase t: t := µt

Other names: Sequential Unconstrained Minimization Technique (SUMT) or path-following method Usually use Newton method for Centering Step

Remarks • Each centering step does not need to be exact • Choice of µ: tradeoff number of inner iterations with number of outer iterations • Choice of t(0) : tradeoff number of inner iterations within the first outer iteration with number of outer iterations • Number of centering steps required: log(m/(ǫt(0) )) log µ where m is the number of inequality constraints and ǫ is desired accuracy

duality gap

Progress of Barrier Method for an LP Example

2

10

0

10

−2

10

−4

10

−6

10

0

c2 c3

20

Three curves for µ = 2, 50, 150

40

c1

60

NT. Ite.

80

Tradeoff of µ Parameter for a Small LP

NT. Ite.

140 120 100 80 60 40 20 0 0

20

40

60

80

100 120 140 160 180 200

µ

duality gap

Progress of Barrier Method for a GP Example 2

10

0

10

−2

10

−4

10

−6

10

0

c3

20

40

c2

c1

60

80

NT Ite.

100

120

Insensitive to Problem Size 35

4

NT. Ite.

duality gap

10

30

2

10

0

25

10

−2

20

10

c1

c2

c3

−4

10

0

10

20

30

NT. Ite.

40

50

15 1 10

Three curves for m = 50, 500, 1000, n = 2m.

2

10

m

3

10

Phase I Method How to compute a strictly feasible point to start barrier method? Consider a phase I optimization problem in variables x ∈ Rn , s ∈ R: minimize

s

subject to

fi (x) ≤ s, i = 1, . . . , m Ax = b

Strictly feasible point: for any x(0) , let s = max fi (x(0) )

1. Apply barrier method to solve phase I problem (stop when s < 0) 2. Use the resulted strictly feasible point for the original problem to start barrier method for the original problem

Not Covered • Self-concordance analysis and complexity analysis (polynomial time) • Numerical linear algebra (large scale implementation) • Generalized inequalities (SDP)

Other (sometimes more efficient) algorithms: • Primal-dual interior point method • Ellipsoid methods • Analytic center cutting plane methods

Lecture Summary • Convert equality constrained optimization into unconstrained optimization • Solve a general convex optimization with interior point methods • Turn inequality constrained problem into a sequence of equality constrained problems that are increasingly accurate approximation of the original problem • Polynomial time (in theory) and much faster (in practice): about 25-50 least-squares effort for a wide range of problem sizes

Readings: Chapters 9.1-9.3, 9.5, 10.1-10.2, 11.1-11.4 in Boyd and Vandenberghe

Why Take This Course • Learn the tools and mentality of optimization (surprisingly useful for other study you may engage in later on) • Learn classic and recent results on optimization of communication systems (over a surprisingly wide range of problems) • Enhance the ability to do original research in academia or industry • Have a fun and productive time on the final project

Where We Started

Application Presentation Source

Session

Source Encoder

Channel Encoder

Transport

Modulator

Channel

Network

Desti.

minimize

f (x)

subject to

x∈C

Source Decoder

Channel Decoder

Demodulator

Topics Covered: Methodologies • Nonlinear optimization (linear, convex, nonconvex), Pareto optimization, Dynamic programming, Integer programming • Convex optimization, convex sets and convex functions • Lagrange duality and KKT optimality condition • LP, QP, QCQP, SOCP, SDP, GP, SOS • Gradient method, Newton’s method, interior point method • Distributed algorithms and decomposition methods • Nonconvex optimization and relaxations • Stochastic network optimization and Lyapunov function • Robust LP and GP • Network flow problems • NUM and generalized NUM

Topics Covered: Applications • Information theory problems: channel capacity and rate distortion for discrete memoryless models • Detection and estimation problems: MAP and ML detector, multi-user detection • Physical layer: DSL spectrum management • Wireless networks: resource allocation, power control, joint power and rate allocation • Network algorithms and protocols: multi-commodity flow problems, max flow, shortest path routing, network rate allocation, TCP congestion control, IP routing • Reverse engineering, heterogeneous protocols • Congestion, collision, interference management • Layering as optimization decomposition • Eigenvalue and norm optimization problems

Some of the Topics Not Covered • Circuit switching and optimal routing • Packet switch architecture and algorithms • Multi-user information theory problems • More digital signal processing like blind equalization • Large deviations theory and queuing theory applications • System stability and control theoretic improvement of TCP • Inverse optimization, robust optimization • Self concordance analysis of interior point method’s complexity • Ellipsoid method, cutting plane method, simplex method ...

Final Projects: Timeline March 30 – May 24: • Talk to me as often as needed • Start early

April 27: Interim project report due

Mid May: Project Presentation Day

May 24: Final project report due Many student continue to extend the project

Final Projects: Topics Wenjie Jiang: Content distribution networking Suhas Mathur: Wireless network secuirty Hamed Mohsenian: Multi-channel MAC Martin Suchara: Stochastic simulation for TCP/IP Guanchun Wang: Google with feedback Yihong Wu: Practical stepsize Yiyue Wu: Coding Yongxin Xi: Image analysis Minlan Yu: Network embedding Dan Zhang: To be decided

The Optimization Mentality • What can be varied and what cannot? • What’re the objectives? • What’re the constraints? • What’s the dual? Build models, analyze models, prove theorems, compute solutions, design systems, verify with data, refine models ...

Warnings: • Remember the restrictive assumptions for application needs • Understand the limitations: discrete, nonconvex, complexity, robustness ... (still a lot can be done for these difficult issues)

Optimization of Communication Systems Three ways to apply optimization theory to communication systems: • Formulate the problem as an optimization problem • Interpret a given solution as an optimizer/algorithm for an optimization problem • Extend the underlying theory by optimization theoretic techniques A remarkably powerful, versatile, widely applicable and not yet fully recognized framework

Applications in communication systems also stimulate new developments in optimization theory and algorithms

Optimization of Communication Systems OPT Problems

Solutions OPT

OPT

Theories

Optimization Beyond Optimality • Modeling (reverse engineering, fairness, etc.) • Architecture • Robustness • Design for Optimizability

This Is The Beginning Quote from Thomas Kailath:

A course is not about what you’ve covered, but what you’ve uncovered

To Everyone In ELE539A You’ve Been Amazing