Introduction to Convex Optimization

Introduction to Convex Optimization

Chee Wei Tan

CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010

Outline • Optimization • Examples • Overview of Syllabus • An Example: The Illumination Problem • References Acknowledgement: Thanks to Mung Chiang (Princeton), Stephen Boyd (Stanford) and Steven Low (Caltech) for the course materials in this class.

1

Optimization Mentality OPT Problems

Solutions OPT

OPT

Theories Optimization as a language, as an engineering system, as a unifying principle 2

Optimization

minimize f (x) subject to x ∈ C Optimization variables: x. Constant parameters describe objective function f and constraint set C 3

Questions • How to describe the constraint set? • Can the problem be solved globally and uniquely? • What kind of properties does it have? another optimization problem?

How does it relate to

• Can we numerically solve it in an efficient, robust, and distributed way? • Can we optimize over a sequence of time instances? • Can we find the problem for a given solution? 4

Solving Optimization Problems • general nonlinear optimization problem often difficult to solve, e.g., if nonconvex methods involve some compromise, e.g., very long computational time • local optimization find a suboptimal solution computationally fast but initial point dependent • global optimization find a global optimal solution computationally slow In convex optimization, the art and challenge is in problem formulation. In nonconvex optimization, the art and challenge is in problem structure. Convex optimization plays an important role in exploiting this problem structure 5

Perspective Widely known: linear programming is powerful and easy to solve Modified view: ... the great watershed in optimization isn’t between linearity and nonlinearity, but convexity and nonconvexity.– SIAM Review 1993, R. Rockafellar • Local optimality is also global optimality • Lagrange duality theory well developed • Know a lot about the problem and solution structures • Efficiently compute the solutions numerically So we’ll start with convex optimization introduction, then specialize into different applications involving convex problems (and nonconvex problems) 6

Examples • Least-Squares:

minimizex kAx − bk2

Analytical solution: x? = (AT A)−1AT b; reliable and efficient algorithms and software; computational time proportional to n2k (A ∈ Rk×n); less with structure; a mature technology • Using Least-Squares: several standard techniques increase the flexibility of least squares, e.g., adding weights, regularization • Linear Programming: minimize cT x subject to aTi x ≤ bi, i = 1, . . . , m No analytical formula for solution; reliable and efficient algorithms and software; computational time proportional to n2m if m ≥ n; less with structure; a mature 7

technology Not as easy to recognize as least squares • Convex Optimization: minimize f0(x) subject to fi(x) ≤ bi, i = 1, . . . , m No analytical formula for solution; reliable and efficient algorithms and software; computational time (roughly) proportional to max{n3, n2m, F } where F is the cost of evaluating the objective and constraint functions fi and their first and second order derivatives; almost a technology • Using convex optimization Often difficult to recognize, many tricks for transforming problems, surprisingly many problems can be solved (or be tackled elegantly) via convex optimization Once the formulation is done, solving the problem is, like least-squares or linear programming, (almost) technology. Learn by examples and applications 8

Why Applications? լᐬլඔΔ լ↾լ࿇Ζ ᜰԫၻլ‫א‬ Կၻ֘Δ ঞլ༚ՈΖ πᓵ፿ ૪ۖ รԮρ

Draw inferences about other cases from one instance Need to know how to recognize and formulate convex optimization, and use optimization tools to solve your problem (an objective of this course) 9

Convex Sets and Convex Functions f (y) f (x) + ∇f (x)T (y − x) (x, f (x))

10

Convex Optimization and Lagrange Duality

11

Linear Programming and Quadratic Programming

−c x∗ P

12

Geometric Programming

(a + b)/2

√

ab

13

Semidefinite Programming

y K K∗

x

14

Decomposition Master Problem prices / resources

Secondary Master Problem prices / resources

... Subproblem 1

Subproblem N

Subproblem

First Level Decomposition

Second Level Decomposition

15

Internet Congestion Control & Network Utility Maximization 92% 2G

48% Average utilization 95% 1G

27% 16% 19%

txq= 100

Linux TCP

txq= 10000

Linux TCP

FAST

txq= 100

txq= 10000

Linux TCP

Linux TCP

FAST

16

Power Control in Wireless

17

Uncovering the Perron-Frobenius Root

Rate region (units in nats/symbol) 5

log(1 + 1/ρ(F + (1/¯ pi)veTi ))(1, 1)T 4

3

r2

w = x(F + (1/¯ pi)veTi ) ◦ y(F + (1/¯ pi)veTi )

2

90 degrees

1

0 0

1

2

r1

3

4

5

18

Combinatorial Problems

Maximize the number of edges between a subset and its complementary subset 19

Convex-Cardinality Problems

Sparsity, Compressed Sensing, Single-Path Routing

20

Data Mining

Face ≈ eyes + nose + lip + chin + etc

21

Illumination Problem m lamps illuminating n (small, flat) patches

Intensity Ik at patch k depends linearly on lamp powers pj : Ik =

m X

akj pj ,

−2 akj = rkj max{cos θkj , 0}

j=1

problem: achieve desired illumination Ides with bounded lamp powers minimize

max | log Ik − log Ides|

k=1,...,n

subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m 22

How to Solve Illumination Problem 1. use uniform power: pj = p, vary p 2. use least-squares: minimize

n X

2

(Ik − Ides)

k=1

round pj if pj > pmax or pj < 0 3. use weighted least-squares:

minimize

n X

k=1

2

(Ik − Ides) +

m X

2

wj (pj − pmax)

j=1

iteratively adjust weights wj until 0 ≤ pj ≤ pmax 23

4. use linear programming: minimize

max |Ik − Ides|

k=1,...,n

subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m which can be solved via linear programming

of course these are suboptimal ‘solutions’ (to the original problem)

5. use convex optimization: problem is equivalent to minimize

f0(p) = max h (Ik /Ides) k=1,...,n

subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m

with h(u) = max{u, 1/u} 24

5

4

h(u)

3

2

1

0 0

1

2 u

3

4

f0(p) is convex because maximum of convex functions is convex exact solution obtained with effort ≈ modest factor × least-squares effort additional constraints: does adding (a) or (b) below complicate the problem? (a) no more than half of total power is in any 10 lamps (b) no more than half of the lamps are on (pj > 0) 25

• answer: with (a), still easy to solve; with (b), extremely difficult • moral: (untrained) intuition doesn’t always work; without the proper background very easy problems can appear quite similar to very difficult problems 6. Can you think of yet another way to solve the Illumination Problem exactly?

26

References • Primary Textbook: Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge University Press, 2004 http://www.stanford.edu/∼boyd/cvxbook • Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering, by A. Ben-Tal and A. Nemirovski, SIAM, 2001 • Parallel and Distributed Computation: Numerical Methods by D. Bertsekas and J. N. Tsitsiklis, 1st Edition, Prentice Hall, 1989

27