Non-convex Optimization for Linear System with ... - Google Sites

Null Space Property for Sparse Recovery from MMV Probability Estimates on Largest q -singular Value Probability Estimates on Smallest q -singular Value Some Numerical Experiments

Non-convex Optimization for Linear System with Pregaussian Matrices and Recovery from Multiple Measurements Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)

University of Georgia July 15, 2010

Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)

Sparse Recovery by `q -minimization and q -Singular Values


`q -minimization

Introduction Null Space Property of `q -minimization for MMV Proof of the main theorem

problem

In compressed sensing, we want to recover a sparse or compressible signal via solving the minimization problem minimizex∈RN where

kxk0

kxk0

subject to

Ax = b

is the the number of non-zero entries of vector

namely the sparsity of

x,

and

A

is a matrix of size

m×N

x,

with

m N.




`q -minimization


problem

In compressed sensing, we want to recover a sparse or compressible signal via solving the minimization problem minimizex∈RN where

kxk0

kxk0

subject to

Ax = b

is the the number of non-zero entries of vector

namely the sparsity of

x,

and

A

is a matrix of size

m×N

x,

with

m N. `q -approach with 0 < q ≤ 1, The

is to consider the

minimizex∈RN

kxkq


`q -minimization

subject to

problem

Ax = b.




Null space property for SMV The null space property has been used to quantify the error of approximations (Cohen, Dahmen, and DeVore, 2009), and it also guarantees the exact recovery. Proposition (Null space property)

Let S ⊆ {1, 2, · · · , N } be a xed index set. Then a vector x with support (x) ⊆ S can be uniquely recovered from Ax = b using `1 -minimization if for all non-zero v in the null space of A, kvS k1 < kvS c k1 ,

where S c is the complement of S in {1, 2, · · · , N }.





Proof of the proposition Proof. We know for any non-zero

z

in the null space of

A,

kxS k1 ≤ kzS k1 + k(z + x)S k1 . By the assumption

kzS k1 < kzS c k1 ,

we then have

kxS k1 < k(z + x)S k1 + kzS c k1 . But

x

vanishes on

Sc,

thus

kxk1 = kxS k1 < k(z + x)S k1 + k(z + x)S c k1 = kz + xk1 , and so

x ∈ RN

is the unique solution to the

`1 -minimization

problem. Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)




Remarks to the proposition

Remark

`1

can be replaced by

`q

.





Remarks to the proposition

Remark

`1

can be replaced by

`q

.

Remark The converse is also true.





MMV problem

Denition Given a set of

r

measurements

Ax(k) = b(k) Find the vectors

x(k)

for

k = 1, · · · , r.

which are jointly sparse.





MMV problem

Denition Given a set of

r

measurements

Ax(k) = b(k) Find the vectors

x(k)

for

k = 1, · · · , r.

which are jointly sparse.

The MMV problem arises in biomedical engineering, especially in neuromagnetic imaging.





`1,2 -minimization

The approach of mixed

`1,2 -minimization

to recover jointly sparse

vectors from MMV is solving the optimization problem

minimize

N q X x21,j + · · · + x2r,j

subject to

Ax(k) = b(k)

j=1 for

k = 1, · · · , r.





Null space property for MMV Theorem (BergFriedlander, 2009)

Let A be a real matrix of m × N and S ⊂ {1, 2, · · · , N } be a xed index set. Denote by S c the complement set of S in {1, 2, · · · , N }. Let k · k be any norm. Then all x(k) with support x(k) in S for k = 1, · · · , r can be uniquely recovered using the following minimize

N X

k(x1,j , · · · , xr,j )k subject to Ax(k) = b(k)

j=1

for k = 1, · · · , r if and only if for all vectors

(u(1) , · · · , u(r) ) ∈ (N (A))r \{(0, 0, · · · , 0)} satisfy the following X X k(u1,j , · · · , ur,j )k < k(u1,j , · · · , ur,j )k, . j∈S Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)

j∈S c Sparse Recovery by `q -minimization and q -Singular Values



Real vs. complex null space properties Theorem (FoucartGribonval, 2009)

Let A be a matrix of size m × N and S ⊂ {1, · · · , N } be the support of the sparse vector y. The complex null space property: for any u ∈ N (A), w ∈ N (A) with (u, w) 6= 0, Xq Xq u2j + wj2 < u2j + wj2 , j∈S c

j∈S

where u = (u1 , u2 , · · · , uN )T and w = (w1 , w2 , · · · , wN )T is equivalent to the following standard null space property: for any u in the null space N (A) with u 6= 0, X

|uj |
0, there exists K > 0 such that (1) P s1 (A) ≤ Km ≤ ε

where K only depends on ε and the pregaussian variable ξ .




Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value

Lower tail probability of largest

1-singular

value

Proof. Since

aij

is pregaussian with variance

exists some

δ>0

1,

then for any

(1)

s1 (A) =

there

ε . 8

P (|aij | ≤ δ) ≤ Since

ε > 0,

such that

Pm

i=1 |aij0 | for some

j0 ,

by the previous lemma, we

have

! m X mδ mδ (1) P s1 (A) ≤ ≤ P ≤ 8P (|aij | ≤ δ) ≤ ε. |aij0 | ≤ 2 2 i=1

Finally let

K=

δ 2 , then the claim follows.




Introduction Upper Tail Probability of the Largest q -singular Value Lower Tail Probability of the Largest q -singular Value

Lower tail probability of largest

For general

0 < q ≤ 1,

q -singular

value

we have

Theorem (Lower tail probability of the largest

q -singular

value )

Let ξ be a pregaussian variable normalized to have variance 1 and A be an m × N matrix with i.i.d. copies of ξ in its entries, then for every 0 < q ≤ 1 and any ε > 0, there exists K > 0 such that 1 (q) P s1 (A) ≤ Km q ≤ ε

where K only depends on q , ε and the pregaussian variable ξ .




Smallest

q -singular

Introduction Lower Tail Probability of the Smallest q -singular Value Upper Tail Probability of the Smallest q -singular Value Additional Results: Probability Estimates on the Largest p-sing

value

Denition The smallest

q -singular

value of an

s(q) n (A) : =

n×n inf

matrix

x∈Rn , kxkq =1


kAxkq .




Smallest singular value in `2

Rudelson and Vershynin rst showed the following result Theorem (RudelsonVershynin, 2008)

If A is a matrix of size n × n whose entries are independent random variables with variance 1 and bounded fourth moment. Then for any δ > 0, there exists > 0 and integer n0 > 0 such that P sn (A) ≤ √ ≤ δ, n


∀n ≥ n0 .




Smallest singular value in `2

Later, they proved the following result Theorem (RudelsonVershynin, 2008)

Let A be an n × n matrix whose entries are i.i.d. centered random variables with unit variance and fourth moment bounded by B. Then, for every δ > 0 there exist K > 0 and n0 which depend (polynomially) only on δ and B, and such that K ≤ δ, P sn (A) > √ n


∀n ≥ n0




Lower tail probabilistic estimate on the smallest

q -singular

value For tall rectangular matrices, we have the lower tail probabilistic of exponential decay on the smallest

q -singular

value

Theorem

Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an m × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0, there exist some γ > 0 and c > 0 and r ∈ (0, 1) dependent on q and ε, such that 1 q P s(q) (A) < γm < e−cm n

if n ≤ rm. Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)





q -singular

value

Theorem

Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0 and 0 < q ≤ 1, there exist some K > 0 and c > 0 dependent on q and ε, such that 1 − 1q P s(q) < Cε + Cαn + P kAk > Kn− 2 . n (A) < εn

where α ∈ (0, 1) and C > 0 depend only on the pregaussian variable and K . Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)





q -singular

value

Theorem (Lower tail probabilistic estimate on the smallest

q -singular

value )

Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any ε > 0, there exists some γ > 0 such that − 1q P s(q) (A) < γn < ε, n

where γ only depends on q , ε and the pregaussian variable ξ .





Upper tail probability of the smallest

q -singular

value

Theorem (Upper tail probabilistic estimate on the smallest

q -singular

value)

Given any 0 < q ≤ 1, and let ξ be the pregaussian random variable with variance 1 and A be an n × n matrix with i.i.d. copies of ξ in its entries. Then for any K > e, there exist some C > 0, 0 < c < 1, and α > 0 only dependent on pregaussian variable ξ , q , such that − 21 P s(q) (A) > Kn ≤ n

C (ln K)α + cn . Kα

In particular, for any ε > 0, there exist some K > 0 and n0 , such that for all n ≥ n0 , − 21 < ε. P s(q) (A) > Kn n Louis Yang Liu (Advisor: Prof. Ming-Jun Lai)




Lower tail probability of the largest

p-singular

value

Theorem (Lower tail probability of the largest -singular value,

p > 1)

Let ξ be a pregaussian random variable normalized to have variance 1 and A be an m × N matrix with i.i.d. copies of ξ in its entries, then for every p > 1 and any ε > 0, there exists γ > 0 such that 1 (p) ≤ ε P s1 (A) ≤ γm p

where γ only depends on p, ε and the pregaussian random variable

ξ.





A Remark

Remark By the duality that for any

p≥1

and

(p)

(q)

s1 (A) = s1 where

1 p

+

1 q

= 1,

in particular, for

n×n AT

matrix

A,

(∞)

p = ∞, s1

(A)

is of order

n

with high probability.




2-singular For

value

p = 2,

we plot the largest

matrices of size

n × n,

Figure: Largest

where

2-singular

2-singular value of Gaussian n runs from 1 through 100.

random

value of Gaussian random matrices




1-singular

value

For p = 1, in the rst numerical experiment we plot the largest 1-singular value of Gaussian random of size n × n, where n runs from 1 through 100.

Figure: Largest

1-singular

value of Gaussian random matrices:

Experiment 1




1-singular

value

In the second numerical experiment for

1-singular value of Gaussian n runs from 1 through 200.

Figure: Largest

1-singular

p = 1,

we plot the largest

random matrices of size

n × n,

where


Experiment 2




1-singular

value

In the third experiment for

p = 1,

we plot the largest

value of Gaussian random matrices of size from

1

through

Figure: Largest

n × n,

1-singular n runs

where

400.

1-singular


Experiment 3




∞-singular For

value

p = ∞,

we plot the largest

random matrices of size

Figure: Largest

n × n,

∞-singular

∞-singular value of Gaussian where n runs from 1 through 500.

value of Gaussian random matrices




1 3 -singular value

For

p=

1 1 3 , we plot the largest 3 -singular value of Gaussian random

matrices of size

n × n,

Figure: Largest

where

n

runs from

1

through

500.

1 3 -singular value of Gaussian random matrices




1 4 -singular value

For

p=

1 1 4 , we plot the largest 4 -singular value of Gaussian random

matrices of size

n × n,

Figure: Largest

where

n

runs from

1

through

300.

1 4 -singular value of Gaussian random matrices




1 4 -singular value

For rectangular matrices, we also plot the largest of Gaussian random matrices of size from

1

through

Figure: Largest

m × n,

1 4 -singular value

where

m

and

n

run

100.

1 4 -singular value of rectangular Gaussian random matrices



Non-convex Optimization for Linear System with ... - Google Sites

Non-convex Optimization for Linear System with ... - Google Sites

Suggest Documents

Non-convex Optimization for Linear System with ... - Google Sites

Nonconvex Optimization for Communication Systems

Nonconvex Optimization Is Combinatorial Optimization

Exploiting Structure for Tractable Nonconvex Optimization - Google Sites

linear optimization - Google Sites

A SMART Stochastic Algorithm for Nonconvex Optimization with ...

Particle swarm optimization with crazy particles for nonconvex ...

Recursive Decomposition for Nonconvex Optimization - Washington

Nonconvex Optimization for Communication Systems - Semantic Scholar

Nonconvex optimization for improved exploitation ... - Semantic Scholar

Optimization with few violated constraints for linear

Deterministic Modeling: Linear Optimization with

NEXT: In-Network Nonconvex Optimization - IEEE Xplore

Duality Theory: Biduality in Nonconvex Optimization

NONCONVEX SEMI-LINEAR PROBLEMS AND CANONICAL ...

Introduction to Linear Optimization - Google Sites

PDF Introduction to Linear Optimization - Google Sites

Approaching nonsmooth nonconvex optimization problems through ...

Globally Solving Nonconvex Quadratic ... - Optimization Online

Efficient global optimization of nonconvex and

Improved particle swarm optimization approach for nonconvex static ...

A Global Optimization Method for General Constrained Nonconvex ...

DC Programming and DCA for Nonconvex Optimization

Graphical Nonconvex Optimization for Optimal Estimation in Gaussian ...