A Unified Approximate Bayesian A Unified

A Unified Approximate Bayesian A Unified Approximate Bayesian Inference Framework for Generalized Li Linear Models M d l

Xiangming Meng Huawei Technologies Co. Ltd., Shanghai, China [email protected] C ll b t Collaborators: Sheng Wu (Beijing University of Post and Telecommunications) Sh W (B iji U i it f P t d T l i ti ) and Jiang Zhu (Zhejiang University) Physics, Inference, and Learning (PIL) 2018 Physics Inference and Learning (PIL) 2018 Nov. 3 2018, Beijing 1

Outline • Problem Statement • AMP: Review and EP perspective • A Unified Approximate Inference Framework A Unified Approximate Inference Framework – Generalized AMP, SBL and VAMP – Bilinear Adaptive Generalized VAMP

• Conclusion

2

Problem Statement  Generalized Linear Models (GLM) ( ) Signal of interest

x  p0  x

x  N

linear mixing

A

M N

probabilistic mapping

z  M

p ( ya | z a )

observations

y  M

• Goal To infer the input x (and/or z) given the output y and A, assuming the distributions of x and p(y|z) are known are known

3

Problem Statement  Generalized Linear Models (GLM) ( ) Signal of interest

x  p0  x

x  N

linear mixing

A

M N


z  M

p ( ya | z a )

observations

y  M


• Applications  Compressed sensing (CS), quantized or 1 Compressed sensing (CS), quantized or 1‐bit bit CS CS  Wireless signal detection: code division multiple access (CDMA), multiple input multiple output (MIMO) in 5G communications, channel estimation, etc.  linear regression or classification and a variety of linear inverse problems

4

Problem Statement  Generalized Linear Models (GLM) ( ) linear mixing

Signal of interest

x  p0  x

x  N

A


M N

z  M

p ( ya | z a )

observations

y  M


• Applications  Compressed sensing (CS), quantized or 1 Compressed sensing (CS), quantized or 1‐bit bit CS CS  Wireless signal detection: code division multiple access (CDMA), multiple input multiple output (MIMO) in 5G communications, channel estimation, etc.  linear regression or classification and a variety of linear inverse problems

• Special case: standard linear models In particular, if p(y|z) is Gaussian, GLM reduces to the common standard linear models (SLM) n  M Signal of interest

linear mixing

x  N

A   M N

x  p0  x

y  Ax  n

observations

y  M n ~  (0, 2I )

5


Signal of interest

x  p0  x

x  N

A

M N


z  M

p ( ya | z a )

observations

y  M

• Optimal Bayesian estimation According to the Bayes’ rule, the posterior distribution can be computed as

p( x | y ) 

p0 (x) p(y | x) p( y )

Posterior mean Posterior mean

marginalize

p( xi | y)   p(x | y)dx\ i  xi

xîMMSE   xi p( xi | y)dxi

Posterior variance viMMSE   xi2 p ( xi | y )dxi   xîMMSE 

2

MMSE MMSE estimates

6


Signal of interest

x  p0  x

x  N

A

M N


z  M

p ( ya | z a )

observations

y  M

• Optimal Bayesian estimation According to the Bayes’ rule, the posterior distribution can be computed as

p( x | y ) 

p0 (x) p(y | x) p( y )

Posterior mean Posterior mean

marginalize

p( xi | y)   p(x | y)dx\ i  xi

xîMMSE   xi p( xi | y)dxi

Posterior variance viMMSE   xi2 p ( xi | y )dxi   xîMMSE 

2

MMSE MMSE estimates

Curse of Dimensionality: The optimal Bayesian inference becomes intractable in high dimensional case due to integration (or summation) operation

We have to resort to approximate inference methods 7

AMP: review and EP perspective  Approximate message passing (AMP) pp g p g( ) AMP iteratively decouples the original vector inference problem to scalar inference problems Initialization AMP Algorithm ìï R = x + n 1 1 ïï 1 For t=1,…,T y = Ax + n  íï  Vat = å Aai2 nit ïï i ïï RN = x N + nN (y - Zat -1) t î A xˆt - a Zt = V • Evolution of AMP a

 Proposed in the field compressed sensing (CS) [DMM09]  Early work in communications Early work in communications [Kabashima [K b hi 03] [T k 02] 03] [Tanaka 02]  Deeply related to TAP equations and replica methods in statistical physics [KMSSZ12]  Extended with EM learning [KMSSZ12][VS11]  Extended to Generalized AMP (GAMP) [Rangan12] for GLM models end  Extended to vector AMP (VAMP) [RSF16] orthogonal AMP (OAMP) [ML17]  Many other extensions….

å

ai i

i

Sti

= 1/ å a

s2 + Vat -1

Aai2

s2 + Vat

Rit = xît + Sti å

a

Onsager term Onsagerterm

Aai (ya - Zat )

s2 + Vat a t +1 xi = E ( xi | Rit , Sti ), nît +1 = Var ( xi | Rit , Sti )

8



å

ai i

i

Sti

= 1/ å a

s2 + Vat -1

Aai2

s2 + Vat


a


Aai (ya - Zat )


• Properties of AMP  For i.i.d. Gaussian matrix A, asymptotically optimal and rigorously analyzed via state evolution (SE) [BM11]

 For general matrices A, AMP may diverge For general matrices A AMP may diverge [BM11] [BM11]  VAMP converges for right‐rotationally invariant matrices [RSF16]

9



å

ai i

i

Sti

= 1/ å a

s2 + Vat -1

Aai2

s2 + Vat


a


Aai (ya - Zat )


• Properties of AMP  For i.i.d. Gaussian matrix A, asymptotically optimal and rigorously analyzed via state evolution (SE) [BM11]

 For general matrices A, AMP may diverge For general matrices A AMP may diverge [BM11] [BM11]  VAMP converges for right‐rotationally invariant matrices [RSF16]

• Derivation of AMP  Originally derived from belief propagation (BP) via central limit theorem and Taylor series expansion [DMM09] [DMM10]

 Alternatively derived from expectation propagation (EP) via neglecting high order terms [MWKL15a]

10

AMP: review and EP perspective  An EP Perspective on AMP p • Expectation Propagation (EP) [Minka01] [MO05]

p(x) =

 fa (x) a

Optimization objective:

approximated as approximated as

q(x) =

 fa (x) a

min KL(p(x) || q (x)) q(x) = h(x)exp { qT f(x) + g(q)}

11


p(x) =

 fa (x) a



q(x) =

 fa (x) a

min KL(p(x) || q (x)) q(x) = h(x)exp { qT f(x) + g(q)} Iterative local optimization

Iteratively Refine each factor

fa (x) =

arg min i KL( fa (x) fb (x) || t(x) fb (x)) t (x)ÎF

b ¹a

b ¹a

12


p(x) =

 fa (x) a



q(x) =

 fa (x) a

min KL(p(x) || q (x)) q(x) = h(x)exp { qT f(x) + g(q)} Iterative local optimization

Iteratively Refine each factor

fa (x) =

arg min i KL( fa (x) fb (x) || t(x) fb (x)) t (x)ÎF

b ¹a

Properties of EP:  Applicable to both discrete and continuous distributions  Equivalent to moment matching  Deeply related to the adaptive Cavity Method in l l d h d h d statistical physics

b ¹a

p ( x)

min KL(p||q)

 q ( x)  Proj  q (x)  13

AMP: review and EP perspective  An EP Perspective on AMP p Target distribution

p(x|y) µ

N

 p0 (x i )  i =1 N

Approximate distribution pp q (x ) µ

M

a =1 M

(ya ; ( Ax )a , s 2 ) N

 q 0 (x i )   q a i (x i ) i=1

fully factorized form y

a =1 i=1

14


p(x|y) µ

N

 p0 (x i )  i =1 N


M

a =1 M

(ya ; ( Ax )a , s 2 ) N

 q 0 (x i )   q a i (x i ) i=1


a =1 i=1

• Then, we can iteratively refine all the approximating factors using EP principle • Intuitively, this optimization process can be realized in an message passing manner as BP



fa

f1

Expectation Propagation (EP)

fM

mat i (x i )

messagepassing

x1

x2

xi

p0 (x1 )

p0 (x 2 )

p0(xi )



+1 mit a (x i )

xN p0(xN )

mat i (xi ) µ

ProjF[mita (xi )ò

+1 mit a (xi ) µ

mtj a (x j )p(ya |x)] j ¹i

mita (xi )

ProjF[p0 (xi )mbt i (xi )] b t ma i (xi )

• Choosing the projection set as Gaussian and neglecting high‐order terms results in the AMP [MWKL15a]

15


p(x|y) µ

N

 p0 (x i )  i =1 N


M

a =1 M

(ya ; ( Ax )a , s 2 ) N

 q 0 (x i )   q a i (x i ) i=1


a =1 i=1

• Then, we can iteratively refine all the approximating factors using EP principle • Intuitively, this optimization process can be realized in an message passing manner as BP



fa

f1

Expectation Propagation (EP)

fM

mat i (x i )

messagepassing

x1

x2

xi

p0 (x1 )

p0 (x 2 )

p0(xi )



+1 mit a (x i )

xN

mat i (xi ) µ

ProjF[mita (xi )ò

+1 mit a (xi ) µ

mtj a (x j )p(ya |x)] j ¹i

mita (xi )

ProjF[p0 (xi )mbt i (xi )]

p0(xN )

b t ma i (xi )

• Choosing the projection set as Gaussian and neglecting high‐order terms results in the AMP [MWKL15a] • The EP perspective of AMP:  Explicitly establishes the relationship between AMP and EP[MWKL15a, MWKL15b, WKNLHDQ14]  Facilitates the extension of AMP to the complex‐valued AMP (simply using circularly‐symmetric Gaussian) [MWKL15b]  Provides a unified view of AMP (derived from scalar EP Provides a unified view of AMP (derived from scalar EP [MWKL15a] [MWKL15a] ) and VAMP(derived from vector EP [RSF16] ) [MWKL15a] X. Meng, S. Wu, L. Kuang, and J. Lu, “An expectation propagation perspective on approximate message passing,” IEEE Signal Process. Lett., vol. 22, no. 8, pp. 1194‐1197, Aug. 2015.

BP

EP

CLT & Taylor

AMP

Scalar factor graph

Vector factor graph

VAMP

Circularly‐symmetric Gaussian

Complex‐valued‐version

16

A Unified Inference Framework for GLM  Motivations • GLM is more general: the measurements are often obtained in a nonlinear way  quantized measurements, e.g., 1‐bit CS, low resolution ADC in wireless communication  incomplete measurements  non‐Gaussian and/or non‐additive noise  discrete measurements, and so on….

17

A Unified Inference Framework for GLM  Motivations • GLM is more general: the measurements are often obtained in a nonlinear way  quantized measurements, e.g., 1‐bit CS, low resolution ADC in wireless communication  incomplete measurements  non‐Gaussian and/or non‐additive noise  discrete measurements, and so on…. • SLM SLM has already been extensively studied has already been extensively studied  simple to design and analyze  a variety of Bayesian methods, e.g., AMP and sparse Bayesian learning (SBL) have been proposed  extension of one existing SLM method to GLM needs careful design, which is often difficult to follow and task‐specific, such as the conventional extensions from AMP to GAMP, VAMP to GVAMP

18

A Unified Inference Framework for GLM  Motivations • GLM is more general: the measurements are often obtained in a nonlinear way  quantized measurements, e.g., 1‐bit CS, low resolution ADC in wireless communication  incomplete measurements  non‐Gaussian and/or non‐additive noise  discrete measurements, and so on…. • SLM SLM has already been extensively studied has already been extensively studied  simple to design and analyze  a variety of Bayesian methods, e.g., AMP and sparse Bayesian learning (SBL) have been proposed  extension of one existing SLM method to GLM needs careful design, which is often difficult to follow and task‐specific, such as the conventional extensions from AMP to GAMP, VAMP to GVAMP • Natural question: Is there a unified framework for GLM, under which SLM inference methods can be easily extended to GLM ones following a common rule? easily extended to GLM ones following a common rule?

Unified framework? SLM Inference Algorithms

GLM Inference Algorithms

19

A Unified Inference Framework for GLM  Two Equivalent Factor Graphs for GLM q p p0  x 

x

p y | x

(a) Factorgraphv1(vectorform)

p0  x 

x

  z  Ax 

z

p y | z

(b)Factorgraphv2(vectorform)

20


p y | x

x

p0  x 

  z  Ax 

x

z

p y | z



 Decoupling GLM into SLM via EP p0  x 

  z  Ax 

x

z

mz  p  z 

p y | z

m p z  z 

m tz1p  z     z; z Aext (t  1), 1) v Aext (t  1) I 

m p z  z   t



Proj p  y | z  mtz1p  z  mtz1p  z 

EPmessagepassing (t-th h iteration)

    z; z

ext B

(t ), ) vBext (t ) I  21


p y | x

x

p0  x 

  z  Ax 

x

z

p y | z



 Decoupling GLM into SLM via EP p0  x 

  z  Ax 

x

z

mz  p  z 

p y | z

m p z  z 

m tz1p  z     z; z Aext (t  1), 1) v Aext (t  1) I 

m p z  z   t



Proj p  y | z  mtz1p  z  mtz1p  z 

EPmessagepassing (t-th h iteration)

    z; z

ext B

(t ), ) vBext (t ) I  22

A Unified Inference Framework for GLM  Decoupling GLM into SLM via EP p g p0  x 

x

 z  Ax 

Pseudo SLM

z

m p z z 

mz  p  z 

p y | z

MMSE module B

23

A Unified Inference Framework for GLM  Decoupling GLM into SLM via EP p g p0  x 

 z  Ax 

x

z

mz  p  z 

m p z z 

p y | z

MMSE module B

Pseudo SLM

• The original GLM is iteratively decoupled into a sequence of SLM problems The original GLM is iteratively decoupled into a sequence of SLM problems ext Component-wise zext A (t 1), vA (t 1)

y  Ax  n

MMSE

n ~  (0, 2I)

 2 ext y  zext B (t ),   vB (t ) Module A

Module B

1 1 1  post  ext v ( t ) v A (t ) v B (t ) ext A

ext A ext A

z (t ) z  v (t ) v

post A post A

ext z ext B (t ), v B (t )

ext  z Bpost  E  z | zext A , vA  ext  vBpost  Var  z | zext A , vA 

ext B ext B

(t ) z (t )  (t ) v (t )

1 1 1   vBext (t ) vBpost (t ) v Aext (t  1) z ext z Bpost (t ) z ext B (t ) A (t  1)   vBext (t ) v Bpost (t ) v Aext (t  1)

Unified Inference Framework for GLM ext ), v Aext (0) ( ) • Initialization Initialization z A ((0), • Fort=1:T,Do 1.Performcomponent-wiseMMSE ext 2.Update z ext B (t ), vB (t ) 3.PerformSLMinferenceoneormore iterations z Apost (t ), v Apost (t ) 4.Computeandthen ext update z ext A (t ), v A (t )

Note: The computation of posterior mean and variance of z in module A may differ for different SLM inference methods.

[MWZ18] X. Meng, S. Wu and J. Zhu, “A unified Bayesian inference framework for generalized linear model,” IEEE Signal Process. Lett., vol. 25, no. 3, Mar. 2018.

24

A Unified Inference Framework for GLM  From AMP to Gr-AMP [[MWZ18]] AMP (T0 iterations) Module A d l

ext zext A (t 1), vA (t 1) ext z ext B (t ), v B (t )

Component-wise MMSE Module B

The Gr-AMP Algorithm h l i h ext ext • Initialization z A (0), v A (0) • Fort=1:T,Do 1.Performcomponent-wiseMMSE ext 2.Update p z ext B (t ), vB (t ) 3.PerformAMPforT0iterations z Apost (t ), v Apost (t ) 4.Computeandthen ext ext update z A (t ), v A (t )

• Gr-AMP is a double-loop iterative algorithm Gr AMP is a double loop iterative algorithm   

Intheouter-loop,moduleAandBexchangesextrinsicmessages ext ext ItisprovedforAMP,theoutputmessageofmoduleAhasalreadycomputedwithinAMP,i.e.,zA (t)  Za(t),vA (t) Va(t) Ineachinner-loop,moduleAperformsAMPforT0iterations,ratherthanbeingfixedto1asGAMP.

25

A Unified Inference Framework for GLM  From AMP to Gr-AMP [[MWZ18]] AMP (T0 iterations) Module A d l

ext zext A (t 1), vA (t 1) ext z ext B (t ), v B (t )


The Gr-AMP Algorithm h l i h ext ext • Initialization z A (0), v A (0) • Fort=1:T,Do 1.Performcomponent-wiseMMSE ext 2.Update p z ext B (t ), vB (t ) 3.PerformAMPforT0iterations z Apost (t ), v Apost (t ) 4.Computeandthen ext ext update z A (t ), v A (t )

• Gr-AMP is a double-loop iterative algorithm Gr AMP is a double loop iterative algorithm   

Intheouter-loop,moduleAandBexchangeextrinsicmessages ext ext ItisprovedforAMP,theoutputmessageofmoduleAhasalreadycomputedwithinAMP,i.e.,zA (t)  Za(t),vA (t) Va(t) Ineachinner-loop,moduleAperformsAMPforT0iterations,ratherthanbeingfixedto1asGAMP.

• Relation of Gr-AMP to GAMP

 Gr-AMPispreciselyequivalenttoGAMPwhenT0=1andthusprovidesaninsightfulperspective onGAMP:Ineffect,GAMPperformsoneiterationofAMPeachtimeaftertransformingtheGLM problem to a pseudo SLM problem Note that [[QZW18] ] also provides an EP derivation of GAMP problemtoapseudoSLMproblem.Notethat[[QZW18] ]alsoprovidesanEPderivationofGAMP butinadifferentway.  Amoreflexiblemessagepassingschedule:double-loopimplementation • QuantizedCSfor1,2,3-bitcases:N=1024,M=512,SNR=50dB • Gr-AMPandGAMPconvergetothesameperformance fori.i.d.GaussianA • TotalnumberiterationsofAMPareaboutthesamewhilethe T t l b it ti f AMP b t th hil th numberofMMSEoperationsisreducedforGr-AMP. Stillneedsfurtherstudy. 26

A Unified Inference Framework for GLM  From VAMP/SBL to Gr-AMP/Gr-SBL / / [[MWZ18]] VAMP/SBL (T0 iterations)

ext zext A (t 1), vA (t 1)

Module A d l



The Gr-VAMP/Gr-SBL Algorithm h l i h ext ext • Initialization z A (0), v A (0) • Fort=1:T,Do 1.Performcomponent-wiseMMSE ext 2.Update p z ext B (t ), vB (t ) 3.PerformVAMP/SBLforT0iterations z Apost (t ), v Apost (t ) 4.Computeandthen ext ext update z A (t ), v A (t )

• Gr-VAMP/Gr-SBL

 Intheouter-loop,moduleAandBexchangesextrinsicmessages  TheposteriormeanandcovarianceofzinmoduleAcanbecomputedas:

z Apost  Axˆ A , v Apost  

1 Trace( A A AT ) N

xˆ A  A aretheposteriormeanandcovarianceofx computedinVAMP/SBL

In each inner-loop module A performs VAMP/SBL for T0 iterations Ineachinner-loop,moduleAperformsVAMP/SBLforT0iterations

27

A Unified Inference Framework for GLM  From VAMP/SBL to Gr-AMP/Gr-SBL / / [[MWZ18]] ext zext A (t 1), vA (t 1)

VAMP/SBL (T0 iterations)

Component-wise MMSE


Module A d l

Module B

The Gr-VAMP/Gr-SBL Algorithm h l i h ext ext • Initialization z A (0), v A (0) • Fort=1:T,Do 1.Performcomponent-wiseMMSE ext 2.Update p z ext B (t ), vB (t ) 3.PerformVAMP/SBLforT0iterations z Apost (t ), v Apost (t ) 4.Computeandthen ext ext update z A (t ), v A (t )

• Gr-VAMP/Gr-SBL

 Intheouter-loop,moduleAandBexchangeextrinsicmessages  TheposteriormeanandcovarianceofzinmoduleAcanbecomputedas:

z Apost  Axˆ A , v Apost 

xˆ A  A aretheposteriormeanandcovarianceofx computedinVAMP/SBL

In each inner-loop module A performs VAMP/SBL for T0 iterations Ineachinner-loop,moduleAperformsVAMP/SBLforT0iterations dNMSE (dB)



1 Trace( A A AT ) N

0 -10 -20 -30 0

5

10

15

20

25

30

35

40

Gr-AMP GAMP Gr-VAMP GVAMP Gr-SBL 45 50

dNMSE (dB)

(a) Number of Iterations ( (A) = 1) 0 -10 -20 -30 0

5

10

15

20

25

30

35

40

45

50

dNMSE (dB)

(b) Number of Iterations ( (A) = 100) 0 -10 -20 -30 30 10 0

10 1

10 2

10 3

(c) Condition Number

10 4

10 5

10 6

Performanceofde-biasedNMSEfor1-bitCS  N =512,M=2048,SNR=50dB, sparse ratio 0.1  T0 = 1 for f both b th Gr-VAMP G VAMP and d Gr-SBL G SBL  When conditional number is 1, all kinds of algorithms performs nearly the same.  As the condition number increases, the recovery performances f d degrade d smoothly hl for f Gr-VAMP/GVAMP/Gr-SBL while both Gr-AMP and GAMP diverge for even mild condition number, which show the robustness of GrVAMP/Gr-SBL/GVAMP G S G f generall matrices. for i

(A)

[MWZ18]X. Meng, S. Wu and J. Zhu, “A unified Bayesian inference framework for generalized linear model,” IEEE Signal Process. Lett., vol. 25, no. 3, Mar. 2018.

28

A Unified Inference Framework for GLM  Bilinear GLM Problems Signal

X  p0  X;X 

Component-wise probabilistic mapping

Affine Mapping

x   N L

A ()

Z   M L

p(Y | Z;Y )

observations

Y   M L Q

A(b)  A 0   b q A q A(.) is a known affine linear function of unknown vector b, i.e., q 1 and known A 0 , A q

• Goal

To jointly infer X and b, given Y with unknown parameters j y g p  X , Y

29

A Unified Inference Framework for GLM  Bilinear GLM Problems Signal

X  p0  X;X 

Component-wise probabilistic mapping

Affine Mapping

x   N L

A ()

Z   M L

p(Y | Z;Y )

observations

Y   M L Q

A(b)  A 0   b q A q A(.) is a known affine linear function of unknown vector b, i.e., q 1 and known A 0 , A q

• Goal

To jointly infer X and b, given Y with unknown parameters j y g p  X , Y

• Applications  Quantized Compressed sensing (CS) under matrix uncertainty  Self‐calibration, dictionary learning, matrix completion from nonlinear measurements  Joint signal detection and channel estimation in wireless communications  Many others…

• Special case: standard bilinear models In particular, if Y = Z + N, where N is i.i.d Gaussian noise, BGLM reduces to the standard bilinear models 30

A Unified Inference Framework for GLM  Bilinear GLM Problems

• The optimal estimate is the Maximum likelihood (ML) and MMSE estimate as

Intractable! Evidence(partition function)

Posterior distribution 31

A Unified Inference Framework for GLM  Bilinear Adaptive VAMP (BAd-VAMP) p ( )

[SFRS18] 1

• Bad-VAMPisproposedbySarkar, Flecher, Rangan, Schniter [SFRS18]veryrecentlytoaddressthebilinearrecovery fromlinearmeasurementsusingtheadaptiveVAMPand EMlearningframework. f • EarlyworkonbilinearrecoverybasedonAMPmethods, e.g.,BiGAMP[PSC14a] [PSC14b] , PBGAMP[PS16] andrelated worksbyKabashima, Krzakala k b and Mézard [KKMSZ16] [KMZ13] • ComparedwithAMPbasedmethods,Bad-VAMPshows improvedconvergenceovergeneralmatrices.

Figure 2 copied from [SFRS18]

32

A Unified Inference Framework for GLM  Bilinear Adaptive VAMP (BAd-VAMP) p ( )

[SFRS18] 1

• Bad-VAMPisproposedbySarkar, Flecher, Rangan, Schniter [SFRS18]veryrecentlytoaddressthebilinearrecovery fromlinearmeasurementsusingtheadaptiveVAMPand EMlearningframework. f • EarlyworkonbilinearrecoverybasedonAMPmethods, e.g.,BiGAMP[PSC14a] [PSC14b] , PBGAMP[PS16] andrelated worksbyKabashima, Krzakala k b and Mézard [KKMSZ16] [KMZ13] • ComparedwithAMPbasedmethods,Bad-VAMPshows improvedconvergenceovergeneralmatrices.

Figure 2 copied from [SFRS18]

Question: How to extend BAd-VAMP to general nonlinear measurements ? 33

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] p0  X; A 

X

  Z  A  A  X 

mx   X 

FactorgraphofBGLM

Z

p  Y | Z;  Y 

mz  p  Z 

m p z  Z 

34

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) p ( ) [[MZ18]] p0  X; A 

  Z  A  A  X 

X

p  Y | Z;  Y 

Z

mx   X 

mz  p  Z 

m p z  Z 

FactorgraphofBGLM

• Similar to GLM, using EP, the BGLM can be decoupled into two modules Similar to GLM, using EP, the BGLM can be decoupled into two modules mz  p  Z      z l ; z , v I    mz  p  z l 

1 1 1  post  ext v (t ) vB (t ) v A (t  1)

m p z  Z      z l ; z , v I 

z ext B , l (t )

L

L

ext A ,l

ext A

l 1 L

l 1

ext B ,l

l 1

mz  p  z l  

ext B



v Bext (t )



Projj m p  z  z l     z l  A  A  xl mx   xl  dxl m p z  z l 

ext ext   zBpost ,l  E  zl | z A,l , vA 

ext B

z Bpost , l (t ) vBpost (t )



z ext ) A , l (t  1)

ext  vBpost  Var  zl | zext A,l , vA 

v Aext (t  1)

  Proj  q

 zl  m p z  zl  

A

mx   x l  hasalreadybeencomputedwithinBAd-VAMPas,then m   x     x ; r ,1 /  I  x

qA  zl   

z ; z l

post A ,l

,  Apost ,l  1

T  T   T  Apost ,l  A  A   2,l I   w A  A  A  A   A  A  T  T   z Apost ,l  A  A   2,l I   w A  A  A  A  

1



r  AT  A  y l 

2,l 2,l w

Gaussian with Scalar covariance matrix

l

l

2,l

2,l

Proj  q A  z l    

v Appost 

z ; z l

post A ,l

, v Apost I 

1 p Trace( Apost ,l ) M 35

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] • Similar to GLM, using EP, the BGLM can be decoupled into two modules

BAd-VAMP d

  A   X  N  Y A

zextA,l  , vAext L

l 1

z  ext B ,l

Component-wise Component wise MMSE ext ext   z Bpost ,l  E  zl | z A,l , vA 

L

l 1

, vBext

ext  vBpost  Var  zl | zext A,l , vA 

Module A

Module B Module B

BAd-GVAMP [MZ18]

• Relation of BAd-GVAMP to BAd-VAMP  An extension of the BAd‐VAMP [SFRS18] from linear measurements to nonlinear measurements.  The BAd‐GVAMP iteratively reduces the original generalized bilinear recovery problem to a sequence of standard bilinear recovery problems f d dbl bl  Note that the message passing schedule within BAd‐VAMP module of BAd‐GVAMP is slightly different from that of the original BAd‐VAMP  In the special case of linear measurements, BAd‐GVAMP reduces to BAd‐VAMP

[MZ18] X. Meng, and J. Zhu, “Bilinear Adaptive Generalized Vector Approximate Message Passing,” arXiv preprint arXiv:1810.08129, 2018

36

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] • Experiment 1: Quantized Compressed Sensing with matrix uncertainty

 c is generated with uniformly random support with K nonzero elements from pp i.i.d N(0,1), we set N = 256, G = 10, K = 10  For M/N = 1, the NMSE in dB is shown i l ft fi in left figure: ‐‐ converges fast (20‐30 iterations) ‐‐ the same as the oracle performance.

37

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] • Experiment 1: Quantized Compressed Sensing with matrix uncertainty

 c is generated with uniformly random support with K nonzero elements from pp i.i.d N(0,1), we set N = 256, G = 10, K = 10  Then, the performance vs. ratio M/N is evaluated: l t d ‐‐ as the increase of M/N, the recovery performance improves ‐‐ approaches the oracle in a wide pp range of M/N values

38

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] • Experiment 2: Self-Calibration from quantized measurements

Aim to recover the K sparse signal vector c and the calibration parameters b Aim: to recover the K‐sparse signal vector c and the calibration parameters b  K = 10, G = 8, M = 128 and SNR = 40 dB.  H is constructed using Q randomly selected columns of the Hadamard matrix, the elements of b and are i.i.d. drawn from N(0; 1), and c is generated with K nonzero elements i.i.d. drawn from N(0; 1).

 As the sampling rate increases, the median NMSE decreases. Also, the reconstruction performance improves as the bit‐depth increases.

39

A Unified Inference Framework for GLM  Bilinear Adaptive Generalized VAMP (BAd-GVAMP) [MZ18] p ( )[ ] • Experiment 3: Structured dictionary learning from quantized measurements

 G=M=N=64 and SNR = 40 dB. G M N 64 and SNR 40 dB

 As the training length L increases, the NMSE decreases. And the structured dictionary can be learned from quantized measurements.

40

Conclusions • • •

Considers the design of efficient GLM inference algorithms Review the AMP algorithm and provides an EP perspective Present a unified approximate inference framework for GLM pp – – –

Facilitates the extension of various SLM inference algorithms to GLM inference in a simple and unified manner Provides some new insights on some well-known algorithms, e.g., GAMP, thus offering a flexible way in the message scheduling of practical implementation a flexible way in the message scheduling of practical implementation Extend further to the bilinear GLM problem and propose the BAd-GVAMP, extending BAd-VAMP to nonlinear measurements

41

Conclusions • • •

Considers the design of efficient GLM inference algorithms Review the AMP algorithm and provides an EP perspective Present a unified approximate inference framework for GLM pp – – –

•

Facilitates the extension of various SLM inference algorithms to GLM inference in a simple and unified manner Provides some new insights on some well-known algorithms, e.g., GAMP, thus offering a flexible way in the message scheduling of practical implementation a flexible way in the message scheduling of practical implementation Extend further to the bilinear GLM problem and propose the BAd-GVAMP, extending BAd-VAMP to nonlinear measurements

Possible future work – – – –

Theoretical analysis of this unified framework Evaluate or analyze the effect of different ways of message scheduling for GLM Design other efficient GLM inference algorithms from SLM ones Design other efficient GLM inference algorithms from SLM ones Extend to multi-layer neural network to see if it helps in the learning and/or inference process in deep neural network.

42

• • • • • • • • • • • •

• • • •

References

[DMM09]Donoho, Maleki, Montanari. "Message‐passing algorithms for compressed sensing." Proceedings of the National Academy of Sciences 106.45 (2009): 18914‐18919. [DMM10] Donoho, Maleki, Montanari, "Message passing algorithms for compressed sensing: I. motivation and construction." Proc IEEE ITW, 20110 [BM11] Bayati, Montanari. "The dynamics of message passing on dense graphs, with applications to compressed sensing." IEEE Transactions on Information Theory 57.2 (2011): 764‐785. [Rangan10] Rangan, Sundeep. "Estimation with random linear mixing, belief propagation and compressed sensing." Information Sciences and Systems (CISS), 2010 44th Annual Conference on. IEEE, 2010. [Tanaka 02] Tanaka, Toshiyuki. "A statistical‐mechanics approach to large‐system analysis of CDMA multiuser detectors." IEEE Transactions on Information theory 48.11 (2002): 2888‐2910. [Minka01] Minka Thomas P "Expectation [Minka01] Minka, Thomas P. Expectation propagation for approximate Bayesian inference. propagation for approximate Bayesian inference " 2001 2001 [OW05] Opper, Manfred, and Ole Winther. "Expectation consistent approximate inference." Journal of Machine Learning Research 6.Dec (2005): 2177‐2204. [Rangan11] Rangan, "Generalized approximate message passing for estimation with random linear mixing." Proc IEEE ISIT 2011 [RSF16] Rangan Schniter Fletcher “Vector [RSF16] Rangan, Schniter, Fletcher, Vector approximate message passing approximate message passing”, 2016 2016 [SRF16] P. Schniter, S. Rangan, and A. K. Fletcher, “Vector approximate message passing for the generalized linear model,” in Proc. 50th Asilomar Conf. Signals, Syst. Comput., Nov. 2016, pp. 1525‐1529. [Kabashima 03] Kabashima Y. A , " CDMA multiuser detection algorithm on the basis of belief propagation” , Journal of Physics A: Mathematical and General, 2003, 36(43) , , ( ) [KMSSZ12] F. Krzakala, M. Mézard, F. Sausset, Y. Sun, and L. Zdeborová, “Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2012, no. 08, p. P08009, 2012. [MWKL15a] X. Meng, S. Wu, L. Kuang, and J. Lu, “An expectation propagation perspective on approximate message passing,” IEEE Signal Process. Lett., vol. 22, no. 8, pp. 1194‐1197, Aug. 2015. [MWKL15b] X. Meng, S. Wu, L. Kuang, and J. Lu, “ Concise derivation of complex Bayesian approximate message passing via expectation propagation,” arXiv preprint arXiv:1509.08658, 2015. [WKNLHDQ14] S. Wu, L. Kuang, Z. Ni, J. Lu, D. Huang, and Q. Guo, “Low‐complexity iterative detection for large‐scale multiuser MIMO OFDM systems using approximate message passing ” IEEE J Sel Topics Signal Process vol 8 no 5 pp 902 915 Oct 2014 MIMO‐OFDM systems using approximate message passing,” IEEE J. Sel. Topics Signal Process., vol. 8, no. 5, pp. 902–915, Oct. 2014. [MWZ18]X. Meng, S. Wu and J. Zhu, “A unified Bayesian inference framework for generalized linear model,” IEEE Signal Process. Lett., vol. 25, no. 3, Mar. 2018. 43

• • • • • • • • • • • • •

References

[MZ18] X. Meng, and J. Zhu, “Bilinear Adaptive Generalized Vector Approximate Message Passing,” arXiv preprint arXiv:1810.08129, 2018 [PSC14a] J. T. Parker, P. Schniter, and V. Cevher, “Bilinear generalized approximate message passing‐Part I: Derivation,” IEEE Trans. Signal Process., vol. 62, no. 22, pp. 5839‐5853, Nov. 2014. [PSC14b] J. T. Parker, P. Schniter, and V. Cevher, “Bilinear generalized approximate message passing‐Part II: Applications,” IEEE Trans. Signal Process., vol. 62, no. 22, pp. 5854‐5867, Nov. 2014. [PS16] J. T. Parker and P. Schniter, “Parametric bilinear generalized approximate message passing,” IEEE J. Sel. Topics Signal Process., vol. 10, no. 4, pp. 795‐808, 2016. [KKMSZ16] Y. Kabashima, F. Krzakala, M. Mézard, A. Sakata and L. Zdeborová, “Phase transitions and sample complexity in bayesoptimal matrix factorization,” IEEE Trans. Inf. Theory, vol. 62, no. 7, pp. 4228‐4265, 2016. [SRF] P Schniter S Rangan and A K Fletcher “Vector [SRF] P. Schniter, S. Rangan, and A. K. Fletcher, Vector approximate message passing for the generalized linear model, approximate message passing for the generalized linear model ” in Proc. 50th in Proc 50th Asilomar Conf. Signals, Syst. Comput., Nov. 2016, pp. 1525‐1529. [ML17] J. Ma and L. Ping, “Orthogonal AMP,” IEEE Access, vol. 5, pp. 2020‐2033, 2017. [HWJ17] H. He, C. K. Wen, and S. Jin, “Generalized expectation consistent signal recovery for nonlinear measurements,” in Proc. IEEE Int. Symp. Inf. Theory, Jun. 2017, pp. 2333‐2337. Int. Symp. Inf. Theory, Jun. 2017, pp. 2333 2337. [QZW18] Zou Q, Zhang H, Wen C K, et al. , “ Concise Derivation for Generalized Approximate Message Passing Using Expectation Propagation ,” IEEE Signal Processing Letters, 2018. [SFRS18] S. Sarkar, A. K. Fletcher, S. Rangan and P. Schniter, “Bilinear recovery using adaptive vector‐AMP,” avaliable at https://arxiv.org/pdf/1809.00024.pdf. p // g/p / p [VS13] J. P. Vila and P. Schniter, “Expectation‐maximization Gaussian‐mixture approximate message passing,” IEEE Trans. Signal Process., vol. 61, no. 19, pp. 4658‐4672, Oct. 2013. [KMZ13] F. Krzakala, M. Mezard, and L. Zdeborova, “Compressed sensing under matrix uncertainty: Optimum thresholds and robust approximate message passing,” ICASSP, 2013.

44

Thanks ！

45