Low Complexity Approximations

Low Complexity Approximations 1 ´ Maciej Skorski

Harvard University, June 14, 2017

1

IST Austria ´ Maciej Skorski

Low Complexity Approximations


1 / 32

1

Introduction Motivating Example Low Complexity Approximation Problem

2

Constructive Approach

3

Non-constructive Approach

4

Conclusion

5

References

´ Maciej Skorski



2 / 32

Introduction

Plan

1


2


3


4

Conclusion

5

References

´ Maciej Skorski



3 / 32

Introduction

Motivating Example

Plan

1


2


3


4

Conclusion

5

References

´ Maciej Skorski



4 / 32

Introduction

Motivating Example

What are regularity lemmas about? Theorem (Regularity Lemmas - informally) The set of vertices of any large graph can be divided so that the parts are of about the same size edges between the parts look random

V2

V1

V3

V5

V4

many applications: extremal graph theory (generalized Turan numbers [ES46; ES66]), number theory (arithmetic progressions in primes [Sze75; GT08]), property testing (monotone graph properties are testable [AS05]) ´ Maciej Skorski



5 / 32

Introduction

Motivating Example

What are regular pairs? Defining regularity Informally: edges running between the vertex subsets look random. Specifically: the fraction of edges between subsets is roughly constant.

X

Y

. For Let E(T, S) be the number of edges between T and S, and d(T, S) = E(T,S) |T |·|S| random graphs d(T, S) ≈ p where p is the edge probability, if T, S are sufficiently large. Definition (-regular pairs) A bipartite graph between X and Y is a -regular if |d(T, S) − d| 6 for some constant d and all T ⊂ X, S ⊂ Y such that |T | > |X|, |S| > |Y |.

´ Maciej Skorski



6 / 32

Introduction

Motivating Example

Statements Theorem (Strong Regularity [LS07]) For any graph (E, V ) there exist an equipartition {Vi }i of V into k = k() parts and numbers di,j such that for all but -fraction of pairs i, j we have |E(T, S) − di,j |Ti ||Sj || 6 |Vi ||Vj | for all T ⊂ Vi , S ⊂ Vj s.t. |S| > |Vi |, |T | > |Vj |, and k is a tower of 2’s of size O −2 .

Theorem (Weak Regularity [FK99]) For any graph (E, V ) there exist an equipartition {Vi }i of V into k = k() parts and numbers di,j such that

X X E(T ∩ Vi , S ∩ Vj ) − di,j |Ti ||Sj | 6 |V |2 i,j i,j −2

for all T, S. Here k equals 2O(

)

and the partition is an overlay of at most O(−2 ) sets.

Note: there are variants of the Strong Regularity Lemma, differnt in powers of [RS10]. ´ Maciej Skorski



7 / 32

Introduction

Motivating Example

More convenient statements Consider the following versions (equipartition could be skipped at little cost) Strong regularity’ = absolute deviation is small strong regularity means that for some constants i, j we have

X 2 (E(Ti,j , Si,j ) − di,j |Ti,j ||Si,j |) = o(|V | ) | {z } 16i,j6k regularity deviation for all Ti,j ⊂ Vi , Si,j ⊂ Vj . Weak regularity = average deviation is small weak regularity means that for some constants di,j

X 16i,j6k

(E(T ∩ Vi , S ∩ Vj ) − di,j |T ∩ Vj ||S ∩ Vi |) = o(|V |2 )

|

{z

regularity deviation

}

for all T, S. ´ Maciej Skorski



8 / 32

Introduction

Motivating Example

Indistinguishability Definition (Indistinguishability) Let g, h be two functions on X , F be a class of functions on X and > 0. We say that g and h are (F, )-indistinguishable if

X X g(x)f (x) − h(x)f (x) 6 x

x

for every f ∈ F .

E(T ∩ Vi , S ∩ Vj ) − di,j |T ∩ Vj ||S ∩ Vi | =

|

{z

regularity deviation

}

X

1E (x)1T ×S (x) − 1Vi 1Vj (x)1T ×S (x)

x∈V 2

|

{z

indistinguishability

}

g is the edge set h is a pair of partition parts F consists of all products of vertex subsets (rectangles) ´ Maciej Skorski



9 / 32

Introduction

Motivating Example

Strong and weak regularity translated Regularity Lemmas in terms of indistinguishability Let X = V × V, g = 1E . Then we have 1

Strong Regularity: g is indistinguishable from some h by functions F, where

( h=

X

di,j 1Vi ×Vj ,

F=

f :f =

i,j

) X

ai,j 1Ti,j ×Si,j

i as having a separating hyperplane f ∈ F between g and h Follow direction f to get closer to the target! Project to be within constraints again - it only gets closer! f constraints C h − γf Proj(P − γD) ”candidate” h

target g

hyperplane associated with f ´ Maciej Skorski



16 / 32

.1

.2

.3

.4

.5

.6


Optimization intuitions Algorithm 2: Low Complexity Approximations Input : target function g, class of tests F, starting point h0 , accuracy , step size t Output: function h of low complexity w.r.t F and indistinguishable from g w.r.t. F n←0 while δ F (hn , g) > do n←n+1 f ← ∂h δ F (h, g)h=hn // compute the subgradient hn ← hn−1 − t · f // go towards the negative subgradient hn ← EuclideanProjectionC (hn ) // update to meet constraints By the subgradient calculus rules, we have

! ∂h

max

X

f

(g(x)f (x) − h(x)f (x))

= f 0 for some f 0 ∈ F

x

distinguisher = subgradient of the computation distance! ´ Maciej Skorski



17 / 32


Applications

Applications include: better bounds for simulating auxiliary inputs (TCC’13,TCC’16B) and better bounds for the EUROCRYPT’09 resilient stream cipher a unified proof for Impagliazzo Hardcore Lemma, Dense Model Theorem, Weak Szemeredi Theorem (CCC’09) showing that ”dense” leakages can be efficiently simulated, with significantly improved bounds a unified proof for both weak and strong regularity lemmas ´ More details can be found in [Sko16a]

´ Maciej Skorski



18 / 32


Plan

1


2


3


4

Conclusion

5

References

´ Maciej Skorski



19 / 32


Playing two-player zero-sum games

Consider a game where players A, B choose their strategies a ∈ A and b ∈ B and B pays v(a, b) to A. The player A seeks to maximize her gain, B minimizes his loss. if pure strategies only min max v(a, b) > max min v(a, b) b∈B a∈A

a∈A b∈B

(advantage of playing second) if mixed (randomized) strategies allowed min max Eb∼pb v(a, b) =

pb ∈P(B) a∈A

max min Ea∼pa v(a, b)

pa ∈P(A) b∈B

the order of the players doesn’t matter; this is the (Nash) equilibrium Mixed strategies can have huge supports - that’s bad for applications.

´ Maciej Skorski



20 / 32


Approximating mixed strategies

Given: strategies A, B for players A and B, a payoff v : A × B → [−1, 1], a mixed strategy p for A (distribution over A) Find: a distribution q over A such that q approximates p in the payoff for every other player’s choice: Ea∼q v(a, b) > Ea∼p v(a, b) − holds for every b ∈ B and some small number q is of small size, e.g. has possibly small support

If q is supported on n points, we say that q is an (n, )-approximation to p.

´ Maciej Skorski



21 / 32


Approximation by subsampling + union bounds

Draw samples S = {a1 , . . . , an } from pa , and let q be uniform over S By the Chernoff and union bound, q is an (n, )-approximation to p with

r =O

log(1/δ) log |B| n

with probability 1 − δ. Note: for one non-constructive approximation, δ =

1 2

!

suffices.

Need for improvements, log |B| may be still too large (e.g. distributions over {0, 1}n )

´ Maciej Skorski



22 / 32


Approximation by subsampling + Rademacher analysis

Theorem (S.) Define the correlation coefficient (Rademacher complexity) γ(n) = Ea1 ,...,an ∼pa Eσi max b∈B

1X σi v(ai , b) n i

where σi are independent Rademacher random variables (uniform over {−1, 1}). Then the previous result improves to

r = γ(n) + O

log(1/δ) n

!

The Rademacher complexity is well-studied and can provide improvements over the union bound, when more is known about B and v.

´ Maciej Skorski



23 / 32


Intuitions - connections to statistical learning Suppose we want to learn some (unknown) function g, based on a sample S from a (huge) dataset X . The learned predictor g comes from some (simple) class F. We want, with high probability over S, that ∀f ∈ F :

Ex∼X [errorf,g (x)] 6 Ex∼S [errorf,g (x)] + o(1),

where errorf,g (x) is a misclassification score (penalty for f (x) 6= g(x)). Results of this sort are called generalization bounds. Mapping the min-max problem to the statistical learning framework player A as the input x player B as the predictor f the payoff as the score function approximating strategy is the empirical distribution of S

´ Maciej Skorski



24 / 32


Applications

From this framework one can derive many results with optimal bounds Impagliazzo Hardcore Lemma Dense Model Theorem transformations between pseudoentropies convex approximation rates for Hilbert spaces Nash equilibria for games with low-rank payoff matrices ... More details can be found in the paper [Sko17].

´ Maciej Skorski



25 / 32

Conclusion

Plan

1


2


3


4

Conclusion

5

References

´ Maciej Skorski



26 / 32

Conclusion

Conclusion

a new framework for low complexity approximations (constructive), utilizing the fact that distinguishers can be seen as subgradients to the computational distance a new framework for low complexity approximations (non-constructive), reducing the problem to statistical learning future work is to find more applications (e.g. to utilize the variance factor in the constructive result)

´ Maciej Skorski



27 / 32

References

Plan

1


2


3


4

Conclusion

5

References

´ Maciej Skorski



28 / 32

References

References I

N. Alon and A. Shapira. “Every Monotone Graph Property is Testable”. In: Proceedings of the Thirty-seventh Annual ACM Symposium on Theory of Com STOC ’05. Baltimore, MD, USA: ACM, 2005. URL: http://doi.acm.org/10.1145/1060590.1060611.

B. Barak, M. Hardt, and S. Kale. “The uniform hardcore lemma via approximate Bregman projections”. In: Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algo 2009. URL: http://dl.acm.org/citation.cfm?id=1496770.1496899. ¨ and A. H. Stone. “On the structure of linear graphs”. In: P. Erdos Bull. Amer. Math. Soc. 52.12 (Dec. 1946). URL: http://projecteuclid.org/euclid.bams/1183510246. ¨ and M. Simonovits. “A limit theorem in graph theory”. In: P. Erdos Studia Sci. Math. Hungar 1.51-57 (1966). A. M. Frieze and R. Kannan. “Quick Approximation to Matrices and Applications”. In: Combinatorica 19 (1999). ´ Maciej Skorski



29 / 32

References

References II B. Green and T. Tao. “The primes contain arbitrarily long arithmetic progressions”. In: Ann. of Math 167 (2008). D. Jetchev and K. Pietrzak. “How to Fake Auxiliary Input”. In: Theory of Cryptography TCC 2014. Ed. by Y. Lindell. Vol. 8349. Lecture Notes in Computer Science. Springer, 2014. URL: http://dx.doi.org/10.1007/978-3-642-54242-8_24. ´ and B. Szegedy. “Szemeredi’s ´ L. Lovasz Lemma for the Analyst”. In: GAFA Geometric And Functional Analysis 17.1 (2007). URL: http://dx.doi.org/10.1007/s00039-007-0599-6. ¨ and M. Schacht. “Regularity Lemmas for Graphs”. In: V. Rodl Fete of Combinatorics and Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010. URL: http://dx.doi.org/10.1007/978-3-642-13580-4_11. M. Skorski. Approximating Min-Max Strategies by Statistical Learning. 2017. URL: https://www.researchgate.net/publication/ 316342484_Approximating_MinMax_Strategies_by_Statistical_Learning. ´ Maciej Skorski



30 / 32

References

References III ´ M. Skorski. “A Subgradient Algorithm For Computational Distances and Applications to Cryptography”. In: IACR Cryptology ePrint Archive 2016 (2016). URL: http://eprint.iacr.org/2016/158. ´ M. Skorski. “Simulating Auxiliary Inputs, Revisited”. In: Theory of Cryptography TCC 2016-B. 2016. URL: https://doi.org/10.1007/978-3-662-53641-4_7. E. Szemeredi. On sets of integers containing no k elements in arithmetic progression. 1975. L. Trevisan, M. Tulsiani, and S. Vadhan. “Regularity, Boosting, and Efficiently Simulating Every High-Entropy Distribution”. In: 2009 24th Annual IEEE Conference on Computational Complexity. 2009. S. Vadhan and C. J. Zheng. “A Uniform Min-Max Theorem with Applications in Cryptography”. In: Advances in Cryptology – CRYPTO 2013. Ed. by R. Canetti and J. A. Garay. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013. URL: http://dx.doi.org/10.1007/978-3-642-40041-4_6.

´ Maciej Skorski



31 / 32

References

Thank you for your attention!

Questions?

´ Maciej Skorski



32 / 32