Coding Schemes for Distributed Storage Systems - Google Sites

Coding Schemes for Distributed Storage Systems Min Ye Advisor: Alexander Barg ECE/ISR, University of Maryland

August 11, 2017

Motivation: Distributed Storage Systems (DSS)

• DSS spread data across thousands of storage nodes

Min Ye, Ph.D. Dissertation Defense

Coding Schemes for Distributed Storage Systems

August 11, 2017

2/1


• DSS spread data across thousands of storage nodes • Individual storage nodes fail frequently



August 11, 2017

2/1


• DSS spread data across thousands of storage nodes • Individual storage nodes fail frequently • Need redundancy to protect data



August 11, 2017

2/1

Replication: large storage overhead

Can tolerate any 2 node failures Storage overhead = 3×



August 11, 2017

3/1

Erasure-Correcting Codes: optimal storage efficiency

Add 2 parity nodes to every 3 data nodes Form an (n = 5, k = 3) code



August 11, 2017

4/1

Erasure-Correcting codes: optimal storage efficiency

Can tolerate any 2 node failures Storage overhead = 1.67×



August 11, 2017

5/1

Disk I/O and network flow of the repair process Repair a failed node



August 11, 2017

6/1


Access more nodes ⇒ greater number of disk I/O operations



August 11, 2017

6/1


Access more nodes ⇒ greater number of disk I/O operations Download more data from functioning nodes ⇒ larger network flow Min Ye, Ph.D. Dissertation Defense


August 11, 2017

6/1

Compare Erasure Codes to Replication

Storage overhead Number of disk I/O (access) Network flow (bandwidth)


Erasure Codes Small Large Large


Replication Large Small Small

August 11, 2017

7/1





We prefer Erasure Codes due to their small storage overhead



August 11, 2017

7/1





We prefer Erasure Codes due to their small storage overhead

Want to improve the access and bandwidth of Erasure Codes



August 11, 2017

7/1

Important metrics and operations Metric

Operation

locality/ access

repair

bandwidth/ communication

error correction



August 11, 2017

8/1

Important metrics and operations Operation

Metric 1

locality/ access bandwidth/ communication

repair

error correction

1 Locally Recoverable codes (local recovery)



August 11, 2017

8/1


Metric 1

locality/ access

repair 2


error correction

1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery)



August 11, 2017

8/1


Metric 1

locality/ access

repair

3 2


error correction

1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery) 3 LDPC codes (global recovery)



August 11, 2017

8/1


Metric 1

locality/ access

repair

3 2


4

error correction

1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery) 3 LDPC codes (global recovery) 4 Fractional decoding (global recovery) Min Ye, Ph.D. Dissertation Defense


August 11, 2017

8/1

MDS codes

• (n, k, `) MDS array code (vector codes) over field F:



August 11, 2017

9/1

MDS codes

• (n, k, `) MDS array code (vector codes) over field F: • code length n



August 11, 2017

9/1

MDS codes

• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes



August 11, 2017

9/1

MDS codes

• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes



August 11, 2017

9/1

MDS codes

• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F

Each Ci is a vector in F `



August 11, 2017

9/1

MDS codes


Each Ci is a vector in F ` • contents of any r nodes can be determined by the other k nodes



August 11, 2017

9/1

MDS codes



• MDS scalar codes: each coordinate is a scalar in F



August 11, 2017

9/1

MDS codes



• MDS scalar codes: each coordinate is a scalar in F • View scalar codes as vector codes over some subfield of F



August 11, 2017

9/1

MDS codes



• MDS scalar codes: each coordinate is a scalar in F • View scalar codes as vector codes over some subfield of F • ` is the degree of the field extension



August 11, 2017

9/1

Main focus and background

• Main focus: The bandwidth of MDS codes for the repair task and error correction

task.



August 11, 2017

10 / 1



task. • Repair bandwidth: the minimum amount of data we need to download in order to

recover failed nodes



August 11, 2017

10 / 1



task. • Repair bandwidth: the minimum amount of data we need to download in order to

recover failed nodes • MDS codes with smallest possible repair bandwidth are called Minimum Storage

Regenerating (MSR) Codes



August 11, 2017

10 / 1

Main contributions

• We give explicit constructions of MSR codes



August 11, 2017

11 / 1

Main contributions

• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular

parameters.



August 11, 2017

11 / 1

Main contributions


parameters. • Explicit construction of MSR codes for general parameters was an open problem.



August 11, 2017

11 / 1

Main contributions


parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters.



August 11, 2017

11 / 1

Main contributions


parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters. • The first explicit constructions for general parameters in the literature



August 11, 2017

11 / 1

Main contributions



• We introduce the “Fractional decoding” problem: study the optimal bandwidth of

MDS codes for error correction task.



August 11, 2017

11 / 1

Main contributions



• We introduce the “Fractional decoding” problem: study the optimal bandwidth of

MDS codes for error correction task. • We give lower bound and explicit constructions that achieve the lower bound



August 11, 2017

11 / 1

Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F



August 11, 2017

12 / 1

Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node



August 11, 2017

12 / 1

Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β



August 11, 2017

12 / 1

Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥


d` d+1−k

(Dimakis et al., 2010)


August 11, 2017

12 / 1


d` d+1−k


• The right-hand side is a decreasing function of d, thus β → min if d = n − 1



August 11, 2017

12 / 1


d` d+1−k



• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes



August 11, 2017

12 / 1


d` d+1−k



• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes β≥


dh` d+h−k

(Cadambe et al., 2013)


August 11, 2017

12 / 1


d` d+1−k



• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes β≥

dh` d+h−k

(Cadambe et al., 2013)

• (h, d)-optimal repair property: if the lower bound can be achieved for the recovery

of any h failed nodes from any d helper nodes



August 11, 2017

12 / 1

(1, d)-optimal repair property • Most common scenario: single node failure



August 11, 2017

13 / 1

(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR

codes)



August 11, 2017

13 / 1


codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11)



August 11, 2017

13 / 1


codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.)



August 11, 2017

13 / 1


codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.) • Various papers relaxed the condition of (1, d)-optimal repair property – only

require optimal repair of systematic nodes



August 11, 2017

13 / 1



require optimal repair of systematic nodes • Even under this relaxed requirement, no explicit construction are known for rate larger

than 1/2 and r > 3



August 11, 2017

13 / 1



require optimal repair of systematic nodes • Even under this relaxed requirement, no explicit construction are known for rate larger

than 1/2 and r > 3 • High-rate constructions are the most important in practice: Facebook employs

MDS code with rate 0.71 and 4 parity nodes



August 11, 2017

13 / 1

Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes



August 11, 2017

14 / 1

Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal

repair property (the first explicit high-rate construction for r > 3 in the literature)



August 11, 2017

14 / 1


repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair

property, k ≤ d ≤ n − 1



August 11, 2017

14 / 1



property, k ≤ d ≤ n − 1 • Further extension: Given any n and r, we present explicit (n, k = n − r, `) MDS

array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously.



August 11, 2017

14 / 1




array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes



August 11, 2017

14 / 1




array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes • The only explicit construction for h > 1 in the literature



August 11, 2017

14 / 1




array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes • The only explicit construction for h > 1 in the literature • Can optimally recover any number of erasures from any set of helper nodes. Min Ye, Ph.D. Dissertation Defense


August 11, 2017

14 / 1

Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node

failure from all the other surviving nodes



August 11, 2017

15 / 1


failure from all the other surviving nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T


c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1


August 11, 2017

15 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1

• Each row forms a Generalized Reed-Solomon code with different evaluation points



August 11, 2017

15 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1


• Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ r − 1: rn distinct elements in F



August 11, 2017

15 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1


• Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ r − 1: rn distinct elements in F • F: any finite field with size |F| ≥ rn



August 11, 2017

15 / 1

Construction 1: (1, n − 1)-optimal repair MDS codes

• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13)



August 11, 2017

16 / 1


• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an )



August 11, 2017

16 / 1


• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )

λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,



t = 0, 1, . . . , r − 1

August 11, 2017

16 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code



August 11, 2017

16 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures



August 11, 2017

16 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures • (1, n − 1)-optimal repair: can recover a column by downloading `/r elements of F

from each of the other columns



August 11, 2017

16 / 1

(1, n − 1)-optimal repair property

• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an )



August 11, 2017

17 / 1


• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •




t = 0, 1, . . . , r − 1

August 11, 2017

17 / 1


• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •


t = 0, 1, . . . , r − 1

•

λti,ai ci,a +

X

λtj,aj cj,a = 0

j6=i



August 11, 2017

17 / 1


• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •


t = 0, 1, . . . , r − 1

•

λti,ai ci,a +

X

λtj,aj cj,a = 0

j6=i

•

λti,u ci,a(i,u) +

X

λtj,aj cj,a(i,u) = 0,

u = 0, 1, . . . , r − 1

j6=i



August 11, 2017

17 / 1


• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •


t = 0, 1, . . . , r − 1

•

λti,ai ci,a +

X

λtj,aj cj,a = 0

j6=i

•

λti,u ci,a(i,u) +

X

λtj,aj cj,a(i,u) = 0,

u = 0, 1, . . . , r − 1

j6=i

• r−1 X u=0


λti,u ci,a(i,u) +

r−1 X X

λtj,aj cj,a(i,u) = 0

u=0 j6=i


August 11, 2017

17 / 1


r−1 X

λti,u ci,a(i,u) +

r−1 XX ( λtj,aj cj,a(i,u) ) = 0, j6=i

u=0

     |  +

X j6=i

    |


t = 0, 1, . . . , r − 1

u=0

1 λi,0 .. . λr−1 i,0

... ... .. . ...

1 λi,1 .. . λr−1 i,1

1



λi,r−1 .. . λr−1 i,r−1

   

{z

Vandermonde, rank = r

1 λj,aj .. . λr−1 j,aj

1 λj,aj .. . λr−1 j,aj {z

... ... .. . ...

aligned, rank = 1


}|

ci,a(i,0) ci,a(i,1) .. .

    

ci,a(i,r−1) {z }

desired information

    

cj,a(i,0) cj,a(i,1) .. .

    

=0

cj,a(i,r−1) }


August 11, 2017

18 / 1


r−1 X

λti,u ci,a(i,u) +

j6=i

u=0

     r−1 X

r−1 XX ( λtj,aj cj,a(i,u) ) = 0,

cj,a(i,u)

|

u=0

 +

X j6=i

    |


t = 0, 1, . . . , r − 1

u=0

1 λi,0 .. . λr−1 i,0

... ... .. . ...

1 λi,1 .. . λr−1 i,1

1



λi,r−1 .. . λr−1 i,r−1

   

{z

Vandermonde, rank = r


1 λj,aj .. . λr−1 j,aj {z

... ... .. . ...

aligned, rank = 1


}|

ci,a(i,0) ci,a(i,1) .. .

    

ci,a(i,r−1) {z }

desired information

    

cj,a(i,0) cj,a(i,1) .. .

    

=0

cj,a(i,r−1) }


August 11, 2017

19 / 1


• ci,a(i,0) , ci,a(i,1) , . . . , ci,a(i,r−1) can be determined by {


Pr−1


u=0

cj,a(i,u) }j6=i

August 11, 2017

20 / 1



Pr−1 u=0

cj,a(i,u) }j6=i

• ` elements in Ci : partition into `/r groups of size r



August 11, 2017

20 / 1



Pr−1 u=0

cj,a(i,u) }j6=i

• ` elements in Ci : partition into `/r groups of size r • Each group can be determined by downloading a scalar in F from each of the

other nodes



August 11, 2017

20 / 1



Pr−1 u=0

cj,a(i,u) }j6=i

• ` elements in Ci : partition into `/r groups of size r • Each group can be determined by downloading a scalar in F from each of the

other nodes • In total: only need to download `/r elements of F from each of the other nodes



August 11, 2017

20 / 1

Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2



August 11, 2017

21 / 1

Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure

from any d helper nodes



August 11, 2017

21 / 1


from any d helper nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T


c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1


August 11, 2017

21 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1




August 11, 2017

21 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1

• Each row forms a Generalized Reed-Solomon code with different evaluation points • Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ s − 1: sn distinct elements in F



August 11, 2017

21 / 1



c1,0 c1,1 .. .

c2,0 c2,1 .. .

c1,`−1

c2,`−1

... ... .. . ...

cn,0 cn,1 .. . cn,`−1

• Each row forms a Generalized Reed-Solomon code with different evaluation points • Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ s − 1: sn distinct elements in F • F: any finite field with size |F| ≥ sn



August 11, 2017

21 / 1

Construction 2: (1, d)-optimal repair MDS codes

• s = d + 1 − k, ` = sn



August 11, 2017

22 / 1


• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an )



August 11, 2017

22 / 1


• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )




t = 0, 1, . . . , r − 1

August 11, 2017

22 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code



August 11, 2017

22 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures



August 11, 2017

22 / 1




t = 0, 1, . . . , r − 1

• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures • (1, d)-optimal repair property: • given any set of d helper nodes • can recover a column by downloading `/s elements of F from each of the d helper nodes



August 11, 2017

22 / 1

(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an )



August 11, 2017

23 / 1

(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •

λt1,a1 c1,a +

n X

λtj,aj cj,a = 0,

t = 0, 1, . . . , r − 1

j=2



August 11, 2017

23 / 1


λt1,a1 c1,a +

n X

λtj,aj cj,a = 0,

t = 0, 1, . . . , r − 1

j=2

•

λt1,u c1,a(1,u) +

n X

λtj,aj cj,a(1,u) = 0,

u = 0, 1, . . . , s − 1

j=2



August 11, 2017

23 / 1


λt1,a1 c1,a +

n X

λtj,aj cj,a = 0,

t = 0, 1, . . . , r − 1

j=2

•

λt1,u c1,a(1,u) +

n X


u = 0, 1, . . . , s − 1

j=2

• s−1 X u=0


λt1,u c1,a(1,u) +

s−1 X n X

λtj,aj cj,a(1,u) = 0

u=0 j=2


August 11, 2017

23 / 1


λt1,a1 c1,a +

n X

λtj,aj cj,a = 0,

t = 0, 1, . . . , r − 1

j=2

•

λt1,u c1,a(1,u) +

n X


u = 0, 1, . . . , s − 1

j=2

• s−1 X

λt1,u c1,a(1,u) +

u=0

s−1 X n X

λtj,aj cj,a(1,u) = 0

u=0 j=2

• s−1 X

λt1,u c1,a(1,u) +

u=0

n X j=2

s−1 X λtj,aj ( cj,a(1,u) ) = 0,

t = 0, 1, . . . , r − 1

u=0

|

{z

}

,µj,a



August 11, 2017

23 / 1

(1, d)-optimal repair property

s−1 X

λt1,u c1,a(1,u) +

u=0


n X

λtj,aj µj,a = 0,

t = 0, 1, . . . , r − 1

j=2


August 11, 2017

24 / 1


s−1 X

λt1,u c1,a(1,u) +

u=0

n X

λtj,aj µj,a = 0,

t = 0, 1, . . . , r − 1

j=2

    

1 λ1,0 .. . λr−1 1,0

|

1



λ1,s−1 .. . λr−1 1,s−1

   

{z

1

1

 λ2,a2  + .  .. λr−1 2,a2

λ3,a3 .. . λr−1 3,a3

... ... .. . ...

c1,a(1,0) c1,a(1,1) .. .

    

c1,a(1,s−1) }

r≥s, full column rank




... ... .. . ...

1 λ1,1 .. . λr−1 1,1

1



λn,an .. . λr−1 n,an

   

µ2,a µ3,a .. . µn,a


   =0 

August 11, 2017

24 / 1


s−1 X

λt1,u c1,a(1,u) +

u=0

n X

λtj,aj µj,a = 0,

t = 0, 1, . . . , r − 1

j=2

    

1 λ1,0 .. . λr−1 1,0

|

... ... .. . ...

1 λ1,1 .. . λr−1 1,1

1



λ1,s−1 .. . λr−1 1,s−1

   

{z

1

1

 λ2,a2  + .  .. λr−1 2,a2

λ3,a3 .. . λr−1 3,a3

... ... .. . ...

    

c1,a(1,s−1) }

r≥s, full column rank



c1,a(1,0) c1,a(1,1) .. .

1




   

µ2,a µ3,a .. . µn,a

   =0 

Want to show that (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code with dimension d.



August 11, 2017

24 / 1

(1, d)-optimal repair property • r−s=n−1−d



August 11, 2017

25 / 1

(1, d)-optimal repair property • r−s=n−1−d • Want to find an (r − s) × r matrix P such that



August 11, 2017

25 / 1

(1, d)-optimal repair property • r−s=n−1−d • Want to find an (r − s) × r matrix P such that •

   P 

1 λ1,0 .. . λr−1 1,0

|

1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z

1



λ1,s−1 .. . λr−1 1,s−1

   

1

 λ2,a2  +P .  .. λr−1 2,a2 |

1 λ3,a3 .. . λr−1 3,a3 {z

... ... .. . ...

1 λn,an .. . λr−1 n,an

parity check matrix of an MDS code


    

c1,a(1,s−1) }

=0



c1,a(1,0) c1,a(1,1) .. .



µ2,a   µ3,a   .   .. µn,a }


   =0 

August 11, 2017

25 / 1


• Each row of P: coefficients of a polynomial of degree less than r



August 11, 2017

26 / 1


• Each row of P: coefficients of a polynomial of degree less than r •

p0 (x) =

s−1 Y (x − λ1,u ),

degree s

u=0



August 11, 2017

26 / 1



p0 (x) =

s−1 Y (x − λ1,u ),

degree s

u=0

p1 (x) = xp0 (x),


degree s + 1


August 11, 2017

26 / 1



p0 (x) =

s−1 Y (x − λ1,u ),

degree s

u=0


p1 (x) = xp0 (x),

degree s + 1

p2 (x) = x2 p0 (x),

degree s + 2


August 11, 2017

26 / 1



p0 (x) =

s−1 Y (x − λ1,u ),

degree s

u=0

p1 (x) = xp0 (x),

degree s + 1

p2 (x) = x2 p0 (x),

degree s + 2 .. .

pr−s−1 (x) = xr−s−1 p0 (x),


degree r − 1


August 11, 2017

26 / 1



p0 (x) =

s−1 Y (x − λ1,u ),

degree s

u=0

p1 (x) = xp0 (x),

degree s + 1

p2 (x) = x2 p0 (x),

degree s + 2 .. .

pr−s−1 (x) = xr−s−1 p0 (x),

degree r − 1

• Write

pi (x) = pi,0 + pi,1 x + pi,2 x2 + · · · + pi,r−1 xr−1



August 11, 2017

26 / 1


•

   P= 


p0,0 p1,0 .. .

p0,1 p1,1 .. .

p0,2 p1,2 .. .

pr−s−1,0

pr−s−1,1

pr−s−1,2

... ... .. . ...


p0,r−1 p1,r−1 .. .

    

pr−s−1,r−1

August 11, 2017

27 / 1


•

   P= 

p0,0 p1,0 .. .

p0,1 p1,1 .. .

p0,2 p1,2 .. .

pr−s−1,0

pr−s−1,1

pr−s−1,2

... ... .. . ...



p0,r−1 p1,r−1 .. .

   

pr−s−1,r−1

•

    P  

1 x x2 .. .



      =    

xr−1




p0 (x) p1 (x) p2 (x) .. . pr−s−1 (x)





      =    

p0 (x) xp0 (x) x2 p0 (x) .. . xr−s−1 p0 (x)


      

August 11, 2017

27 / 1

(1, d)-optimal repair property •

p0 (x) =

s−1 Y (x − λ1,u ) u=0

•

   P     = 

1 λ1,0 .. . λr−1 1,0

1 λ1,1 .. . λr−1 1,1

p0 (λ1,0 ) λ1,0 p0 (λ1,0 ) .. . λr−s−1 p0 (λ1,0 ) 1,0

... ... .. . ...

1



λ1,s−1 .. . λr−1 1,s−1

   

p0 (λ1,1 ) λ1,1 p0 (λ1,1 ) .. . r−s−1 λ1,1 p0 (λ1,1 )

... ... .. . ...

p0 (λ1,s−1 ) λ1,s−1 p0 (λ1,s−1 ) .. . r−s−1 λ1,s−1 p0 (λ1,s−1 )

    

=0



August 11, 2017

28 / 1


p0 (x) =

s−1 Y (x − λ1,u ) u=0

•



1

1

 λ2,a2 λ3,a3  P . ..  .. . λr−1 λr−1 2,a2 3,a3  p0 (λ2,a2 )  λ2,a2 p0 (λ2,a2 )  = ..  . λr−s−1 p0 (λ2,a2 ) 2,a2 |

... ... .. . ...

1




   

p0 (λ3,a3 ) λ3,a3 p0 (λ3,a3 ) .. . r−s−1 λ3,a p0 (λ3,a3 ) 3 {z

... ... .. . ...

p0 (λn,an ) λn,an p0 (λn,an ) .. . r−s−1 λn,a p0 (λn,an ) n

the parity matrix of a GRS code with length n − 1 and dimension d



     }

August 11, 2017

29 / 1


   P 

1 λ1,0 .. . λr−1 1,0

|

1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z

1



λ1,s−1 .. . λr−1 1,s−1

   

1

 λ2,a2  +P .  .. λr−1 2,a2 |

1 λ3,a3 .. . λr−1 3,a3 {z

... ... .. . ...

   

c1,a(1,s−1)

1




   

parity check matrix of a GRS code




}

=0



c1,a(1,0) c1,a(1,1) .. .

µ2,a µ3,a .. . µn,a

   =0 

}


August 11, 2017

30 / 1


   P 

1 λ1,0 .. . λr−1 1,0

|

1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z

1



λ1,s−1 .. . λr−1 1,s−1

   

1

 λ2,a2  +P .  .. λr−1 2,a2 |

1 λ3,a3 .. . λr−1 3,a3 {z

... ... .. . ...

    

c1,a(1,s−1) }

=0



c1,a(1,0) c1,a(1,1) .. .

1




   


µ2,a µ3,a .. . µn,a

   =0 

}

• (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code: can correct errors



August 11, 2017

30 / 1


   P 

1 λ1,0 .. . λr−1 1,0

|

1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z

1



λ1,s−1 .. . λr−1 1,s−1

   

1

 λ2,a2  +P .  .. λr−1 2,a2 |

1 λ3,a3 .. . λr−1 3,a3 {z

... ... .. . ...

    

c1,a(1,s−1) }

=0



c1,a(1,0) c1,a(1,1) .. .

1




   


µ2,a µ3,a .. . µn,a

   =0 

}

• (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code: can correct errors • Optimal error resilience capability



August 11, 2017

30 / 1

Construction 3: Optimally recover any erasure pattern

• Set s = lcm(1, 2, . . . , n − k) in Construction 2



August 11, 2017

31 / 1


• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all

1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously



August 11, 2017

31 / 1



1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes



August 11, 2017

31 / 1



1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn



August 11, 2017

31 / 1



1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn • Low-complexity encoding, update and repair procedures



August 11, 2017

31 / 1



1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn • Low-complexity encoding, update and repair procedures • Optimal error resilience in the repair process



August 11, 2017

31 / 1

Systematic generator VS. non-systematic parity check

• k × n systematic generator matrix

    


I 0 .. . 0

0 I .. . 0

... ... .. . ...

0 0 .. . I

A1,1 A1,2 .. . A1,k

A2,1 A2,2 .. . A2,k

... ... .. . ...


Ar,1 Ar,2 .. . Ar,k

    

August 11, 2017

32 / 1



    

I 0 .. . 0

0 I .. . 0

... ... .. . ...

0 0 .. . I

A1,1 A1,2 .. . A1,k

A2,1 A2,2 .. . A2,k

... ... .. . ...

Ar,1 Ar,2 .. . Ar,k

    

• Every square submatrix needs to be invertible. Optimally repair systematic nodes.



August 11, 2017

32 / 1



    

I 0 .. . 0

0 I .. . 0

... ... .. . ...

0 0 .. . I

A1,1 A1,2 .. . A1,k

A2,1 A2,2 .. . A2,k

... ... .. . ...

Ar,1 Ar,2 .. . Ar,k

    

• Every square submatrix needs to be invertible. Optimally repair systematic nodes. • r × n parity check matrix

    


A1,1 A2,1 .. . Ar,1

A1,2 A2,2 .. . Ar,2

A1,3 A2,3 .. . Ar,3

... ... .. . ...

A1,n A2,n .. . Ar,n


    

August 11, 2017

32 / 1



    

I 0 .. . 0

0 I .. . 0

... ... .. . ...

0 0 .. . I

A1,1 A1,2 .. . A1,k

A2,1 A2,2 .. . A2,k

... ... .. . ...

Ar,1 Ar,2 .. . Ar,k

    

• Every square submatrix needs to be invertible. Optimally repair systematic nodes. • r × n parity check matrix

    

A1,1 A2,1 .. . Ar,1

A1,2 A2,2 .. . Ar,2

A1,3 A2,3 .. . Ar,3

... ... .. . ...

A1,n A2,n .. . Ar,n

    

• Only require every r × r submatrix to be invertible. Optimally repair all nodes.



August 11, 2017

32 / 1

Block Vandermonde structure

•

      


I A1 A21 .. . Ar−1 1

I A2 A22 .. . Ar−1 2

I A3 A23 .. . Ar−1 3

... ... ... .. . ...

I An A2n .. . Ar−1 n


      

August 11, 2017

33 / 1


•

      

I A1 A21 .. . Ar−1 1

I A2 A22 .. . Ar−1 2

I A3 A23 .. . Ar−1 3

... ... ... .. . ...


      

• Commuting: Ai Aj = Aj Ai



August 11, 2017

33 / 1


•

      

I A1 A21 .. . Ar−1 1

I A2 A22 .. . Ar−1 2

I A3 A23 .. . Ar−1 3

... ... ... .. . ...


      

• Commuting: Ai Aj = Aj Ai • Ai − Aj invertible



August 11, 2017

33 / 1


•

      

I A1 A21 .. . Ar−1 1

I A2 A22 .. . Ar−1 2

I A3 A23 .. . Ar−1 3

... ... ... .. . ...


      

• Commuting: Ai Aj = Aj Ai • Ai − Aj invertible • Construction 1-3: diagonal matrices naturally commute



August 11, 2017

33 / 1

Permutation matrices and optimal access

• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13)



August 11, 2017

34 / 1


• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O



August 11, 2017

34 / 1


• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the

(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously



August 11, 2017

34 / 1



(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1



August 11, 2017

34 / 1



(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process



August 11, 2017

34 / 1



(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process • Encoding and update complexity is higher



August 11, 2017

34 / 1



(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process • Encoding and update complexity is higher • M. Ye and A. Barg, “Explicit constructions of high-rate MDS array codes with

optimal repair bandwidth,” IEEE Transactions on Information Theory, vol. 63, no. 4, pp. 2001–2014, April 2017.



August 11, 2017

34 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures • M. Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with

nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017 (arXiv:1605.08630).



August 11, 2017

35 / 1

Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures • M. Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with

nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017 (arXiv:1605.08630). • Sasidharan et al., 2016, Li et al., ISIT 2017: Same parameters, very similar

constructions, appeared after our paper was posted on arXiv



August 11, 2017

35 / 1

Evaluation in Amazon Cluster

M. Ye et al., “Pairing up for regeneration: The Mantra for fast and efficient node repair in distributed storage,” poster presented at USENIX Annual Technical Conference, July 12-14, 2017, Santa Clara, CA. Min Ye, Ph.D. Dissertation Defense


August 11, 2017

36 / 1

Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017)



August 11, 2017

37 / 1

Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework



August 11, 2017

37 / 1

Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field



August 11, 2017

37 / 1

Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the

cut-set bound



August 11, 2017

37 / 1


cut-set bound

• Dau-Milenkovic (ISIT-2017)



August 11, 2017

37 / 1


cut-set bound

• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters



August 11, 2017

37 / 1


cut-set bound

• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters • Repair bandwidth is still far away from the cut-set bound



August 11, 2017

37 / 1


cut-set bound

• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters • Repair bandwidth is still far away from the cut-set bound

• Open problem: Can RS codes (or any scalar MDS codes) achieve the cut-set

bound?



August 11, 2017

37 / 1

Our results • Explicit construction of RS codes achieving the cut set bound



August 11, 2017

38 / 1

Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair

property



August 11, 2017

38 / 1


property • ` = e(1+o(1))n log n , super-exponential



August 11, 2017

38 / 1


property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough



August 11, 2017

38 / 1


property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?



August 11, 2017

38 / 1



• An almost matching lower bound on ` of scalar linear MSR codes



August 11, 2017

38 / 1



• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k



August 11, 2017

38 / 1




• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017.



August 11, 2017

38 / 1




• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically



August 11, 2017

38 / 1




• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r



August 11, 2017

38 / 1




• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r • Cut-set bound is (n − 1)`/r, ratio goes to 1, asymptotically optimal



August 11, 2017

38 / 1




• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r • Cut-set bound is (n − 1)`/r, ratio goes to 1, asymptotically optimal • M. Ye and A. Barg, “Explicit constructions of MDS array codes and RS codes with

optimal repair bandwidth,” ISIT 2016, pp.1202-1206. Min Ye, Ph.D. Dissertation Defense


August 11, 2017

38 / 1

Fractional decoding: Global decoding from partial information Code of length n over Fq

C3

C2

C1

)

) C2

C1

f 2(

f1 (

Cn

...

f3 (C3 )

f n(

) Cn

decoder Suppose that Size of (f1 (C1 ), f2 (C2 ), . . . , fn (Cn )) ≤ αn`,

α≤1

How many errors can the decoder correct?



August 11, 2017

39 / 1

Fractional decoding: Global decoding from partial information

1 Min Ye, Ph.D. Dissertation Defense


August 11, 2017

40 / 1


• (n, k) MDS code C = {C = (C1 , . . . , Cn )}



August 11, 2017

40 / 1


• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F



August 11, 2017

40 / 1


• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the

codeword symbol (possibly, distorted by noise), α < 1



August 11, 2017

40 / 1



codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be

said about the number of (worst-case) correctable errors?

1

these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense


August 11, 2017

40 / 1




said about the number of (worst-case) correctable errors? Call this number the α-decoding radius

1



August 11, 2017

40 / 1




said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results:

1



August 11, 2017

40 / 1




said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results: • A bound on the α-decoding radius

1



August 11, 2017

40 / 1




said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results: • A bound on the α-decoding radius • A matching code construction

1



August 11, 2017

40 / 1

Decoding radius as a function of α



August 11, 2017

41 / 1

Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c



August 11, 2017

41 / 1

Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c • A trivial lower bound: Puncturing an (n, k) MDS code, we obtain

rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1



August 11, 2017

41 / 1


rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1 • Singleton-like upper bound:

rα (n, k) ≤ b(n − k/α)/2c



August 11, 2017

41 / 1



rα (n, k) ≤ b(n − k/α)/2c Main result: We construct codes with α-decoding radius that matches the upper bound



August 11, 2017

41 / 1



rα (n, k) ≤ b(n − k/α)/2c Main result: We construct codes with α-decoding radius that matches the upper bound



August 11, 2017

41 / 1

Summary • We constructed various families of MSR array codes



August 11, 2017

42 / 1

Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes



August 11, 2017

42 / 1

Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process



August 11, 2017

42 / 1

Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `



August 11, 2017

42 / 1


• We constructed explicit RS codes with optimal repair property



August 11, 2017

42 / 1


• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal

repair property



August 11, 2017

42 / 1



repair property • An almost matching lower bound on the sub-packetization of scalar linear MSR codes



August 11, 2017

42 / 1




• We introduced a new problem of fractional decoding (decoding from an αn

proportion of the received vector)



August 11, 2017

42 / 1





proportion of the received vector) • The decoding radius rα (n, k) is a function of α and (n, k)



August 11, 2017

42 / 1





proportion of the received vector) • The decoding radius rα (n, k) is a function of α and (n, k) • We give the exact expression of the decoding radius rα (n, k).



August 11, 2017

42 / 1

Some open problems

• Construct explicit MSR codes with sub-packetization ` = rn/(r+1)



August 11, 2017

43 / 1

Some open problems

• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16)



August 11, 2017

43 / 1

Some open problems

• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions



August 11, 2017

43 / 1

Some open problems


• Close the gap between achievable sub-packetization value and the lower bound



August 11, 2017

43 / 1

Some open problems


• Close the gap between achievable sub-packetization value and the lower bound • the best known lower bound (Goparaju et al., ’14):

2 log2 `(logr/(r−1) ` + 1) ≥ k.



August 11, 2017

43 / 1

Some open problems



2 log2 `(logr/(r−1) ` + 1) ≥ k. • Sub-packetization dependent bound on repair bandwidth (Cadambe-Mazumdar,

’15)



August 11, 2017

43 / 1

Some open problems




’15) • Cut-set bound is obtained without any restrictions on `



August 11, 2017

43 / 1

Some open problems




’15) • Cut-set bound is obtained without any restrictions on ` • What if we require ` < L? How does the lower bound change as L decreases?



August 11, 2017

43 / 1

Publication list: Journal papers Coding for distributed storage: 1. Min Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017, arXiv:1605.08630. 2. Min Ye and A. Barg, “Explicit constructions of high-rate MDS array codes with optimal repair bandwidth,” IEEE Transactions on Information Theory, vol. 63, no. 4, pp. 2001–2014, April 2017, arXiv:1605.00454. Statistics 1. Min Ye and A. Barg, “Optimal Schemes for Discrete Distribution Estimation under Locally Differential Privacy,” submitted to IEEE Transactions on Information Theory, May 2017, arXiv:1702.00610. 2. Min Ye and A. Barg, “Asymptotically optimal private estimation under mean square loss,” arXiv:1708.00059. Polar codes 1. T. C. Gulcu, Min Ye and A. Barg, “Construction of polar codes for arbitrary discrete memoryless channels,” submitted to IEEE Transactions on Information Theory, March 2016, arXiv:1603.05736. 2. Min Ye and A. Barg, “Polar codes for distributed hierarchical source coding” Advances in Mathematics of Communications, vol. 9, no. 1, pp. 87–103, 2015, arXiv:1404.5501. Min Ye, Ph.D. Dissertation Defense


August 11, 2017

44 / 1

Publication list: Conference papers Coding for distributed storage: 1. I. Tamo, Min Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” 58th Annual IEEE Symposium on Foundations of Computer Science (FOCS), Berkeley, CA, 2017. 2. I. Tamo, Min Ye and A. Barg, “Fractional decoding: Error correction from partial information,” in IEEE International Symposium on Information Theory (ISIT), Aachen, Germany, 2017, pp. 998–1002 (arXiv:1701.06969). 3. M. Vajha, G. Kini, B. Puranik, V. Ramkumar, E. Lobo, B. Sasidharan, P.V. Kumar, Min Ye, A. Barg, S. Hussain, S. Narayanamurthy, and S. Nandi, “Pairing up for regeneration: The Mantra for fast and efficient node repair in distributed storage,” poster presented at USENIX Annual Technical Conference, July 12-14, 2017, Santa Clara, CA. 4. Min Ye and A. Barg, “Explicit constructions of MDS array codes and RS codes with optimal repair bandwidth,” in IEEE International Symposium on Information Theory, Barcelona, Spain, 2016, pp. 1202–1206.

Statistics: 5. Min Ye and A. Barg, “Optimal Schemes for Discrete Distribution Estimation under Local Differential Privacy,” in IEEE International Symposium on Information Theory, Aachen, Germany, 2017, pp. 759–763.

Polar codes: 6. T. C. Gulcu, M. Ye, and A. Barg, “Construction of polar codes for arbitrary discrete memoryless channels,” in IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, July 11-15, 2016, pp. 51–55 7. Min Ye and A. Barg, “Polar codes using dynamic kernels,” in IEEE International Symposium on Information Theory (ISIT), Hong Kong, 2015, pp. 231–235. 8. Min Ye and A. Barg, “Universal source polarization and an application to a multi-user problem” in 52nd Annual Allerton Conference on Communication Control and Computing, 2014, pp. 805–812. Min Ye, Ph.D. Dissertation Defense


August 11, 2017

45 / 1

Thank you



August 11, 2017

46 / 1

Coding Schemes for Distributed Storage Systems - Google Sites

Coding Schemes for Distributed Storage Systems - Google Sites

Suggest Documents

Network Coding for Distributed Storage Systems - arXiv

Minimum-Cost Coding for Distributed Storage Systems - DiVA portal

Minimum-Cost Coding for Distributed Storage Systems - DiVA portal

Distributed Storage Allocation Problems - Erasure Coding for ...

Architecture-aware Coding for Distributed Storage ...

Randomized Network Coding in Distributed Storage Systems with ...

Randomized Network Coding in Distributed Storage Systems with

Linear Coding Schemes for the Distributed Computation of Subspaces

Tradeoff for Heterogeneous Distributed Storage Systems ... - arXiv

Experiences Building Network-Coding-Based Distributed Storage ...

Network Coding for Distributed Storage in Wireless ... - CiteSeerX

Remote Data Checking for Network Coding-based Distributed Storage ...

Decentralized Coding Algorithms for Distributed Storage in ... - arXiv

Coding Schemes for Securing Cyber-Physical Systems Against ...

Distributed Storage Schemes over Unidirectional Ring Networks - arXiv

Replication-based Distributed Storage Systems with ...

Securing Dynamic Distributed Storage Systems against ... - arXiv

Symmetry in Distributed Storage Systems - arXiv

Consistency in Distributed Storage Systems - Semantic Scholar

Availability in Globally Distributed Storage Systems - Usenix

Consistency in Distributed Storage Systems - Semantic Scholar

Proportional Service Allocation in Distributed Storage Systems

Highly Available Distributed Storage Systems Thesis

Distributed Space-Time Coding - Google Sites