Tradeoff for Heterogeneous Distributed Storage Systems between Storage and Repair Cost Krishna Gopal Benerjee and Manish K. Gupta
arXiv:1503.02276v1 [cs.IT] 8 Mar 2015
Laboratory of Natural Information Processing, Dhirubhai Ambani Institute of Information and Communication Technology Gandhinagar, Gujarat, 382007 India Email: krishna
[email protected],
[email protected] Abstract—In this paper, we consider heterogeneous distributed storage systems (DSSs) having flexible reconstruction degree, where each node in the system has dynamic repair bandwidth and dynamic storage capacity. In particular, a data collector can reconstruct the file at time t using some arbitrary nodes in the system and for an arbitrary node failure the system can be repaired by some set of arbitrary nodes. Using min-cut bound, we investigate the fundamental tradeoff between storage and repair cost for our model of heterogeneous DSS. In particular, the problem is formulated as bi-objective optimization linear programing problem. For an arbitrary DSS, it is shown that the calculated min-cut bound is tight.
I.
I NTRODUCTION
Cloud storage is a distributed storage system (DSS) in which information is stored on distinct nodes as encoded packets in a redundant manner. One can retrieve the file by contacting certain nodes in the system. In case of node failure, it can be repaired using other nodes in the system. For such DSSs, one has to optimize various parameters in the system such as storage capacity, repair bandwidth, availability, reliability, security and scalability. Such DSSs are used by many commercial systems like Facebook, Yahoo, IBM, Amazon and Microsoft Windows Azure system[1–4]. In homogeneous DSSs (where each node has same storage capacity and same repair degree) [5], encoded data packets of a file with size B are distributed among n nodes (each having storage capacity α) such that connecting any k(< n) nodes, one can retrieve the whole file. In the case of any arbitrary node failure, system is repaired by downloading β packets from any d(< n) nodes, called helper nodes [5]. In these systems, one can provide reliability by simply replicating or encoding the massage data packets. In the case of simple replication, storage minimization is inefficient. On the other hand, encoding of data packets using erasure MDS (maximum distance separable) codes leads to inefficiecy for bandwidth minimization during node repair process. To optimize these conflicting parameters, in a seminal work Dimakis et. al [5] introduced regenerating codes. In [6, 7], tradeoff between storage capacity α and repair bandwidth dβ is analyzed by plotting tradeoff curve for regenerating code. All points on the tradeoff curve can be obtained by linear network codes over finite fields [8, 9]. In the tradeoff curve, by minimizing both parameters in different order, Minimum Bandwidth Regenerating (MBR) codes and Minimum Storage Regenerating (MSR) codes are obtained [6]. Tradeoff between storage and repair bandwidth for exactrepair is studied in [10]. In [11], Shah et al calculated cutset lower bound on repair bandwidth for a special flexible
Fig. 1. A model of considered heterogeneous DSS is given here. In the heterogeneous DSS each node has flexible storage capacity αi (i ∈ {1, 2, . . . , n}) and repair bandwidth. In the system, at time t (in particular, t ∈ {t1 , t2 }) flexible reconstruction degree for a data collector, is kt . Repair degree for an arbitrary node failure is also dynamic with respect to time. At time t, a failed (t) node Ui is repaired by some di nodes.
setting for homogeneous DSS. In a nice survey [12], an overview of some existing results and repair models on DSS are explored. Recently in [14], the tradeoff between storage capacity and repair bandwidth is investigated for exact repair linear regenerating codes for k = d = n − 1. Heterogeneous DSSs are more close to real world scenarios where characterization of all storage nodes in various aspects are not necessarily uniform due to geographical environment and storage devices cost etc. Many such heterogeneous DSS have been studied recently [15–17]. In [18–20], storage allocation problem is investigated to maximizes the probability of successful recovery. For heterogeneous DSS, [21] proved that repair cost can be reduced by allowing helper nodes to encode the codewords of other nodes. In [13], Akhlaghi et al investigated the tradeoff between storage capacity and repair bandwidth for the generalized regenerating codes and shown that each point on curve is achievable. In the generalized regenerating code, set of all nodes is divided into two partitions. Every node in each partition has uniform parameters (αi , di , βi ) (∀i ∈ {1, 2}) [13]. In [22], Ernvall et al calculated the capacity bounds of a heterogeneous DSS having dynamic repair bandwidth. The tradeoff curve is explored for non-homogeneous two rack model of DSS in [23]. In [24] capacity bound is calculated for heterogeneous DSSs with dynamic repair bandwidth, where
packets y3 and y4 are distributed on node U2 , packets y5 and y6 are on node U3 , packets y7 , y8 and y9 are on node U4 and remaining two packets are on node U5 . Clearly the storage node capacity αi = 2(i ∈ [5]\{4}) and α4 = 3. In this example, if node U5 fails then it can be repaired by downloading packets y7 and y9 from node U4 . Since the recovered packets are function of lost packets so it is functional repair. On the other hand, node U5 can be repaired exactly by downloading packets y1 , y2 and y5 from nodes U1 and U3 and solving y10 = y2 and y11 = y1 + y5 .
Fig. 2. A file with size 4 units (= B) is divided into 11 encoded packets on field Fq . These packets are distributed among 6(= n) nodes in such a way that any data collector can download whole file by contacting at most 3(= k) nodes. In this heterogeneous DSS, α ~ = (α1 , α2 , α3 , α4 , α5 ) = (2 2 2 3 2). A failed node can be repaired by at most 2(= d) nodes. Functional and exact repairs are shown for the node failure of U5 with the help of surviving sets (2) (3) (3) S5 = {U4 } and S5 = {U1 , U3 }. Surviving set S5 is not considered in (1) (3) Table I since S5 $ S5 .
node repair is done by some specific helper nodes. For the heterogeneous DSS, the storage node capacity depends on the repair bandwidth of each rack. In [25], tradeoff between system storage cost and system repair cost is investigated for heterogeneous DSSs with dynamic storage and repair cost. In this work, we consider a heterogeneous DSS where a file of size B is distributed among n nodes each with different storage capacities. File reconstruction is done in a flexible manner, where at any time instant t, data collector can reconstruct the file by connecting some kt number of nodes. Hence the reconstruction degree kt for a file is flexible with respect to time and the number of nodes. On the other hand, in case of a node failure Ui (1 ≤ i ≤ n), it can be repaired at time (t) t by downloading packets from di number of some nodes. (t) Hence the repair degree di is also flexible with respect to time and the number of nodes. Repair of failed node can be done in two ways, exact repair and functional repair. If the recovered packets in repair process is exact copy of lost packets then it is called exact repair. On the other hand, if the recovered packets is some function of lost packets then the repair is functional repair. The model of such heterogeneous DSS is shown in Figure 1. A data collector reconstructs a distributed file by connecting ktj (j ∈ {1, 2}) number of nodes at time tj . (t ) In addition, a failed node Ui repairs by di j number of some nodes at time tj . An example of such heterogeneous DSS is considered in Figure 2. In this system, a file B is divided into 4 massage information packets x1 , x2 , x3 and x4 . The massage information packets are encoded into 11 packets by taking linear combination of massage information packets as y1 = x1 , y2 = x2 , y3 = x3 , y4 = x1 + x2 , y5 = x4 , y6 = x1 + x2 , y7 = x1 , y8 = x3 , y9 = x2 + x4 , y10 = x2 and y11 = x1 + x4 . The encoded packets ym (m ∈ [11]) are distributed on the 5 nodes such that packets y1 and y2 are stored on node U1 ,
Contribution: In this paper, we have calculated min-cut bound for the considered heterogeneous DSS. For such heterogeneous DSS, we have established a bi-objective optimization linear programing problem subject to min-cut bound. The solutions of the LP problem are plotted as a tradeoff curve between system storage and repair cost. In a heterogeneous DSS, system storage cost and system repair cost are average costs to store and repair unit information data on a node respectively. We have plotted some tradeoff curve and compared it with tradeoff for heterogeneous DSS as considered in [25] and homogeneous DSS as investigated in [7]. Some specific cases are investigated for the established bi-objective optimization problem. Organization: The paper is organized as follows. Section 2 collects the required preliminary concepts and describes our model. Section 3 investigates the min-cut bound for our model. Under the constraints of the min-cut bound, we also establish a bi-objective linear optimization problem to plot the tradeoff curve between storage and repair costs per node. Finally Section 4 concludes the paper with general remarks. II.
P RELIMINARIES
In this paper, we focus our attention to heterogeneous DSS with parameters (n, k, d), where file is distributed among n nodes, k = maxt {kt } is the maximum reconstruction degree (t) for the file, di = maxt {di } is the maximum repair degree for a node Ui at all time and d = maxi {di } is the maximum repair degree among all nodes at any time. For each time t, one can define a reconstruction set At as collections of the nodes having sufficient packets to reconstruct the file i.e. At = Ut1 , Ut2 , . . . , Utkt . Clearly |At | = kt and intersection of any two reconstruction set may be non-empty. Define A = {A1 , A2 , . . . , At , . . .} as a set of all reconstruction sets. Note that the set A will be finite if all reconstruction sets A ∈ A are distinct. Hence ∃ ω ∈ N such that |A | = ω. For the considered example in Figure 2, A = {Ai : ∀i ∈ [7]}, where A1 = {U1 , U2 , U3 }, A2 = {U1 , U3 , U5 }, A3 = {U1 , U4 }, A4 = {U2 , U4 }, A5 = {U2 , U5 }, A6 = {U3 , U4 } and A7 = {U4 , U5 }. In the heterogeneous DSS, at time t, if a node Ui (i ∈ [n]) fails then certain nodes called helper nodes, download required packets and generate a new node say Ui0 . The new node Ui0 replaces the failed node Ui and the system is repaired. In particular, set of those helper nodes are called surviving set. For a node Ui , let the number of distinct surviving sets are τi . At the time instant t, indexing the surviving set by `, one (`) 4 can denote them by Si = {Uj : some j ∈ {1, 2, ..., n}\{i}}, where ` ∈ [τi ] [24]. If a node failure Ui repairs by nodes of (`) surviving set Si then repair degree at the time instant t,
(t)
(`)
is di = |Si |. Surviving sets for the heterogeneous DSS considered in Figure 2 is listed in Table I. In this example, one can see that if a node U4 fails then it can be repaired by connecting nodes U2 and U3 or nodes U2 and U5 . Hence (1) (2) surviving sets for the node U4 are S4 and S4 . In Table I, (`) for a given i(i ∈ [5]), |Si | is identical for all ` ∈ [τi ]. In general, it may not be true. Also note that, in the table, we have chosen those surviving sets which are not the super set of other surviving set for the same node failure. In particular, the condition ensures the active participation of each node of an arbitrary surviving set during system repair process. TABLE I.
S URVIVING SETS FOR NODES IN DSS AS CONSIDERED IN F IGURE 2.
Nodes
Surviving sets Si (3)
S1 (1) S2
τi
(2)
(1)
U1 U2 U3 U4 U5
# sets
(`)
Ui
S1 = {U2 , U4 }, S1 = {U2 , U5 }, (4) (5) = {U3 , U4 }, S1 = {U3 , U5 }, S1 = {U4 , U5 }. (2) (3) = {U1 , U4 }, S2 = {U3 , U4 }, S2 = {U4 , U5 }. (2) (1) S3 = {U4 }, S3 = {U5 }. (1) (2) S4 = {U2 , U3 }, S4 = {U2 , U5 }. (1) (2) S5 = {U3 }, S5 = {U4 }.
5 3 2 2 2
In brief, for a failed node Ui , if system is repaired by nodes (`) of specific surviving set Si then the number of information (`) packets downloaded by node Uj ∈ Si will be given by (`) β Ui , Uj , Si > 0. For example, in Figure 2, all two packets from node U5 and packet y3 = x3 from node U2 is downloaded (2) torepair node failure U4 . Note that U2 ,U5 ∈ S4 . Hence (2) (2) β U4 , U5 , S4 = 2 and β U4 , U2 , S4 = 1. If a failed node Ui (i ∈ [n]) is repaired by nodes of surviv (`) (`) ing set Si then repair bandwidth (denoted by γ Ui , Si ) for the node Ui is the total number of packets downloaded by (`) every nodes of the surviving set Si . Mathematically (`) γ Ui , Si =
(`) β Ui , Uj , Si .
X
(1)
Fig. 3. Information flow graph G = (V, E) for a heterogeneous DSS. The graph is divided into kt + 3 (kt is flexible reconstruction degree at time t associated with data collector node “D”) step labels. Node Uλi (∀i ∈ [n]) in heterogeneous DSS is represented by a pair of nodes Inλi and Outλi . Sours nose “s” is in step label −1, data collector node “D” is in label kt + 1. Step label 0 has n pairs of nodes Inλi and Outλi . Each step from label 1 to kt has one pair of nodes Inλj and Outλj (∀j ∈ [kt ]). Mansion that node “D” (data collector node) at right upper corner is in step label kt + 1, in place of step label kt .
repair and storage cost can be plotted using optimization Problem 17, if any two data collectors are not connected with some common node. To plot the tradeoff curve between storage capacity α and repair bandwidth dβ in a homogeneous DSS, Wu et al [7] solved an optimization problem with constraint of min-cut bound between the parameters. The bound is calculated by analyzing the information flow graph for the homogeneous DSS [7]. In the similar manner, one can plot the trade off curve for our model. We consider the information flow graph (acyclic weighted directed graph G = (V, E)) [24, 25] for heterogeneous DSS as described in Figure 3.
j
such that
(`)
Uj ∈Si
For example, in Figure 2, if node U5 fails and it is repaired (3) by nodes of surviving set S5 (not considered in Table I since (1) (3) S5 $ S5 ) then (3) (2) (3) + β U5 , U3 , S5 = 2+1 = γ U5 , S5 = β U5 , U1 , S5 3 units. Remark 1. In this paper, at time instant t, single node failure is considered because simultaneously multi-node failures can be assumed as a sequence of single node failure. Remark 2. One can find the tradeoff curve between repair cost and storage cost by optimization Problem 17 for the exact or functional repair using the surviving sets as the collection of those helper nodes which repair failed nodes as exact or functional respectively. Remark 3. One can modify our heterogeneous DSS model by allowing some data collectors to reconstruct file separately at same time instant t with flexible reconstruction degree each. For the particular modified model, the tradeoff curve between
For a heterogeneous DSS, at time t, the information flow graph G as shown in Figure 3, is divided into kt + 3 (kt being flexible reconstruction degree for data collector at time t) steps, starting from step label −1 to label kt + 1. Step label −1 contains source node say “s” and step label kt + 1 contains data collector node say “D”. A typical node Uλi (∀i ∈ [n]) in heterogeneous DSS, is mapped to a pair of vertices “Inλi ” and “Outλi ” in V s.t. (Inλi , Outλi ) ∈ E, where λi is permute index on nodes. Storage capacity αλi of node Uλi is mapped to w(Inλi , Outλi ), where w(Inλi , Outλi ) is weight associated with edge (Inλi , Outλi ) ∈ E. In graph G as given in Figure 3, at step label 0, there are 2n number of vertices named Inλi and Outλi associated with nodes Uλi (i ∈ [n]) in heterogeneous DSS. A failed node Uλi (i ∈ [n]) in heterogeneous DSS, is repaired by generating new node Uλ0 i . The node Uλ0 i is mapped to a new pair of nodes In0λi and Out0λi s.t. (In0λi , Out0λi ) ∈ E with w(In0λi , Out0λi ) = αλi . Every step label j ∈ [kt ] contains one pair of nodes In0λj and Out0λj . As shown in Figure 3, in the heterogeneous DSS, system is repaired for the node failure
2)
(flow conservation constraint:) ∀y ∈ V\{s, t}, X X f ((x, y)) = f ((y, z)). x
z
(x,y)∈E
(y,z)∈E
For more details and example on flow function, cite [26, 27].
Fig. 4. For a heterogeneous DSS as considered in Figure 2, a information flow graph is shown for a specifics data collector connects with the nodes of A1 ={U1 , U2 , U3D }. The particular Einformation flow graph is plotted for (1) (1) (1) surviving sequence S1 , S2 , S3 ∈ S (hU1 , U2 , U3 i). Mansion that node “D” at right upper corner is in step label 4, in place of step label 3.
(`) Uλj by downloading β Uλj , Uµp , Sλj amount of data from (t) (`) every node Uµp ∈ Sλj = {Uµp : j ∈ [dλj ]} , where µp is some permutation on nodes. For the particular system repair, each downloading process maps by one distinct edge from some previous step label to step label j s.t. (Out0µp , In0λj ) ∈ E (`) with w(Out0µp , In0λj ) = β Uλj , Uµp , Sλj . In particular, if node Out0µp does not exist then consider Outµp from step label 0 s.t. (Outµj , In0λj ) ∈ E. In graph G exactly one node failure is considered in each step label. A data collector D connects kt number of nodes of At = {Uλ0 1 , Uλ0 2 ,...,Uλ0 j ,...,Uλ0 k }. In information graph as in Figure t 3, data collector D connects nodes Out0λj (∀j ∈ [kt ]) from step label 1 to step label kt and downloads certain data file then (Out0λj , D) ∈ E such that w(Out0λj , D) → ∞. Example 4. At time instant t, for the heterogeneous DSS in Figure 2, an example of information flow graph is shown in Figure 4. In particular, a data collector is connected with the nodes of A1 = {U1 , U2 , U3 }. In the information flow graph, if the nodes are failed then it will be repaired by nodes of (1) (1) (1) S1 , S2 and S3 respectively.
For a given information flow graph G = (V, E), value of flow delivered to a data collector node say D is defined as total amount of flow passes through the edges (x, D) ∈ E for all possible x ∈ V. For networks, maximum possible value of flow delivered to D is governed by min-cut max-flow theorem [26– 28]. Min-cut max-flow theorem says that across the network, maximum possible value of flow passes from source s to specific data collector D denoted by max-f low(s, D), is equal to minimum cut-capacity(s, D), where min cut-capacity(s, D) = min cut-capacity(X , X ) . cut(X ,X ); s∈X ,D∈X ;
X ∪X =V.
Note that cut(X , X ) represents the set of all edges having one end vertex in set X and other vertex in set X such that removing those all edges will improve the number of components in graph G = (V, E). Here cut-capacity(X , X ) is the sum of capacity of all edges in cut(X , X ). At time t, for a specific data collector D which connects nodes Uλi of set At ∈ A, has Q|A | |At | ! i=1t τλi number of distinct information flow graphs are exist. For every information flow graph G = (V, E), D can recover the whole file B so B ≤ min max -f low(s, D). G
By min-cut max-flow theorem for an arbitrary data collector D one can compute, B ≤ min min max -f low(s, D). t
G
In [7], for an information flow graph, flow analysis is done by taking topological order of failed node connected with data collector. In this paper, we are defining some sequences of nodes and corresponding surviving sets for our model to analyze flow. The definitions are as follows. Definition 6. A set of all possible sequences of nodes in a reconstruction set Aj ∈ A is ncalled reconstruction sequence o |A | set and denoted by A (Aj ) = hUλi ii=1j : Uλi ∈ Aj , where |A |
In [7, 22, 24, 25], min-cut bound is calculated by analyzing flow passes through source node s to data collector node D across the information flow graph for a DSS. In the similar manner flow analysis is done for the model considered in this paper. Hence one can define flow across the information flow graph as follows. Definition 5. A function f : E → [0, ∞) is called flow on a information flow graph G = (V, E) if,
hUλi ii=1j represents a sequence of distinct nodes of set Aj ∈ A. Clearly |A (Aj )| = |Aj | !. For example, in Figure 2, A (A3 ) = {hU1 , U4 i , hU4 , U1 i} etc. Definition 7. For a reconstruction set Aj ∈ A, one can (`) define sequences of surviving sets Sλi (∀i ∈ [|Aj |], ∃` ∈ [τλi ]) such that Uλi ∈ Aj . Surviving sequence associated with node D E|Aj | |A | (`) sequence hUλi ii=1j ∈ A can be denoted by Sλi . i=1
1)
(capacity constraint:) ∀(x, y) ∈ E, f ((x, y)) ≤ c((x, y)), where c((x, y)) = w(x, y) and c((x, y)) is capacity of edge (x, y).
For example, in Figure 2, a possible surviving sequence D E (3) (2) for the node sequence hU1 , U4 i is S1 , S4 .
Definition 8. Set of all surviving sequences associated with a |A | node sequence hUfi ii=1j can be defined as follows, D E|Aj | |A | (`) S hUλi ii=1j = Sλi : ∃` ∈ [τλi ] . i=1
Q |Aj | |A | τ !. Clearly S hUλi ii=1j = λ i i=1 nD For example, E in Figure 2, one canosee that S (hU1 , U4 i) = (`1 ) (`2 ) S1 , S4 : ∃`1 ∈ [5], ∃`2 ∈ [2] etc. In [25], Quan et al have given tradeoff curve between system storage cost and system repair cost for heterogeneous DSS with uniform reconstruction degree. Similarly one can give tradeoff curve between system storage cost and system repair cost for heterogeneous DSS model considered in our paper. For our model, we define system storage cost, node storage cost and system repair cost as follows. Definition 9. (System storage cost): Total amount of cost Cs (~ α) to store unit data in heterogeneous DSS(n, k, d) is called system storage cost, where storage amount vector α ~, (α1 , α2 , . . . , αn ), storage cost vector ~s , (s1 , s2 , . . . , sn ), αi is storage capacity of node Ui and si is the cost to store unit information data in node Ui (∀i ∈ [n]). Clearly n 1 X Cs (~ α) = sj αj B j=1
System storage cost Cs (~ α) for the example considered in Figure 2 with ~s = (100, 10, 10, 10, 1) is 68 cost units. Definition 10. (Node repair cost): The average amount of cost to repair a node Ui (i ∈ [n]) in heterogeneous DSS(n, k, d) is called node repair cost r(βi ) associated with repair cost vector ~r , (r1 , r2 , . . . , rn ) s.t. r(βi ) =
τi X 1 X (`) rj β(Ui , Uj , Si ), Bτi j
(2)
`=1
(`)
Uj ∈Si
where rj is cost to download unit amount of data from node ~ , Uj during repair process. Clearly node repair vector r(β) (r(β1 ), r(β2 ), . . . , r(βn )). In the example considered in Figure 2, if ~r ~ (10, 1, 1, 1, 1) then node repair cost vector r(β) 1 4 1 3 1 (r(β1 ), r(β2 ), r(β3 ), r(β4 ), r(β5 )) = ( 2 , 3 , 2 , 4 , 2 ).
= =
Definition 11. (System repair cost): System repair cost is total amount of cost to repair all nodes in heterogeneous ~ Mathematically DSS(n, k, d) and denoted by Cr (β). ~ = Cr (β)
n X
r(βj ).
j=1
~ = 43 cost unit for ~r = (10, 1, 1, 1, 1) in the Clearly Cr (β) 12 example considered in Figure 2. Now we give some results and analysis for the min-cut bound for the model of heterogeneous DSS considered in this paper in the next section.
III.
R ESULTS
For our model, it is shown that minimum possible value of flexible reconstruction degree is lower bound of cardinality of any cut set which separates source node and data collector node. For the heterogeneous DSS min cut bound is calculated in Theorem 13. Using that min cut bound, it is shown that file size should be lower bound of min cut bound for the heterogeneous DSS. Using the particular bound as constraint, a bi-objective optimization linear programing problem is formulated to minimize system storage cost and system repair cost for the considered heterogeneous model. A family of solutions is calculated for the optimization problem by substituting some numerical values of system parameters. The numerical parameter is plotted the tradeoff curve between system storage cost and system repair cost. The curve is compared with tradeoff curve for homogeneous DSS [7] and tradeoff curve for heterogeneous DSS [25]. Lemma 12. An arbitrary information flow graph G = (V, E) with source node s ∈ X , flexible reconstruction degree kt associated with data collector node D ∈ X has min |cut(X , X )| : cut(X , X ) 6= φ ≥ min{kt }, t
X ⊂V
where X ∪ X = V. Proof: Consider a heterogeneous DSS associated with some information flow graphs. For any arbitrary information flow graph G = (V, E), ∃ X ⊂ V such that s ∈ X , D ∈ X . Since information flow graph is connected graph so cut(X , X ) 6= φ for any nonempty set X and X . To retrieve the distributed file for the case X = {D}, one has to connect at least mint {kt } number of nodes among n nodes. Hence each node of an arbitrary set of mint {kt } number of nodes, has some encoded data of distinct part of massage data. So there are at least mint {kt } number of edges having end point as node D. Mathematically, |cut(X , X )| = mint {kt } for X = {D}. Again, if |X | > 1 then some edges in cut(X , X ) represents downloading process for system repair. In particular, a node failure among the mint {kt } number of the specific nodes, can not be repaired by the some subset of the remaining mint {kt } − 1 number of the nodes. Reason behind that, each node in the set of mint {kt } number of nodes has encoded data packets of some unique message data packets. Hence, there must exist some helper nodes other then the mint {kt } − 1 number of nodes for the repair the failed node. So |cut(X , X )| ≥ mint {kt }. But X is any arbitrary nonempty subset such that cut(X , X ) exist, so |cut(X , X )| ≥ mint {kt } for all possible cut(X , X ) 6= φ. This proves the lemma. In a heterogeneous DSS, information delivered to data collector D depends on mincut-capacity(s, D). The Theorem 13 gives the lower bound of min cut-capacity(s, D). Theorem 13. (min-cut bound) For a given heterogeneous DSS with an arbitrary data collector D associated with flexible reconstruction degree kt , the min cut-capacity(s, D) is bounded below by Q as given by Equation (5), i.e. min cut-capacity(s, D) ≥ Q.
(3)
Proof: Consider a heterogeneous DSS (n, k, d) associated with some information flow graphs. Every information flow graph G = (V, E) has a source node s, a data collector node D associated with effective reconstruction degree kt . In the heterogeneous DSS, a failed node Ui can be repaired by nodes (`) of some surviving set Si , where ` ∈ [τi ]. Let X ⊂ V, X ∪ X = V, s ∈ X and D ∈ X such that some nonempty subset cut(X , X ) ⊂ E exist. Now if X = V\{D} then cut-capacity(X , X ) → ∞. Similarly if X = {s} then again cut-capacity(X , X ) → ∞. Hence min cut-capacity(X , X ) would be obtained by all those Out0j ∈ X and Ini ∈ X since it will give a finite cutcapacity(X , X ), where i ∈ [n] and j ∈ [kt ]. Information flow graph G = (V, E) is directed acyclic graph so it can be represented in a topological order of its vertices. For the topological order, sequences of node failure and corresponding sequence of surviving sets are arranged by using definitions as given in previous section. For that assume at time t, data collector D connects with all nodes of a set At ∈ A and reconstruct the file B. A(At ) is the set of all possible sequences of nodes of At ∈ A. A sequence kt ∈ A(At ) represents the order of nodes failure of hUλi ii=1 specific D set AEt . Recall the set of all possible surviving se(`)
kt
k
t associated with a node sequence hUλi ii=1 , quences Sλi i=1 kt is S hUλi ii=1 .
kt hUλi ii=1
For a specific node sequence with a specific D Ekt (`) , one can analyze the following surviving sequence Sλi i=1
Out0λ1
For ∈ X associated with the first node in node kt , the following two cases are possible. sequence hUλi ii=1 •
If In0λ1 ∈ X then edge (In0λ1 , Out0λ1 ) ∈ cut(X , X ). Hence αλ1 will contribute in cut-capacity(X , X ).
•
If In0λ1 ∈ X then edges (Outµj , In0λ1 ) ∈ cut(X , X ), D Ekt (`) (`) (`) for where Uµj ∈ Sλ1 and Sλ1 ∈ Sλi i=1 any ` ∈ [τλ1 ]. Hence this case contribute in cutcapacity(X , X ) by X (`) β Uλ1 , Uµj , Sλ1 .
For the remaing part of the proof we have used the notation Up (∀p ∈ [n]) in place of Up0 since characteristics of both nodes Up and Up0 are same and one of them appears at instant. In general to compute contribution in min cutkt capacity(X , X ) supported by node Uλi ∈ hUλi ii=1 assume Out0λi ∈ X . Again following two cases are possible. •
If In0λi ∈ X , then edge (In0λi , Out0λi ) ∈ cut(X , X ). Hence αλi will contribute in cut-capacity(X , X ).
•
If In0λi ∈ X then all possible edges (Outµj , In0λi ) (`) s.t. Uµj ∈ Sλi \{Uλ1 , Uλ2 , . . . , Uλi−1 } associated D Ekt (`) (`) Sλi ∈ Sλi for any ` ∈ [τλi ], will contribute i=1
in cut(X , X ). Edges (Outλj , In0λi ) associated with (`) node Uµj ∈ Sλi \{Uλ1 , Uλ2 , . . . , Uλi−1 } are newly investigated from step label 0 for cut(X , X ). Edges (Out0λm , In0λi ) must be excluded because they have investigated earlier at step label m, where m ∈ [i − 1] (`) s.t. Uλm ∈ Sλi . Hence this case contribute in cutcapacity(X , X ) by X (`) β Uλi , Uµj , Sλi . µj (`)
Uµj ∈Sλ \{Uλ1 ,Uλ2 ,...,Uλi−1 } i
So contribution in min cut-capacity(X , X ) by node Uλi is X (`) . β Uλi , Uµj , Sλi min αλi , µ j (`) Uµj ∈Sλ \{Uλ1 ,Uλ2 ,...,Uλi−1 } i
At the time instant t, if data collector D connects with each nodes Uλi ∈ At , i ∈ [kt ] then for a specific node sekt quence hUλi ii=1 associated with a specific surviving sequence Ekt D (`) the contribution in min cut-capacity(X , X ) is Sλi i=1 kt X X (`) min αλi , β Uλi , Uµj , Sλi . µj i=1 (`) Uµj ∈Sλ \{Uλ1 ,Uλ2 ,...,Uλi−1 } i
µj (`)
Uµj ∈Sλ
1
So contribution in min cut-capacity(X , X ) supported by node Uλ1 is X (`) min αλ1 , β Uλ1 , Uµj , Sλ1 . µj (`) Uµj ∈Sλ
1
If a node Up (∀p ∈ [n]) fails in the system then all nodes (`) of some surviving set Sp will generate a new node Up0 with same characteristic. At a time instant t, one of them is in the system. Hence for the remaining part of the proof, we are writing Up in place of Up0 .
Now one can find min cut-capacity(s, D) for a specific D by taking minimum among all possible cut-capacity(X , X ) kt which is calculated for all possible node sequences hUλi ii=1 D Ekt (`) among all possible associated surviving sequences Sλi . i=1 The mincut-capacity(s, D) is the minimum value of the particular mincut-capacity(s, D) calculated for all possible specific D. For a given heterogeneous DSS, associated with any arbitrary data collector D one can find min cut-capacity(s, D) as Inequality (3) by using Equation (4). The particular Equation (4) holds because index λi of storage node capacity is kt governed by index λi of nodes in node sequence hUλi ii=1 . One can easily observe that the bound is tight since the min-cut bound is calculated by taking the minimum value of all possible cut bounds. Hence one can say the following.
min
D Ek min kt ∈A (At ) S (`) t ∈S hUλ ikt hUλi ii=1 λi i i=1 i=1
=
min
k
t ∈A (At ) hUλi ii=1
kt X i=1
Uλi ∈At
min
(`) β Uλi , Uµj , Sλi
X
αλi ,
Uλi ∈At
X
Remark 14. For a given heterogeneous DSS, at time t, if an arbitrary data collector connects each node Uλj in subset At ∈ A then total number of possible information flow graphs are given by |At | X Y |At | ! τλj .
Problem 17.
j=1
In particular, for a specific information flow graph, the total number of computational comparisons are 2|At | . Hence One can say that the time complexity to calculate min-cut bound is |At | Y X |At | O 2 (|At | !) τλj . j=1
At
At ∈A
By Theorem 13 one can calculate the minimum requirement of storage node capacity and repair bandwidth to store a file with size B. In other words the upper bound of stored file with size B is given by the following lemma. Lemma 15. If a file with size B is stored in a given heterogeneous DSS (n, k, d) then B ≤ Q,
(6)
whereQ is given in Equation (5) and remaining used notations have common meaning as defined in previous sections. Proof: Any arbitrary data collector node D must be able to reconstruct the whole file with size B. Hence maximum information flow value delivered to any data collector, should be at least B. Now using min-cut max-f low theorem and Theorem 13 one can prove the lemma. Example 16. The min cut-capacity(s, D) for the information flow n graph as shown in Figure 4, will o (1) (1) be min α1 , β U1 , U2 , S1 + β U1 , U4 , S1
(4)
n o (1) min α2 , β U2 , U4 , S2 n o (1) min α3 , β U3 , U4 , S3 =2 + 1 + 2 = 5 units.
+
At ∈A
(`) β Uλi , Uµj , Sλi
µj (`) Uµj ∈Sλ \{Uλ1 ,...,Uλi−1 } i
The min-cut bound is calculated for all possible node kt associated with all possible surviving sequences hUλi ii=1 D Ekt (`) sequences Sλi . Hence there exist at least one surviving Di=1 ∗ Ekt (` ) sequences, say, Sλi associated with node sequence, i=1
∗ kt say, Uλi i=1 for which the inequality holds with equality i.e. the min-cut bound Inequality (3) is tight.
At
µj (`)
Uµj ∈Sλ \{Uλ1 ,...,Uλi−1 }
i kt X min αλi , D E min k (`) kt Sλ ∈S hUλi i t i=1 i=1 i i=1
+
Now one can frame a optimization problem to find minimum system storage cost and system repair cost under the constraint that the maximum possible information deliver to data collector node D is at lest B.
~ Minimize: [Cs (~ α), Cr (β)] subject to Inequality (6); αi ≥ 0; (`) β Ui , Uj , Si ≥ 0; (`)
where i ∈ [n], ` ∈ [τi ] and Uj ∈ Si
for some j ∈ [n]\{i}.
Optimum values for the both objective functions of biobjective optimization Problem 17 are plotted as tradeoff curve ~ In this paper the optimization between Cs (~ α) and Cr (β). Problem 17 is solved by weighted sum method for some numeric example. Some specific cases for optimization Problem 17 are analyzed in the following subsection.
A. Some Specific Cases Considered heterogeneous DSS can be reduced to following cases under some specific restrictions. The cases are as follow: 1) (Uniform Reconstruction): At time t, if an arbitrary data collector can retrieve the file by downloading data from exactly k nodes for any combination out of n nodes then the constraint Inequality (6) for the optimization Problem 17 has additional property kt = k, ∀t. 2) (Uniform Repair Degree): For a heterogeneous DSS let a node failure can repair by any d nodes out of remaining n − 1 nodes. Under the particular assumption the constraint Inequality (6) for the optimization Problem 17 reduced to
Q = min
kt X
min
k
At ∈A
t U ∈A (At ) |At |=kt h λi ii=1
i=1 Uλi ∈At
min
αλi , D
(`)
Sλ
min k ∈S hUλi i t i=1 i=1
i
(`) β Uλi , Uµj , Sλi
X
Ekt
µj (`)
Uµj ∈Sλ \{Uλ1 ,...,Uλi−1 }
(5)
i
Problem 18. ~ Minimize: [Cs (~ α), Cr (β)] subject to kt X X (`) B ≤ min min αλi , β Uλi , Uµj , Sλi ; At ∈A i=1; µ
One can easily find solution of the bi-objective optimization Problem (17) for some numerical values and plot the solution as tradeoff curve for the same. One can compare the tradeoff curve with the tradeoff curve for the existing heterogeneous DSS investigated in [25]. Hence in the next section we are calculating some optimum solutions for numerical parameter for our model and comparing it with homogeneous model [7] and heterogeneous model [25].
0 ≤ α1 ≤ α2 ≤ . . . ≤ αn ; 1 ≤ λ1 ≤ λ2 ≤ . . . ≤ λkt ≤ n;
B. Numerical Work
|At |=kt
j
Uλi ∈At
where index µj is the index of node Uµj ∈ (`) (`) Sλi \{Uλ1 , . . . , Uλi−1 } such that {Uλ1 , . . . , Uλi−1 } ⊂ Sλi , ∀i ∈ [kt ] and some j ∈ [d]. (`) In this case |Sm | = d, τm = n−1 d , ∀m ∈ [n]. Here min cut-capacity(s, D) will be given by the node sekt ∈ A(At ) associated with surviving sequence quence hUλi ii=1 Ekt D (`) Sλi such that αλ1 ≤ αλ2 ≤ . . . ≤ αλkt and i=1
(`)
{Uλ1 , Uλ2 , . . . , Uλi−1 } ⊂ Sλi . 3) (Uniform Repair Download Amount): In this case we assume that downloaded amount from any arbitrary helper node to repair the system is constant say β. Hence optimization Problem 17 under the restriction has additional properties (`) as β Ui , Uj , Si = β, β ≥ 0 (∀i ∈ [n], all possible j ∈ [n]\{i} and ∀` ∈ [τi ]). 4) (Homogenous DSS): In heterogeneous DSS become a homogeneous DSS if characteristics of parameters are uniform. Hence assume effective reconstruction degree for any data collector is k and storage capacity of each node is α. In addition let a node failure can repair by any d nodes out of remaining n − 1 nodes by downloading β packets from each helper node. Under these restrictions, the constraint Inequalities (6) for the optimization Problem 17 reduced to Problem 19. ~ Minimize: [Cs (~ α), Cr (β)] subject to k X B≤ min {α, (d − i − 1) β} ; i=1
α ≥ 0; β ≥ 0. 5) (Other): In this paper, the considered heterogeneous DSS model can be reduced into some more specific DSS by applying some appropriate restrictions on constraints. For example, heterogeneous DSS with uniform reconstruction and uniform repair degree (case 1 and 2 respectively) collectively reduces to heterogeneous DSS as investigated in [25].
For the optimization Problem (17), LP problems with single objective function is solved. The single objective function is calculated by taking linear combination of the two objective functions of optimization problem (17). Ten such LP problems are solved by taking distinct linear combination factor between 10−3 and 103 . Plotting tradeoff and solving LP problems are done with the help of ‘MATLAB’ and ‘lp solve’ [29]. In Figure 5, four tradeoff curves are plotted between system repair cost Cr and system storage cost Cs for the respective DSSs. In particular Figure 5, one curve is plotted for homogeneous DSS as investigated in [7], another one is drown for a heterogeneous DSS as investigated in [25] and remaining two curves are plotted for two heterogeneous DSSs as studied in this paper. In particular, one of the remaining two curves has minimum effective reconstruction degree kmin is 2 and other has maximum effective reconstruction degree kmax is 2. For all considered DSSs the common parameters are as follow: n = 4, B = 1 unit, ~s = (1 10 10 100) and ~r = (10 1 1 1). For homogeneous DSS and heterogeneous DSS studied in [25] have reconstruction degree k = 2 and repair degree d = 3. Remaining both heterogeneous DSS have (1) (1) (1) surviving sets S1 = {U2 , U3 , U4 }, S2 = {U1 , U4 }, S3 = (1) {U1 , U2 }, S4 = {U2 , U3 }. In Figure 5, one can see that our heterogeneous DSS model has more optimum system storage and repair cost then the homogeneous DSS studied in [7]. Although the characteristics of our heterogeneous model and heterogeneous model investigated in [25] are different, but we obtained some more optimum points for our model as in Figure 5. It is shown in the last subsection that one can find heterogeneous DSS considered in [25] by taking some restrictions on our model. Remark 20. In the particular tradeoff curves, non-integer solution of bi-objective optimization problem 17 is also considered. Since the scaling of an arbitrary file size B to 1, leads to respective integer solution that is not necessarily scale to some integer. IV.
C ONCLUSION
In this paper, we proposed a model of heterogeneous DSS with dynamic reconstruction degree, storage node capacity and repair bandwidth. In particular, at time t, a file can be
[6]
[7]
[8]
[9]
Fig. 5. For various DSSs the optimal tradeoff curve is plotted between system repair cost Cr and system storage cost Cs .
reconstructed using certain set of nodes and system is repaired for any failed node by contacting some set of helper nodes. For such heterogeneous DSS, the fundamental tradeoff curve between system repair cost and system storage cost is investigated. To plot the tradeoff curve, a bi-objective optimization problem is formulated with the constraints of min-cut bound and non-negative parameters of the heterogeneous DSS. The bi-objective optimization problem is solved by weighted sum method for some numerical values of parameters of the heterogeneous model. Analyzing the tradeoff curve, we observed some more optimum points then the existing heterogeneous model [25]. The considered model is close to real world scenario. Our heterogeneous model is flexible enough to mold it into any existing heterogeneous or homogeneous DSS by considering appropriate restrictions. It would be interesting to construct codes achieving the optimum points on the tradeoff curve.
[10]
[11]
[12]
[13]
[14]
R EFERENCES [1] M. Sathiamoorthy, M. Asteris, D. Papailiopoulos, A. G. Dimakis, R. Vadali, S. Chen, and D. Borthakur, “Xoring elephants: Novel erasure codes for big data,” Proceedings of the VLDB Endowment (to appear), 2013. [2] C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, “Erasure coding in windows azure storage,” in Proceedings of the 2012 USENIX conference on Annual Technical Conference, ser. USENIX ATC’12. Berkeley, CA, USA: USENIX Association, 2012, pp. 2–2. [Online]. Available: http: //dl.acm.org/citation.cfm?id=2342821.2342823 [3] Microsoft, “SkyDrive Live,” Jan. 2013. [Online]. Available: https://skydrive.live.com/ [4] Amazon, “Amazon elastic compute cloud (Amazon EC2),” Jan. 2013. [Online]. Available: http://aws. amazon.com/ec2/ [5] A. Dimakis, P. Godfrey, M. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” in INFOCOM 2007. 26th IEEE International
[15]
[16]
[17]
Conference on Computer Communications. IEEE, May 2007, pp. 2000 –2008. A. Dimakis, P. Godfrey, Y. Wu, M. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” Information Theory, IEEE Transactions on, vol. 56, no. 9, pp. 4539–4551, Sept 2010. Y. Wu, R. Dimakis, and K. Ramchandran, “Deterministic regenerating codes for distributed storage,” in The Allerton Conference on Communication, Control and Computing (Urbana-Champaign), 2007. Y. Wu, “Existence and construction of capacity-achieving network codes for distributed storage,” in Information Theory, 2009. ISIT 2009. IEEE International Symposium on, 28 2009-july 3 2009, pp. 1150 –1154. ——, “Existence and construction of capacity-achieving network codes for distributed storage.” IEEE Journal on Selected Areas in Communications, vol. 28, no. 2, pp. 277–288, 2010. [Online]. Available: http://dblp.uni-trier. de/db/journals/jsac/jsac28.html#Wu10 S. Goparaju, S. El Rouayheb, and R. Calderbank, “New codes and inner bounds for exact repair in distributed storage systems,” in Information Theory (ISIT), 2014 IEEE International Symposium on, June 2014, pp. 1036– 1040. N. Shah, K. Rashmi, and P. Vijay Kumar, “A flexible class of regenerating codes for distributed storage,” in Information Theory Proceedings (ISIT), 2010 IEEE International Symposium on, june 2010, pp. 1943 –1947. A. Dimakis, K. Ramchandran, Y. Wu, and C. Suh, “A survey on network codes for distributed storage,” Proceedings of the IEEE, vol. 99, no. 3, pp. 476 –489, march 2011. S. Akhlaghi, A. Kiani, and M. R. Ghanavati, “Cost-bandwidth tradeoff in distributed storage systems,” Computer Communications, vol. 33, no. 17, pp. 2105 – 2115, 2010, special Issue:Applied sciences in communication technologies. [Online]. Available: http://www.sciencedirect.com/science/article/ pii/S0140366410003506 N. Prakash and M. Nikhil Krishnan, “The StorageRepair-Bandwidth Trade-off of Exact Repair Linear Regenerating Codes for the Case d = k = n − 1,” ArXiv e-prints, Jan. 2015. J. Kubiatowicz, D. Bindel, Y. Chen, S. Czerwinski, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, and B. Zhao, “Oceanstore: An architecture for global-scale persistent storage,” 2000, pp. 190–201. G. Bianchi and R. Melen, “Performance and dimensioning of a hierarchical video storage network for interactive video services,” European Transactions on Telecommunications, vol. 7, no. 4, pp. 349–358, 1996. [Online]. Available: http://dx.doi.org/10.1002/ett.4460070407 S. Pawar, S. Y. E. Rouayheb, H. Zhang, K. Lee, and K. Ramchandran, “Codes for a distributed caching based video-on-demand system,” in Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers, ACSCC 2011, Pacific Grove, CA, USA, November 6-9, 2011, 2011, pp. 1783–1787. [Online]. Available: http://dx.doi.org/10.1109/ACSSC. 2011.6190328
[18] V. Ntranos, G. Caire, and A. G. Dimakis, “Allocations for heterogenous distributed storage,” CoRR, vol. abs/1202.1596, 2012. [Online]. Available: http://arxiv. org/abs/1202.1596 [19] Z. Li, T. Ho, D. Leong, and H. Yao, “Distributed storage allocation for heterogeneous systems,” in Communication, Control, and Computing (Allerton), 2013 51st Annual Allerton Conference on, Oct 2013, pp. 320–326. [20] D. Leong, A. Dimakis, and T. Ho, “Distributed storage allocations,” Information Theory, IEEE Transactions on, vol. 58, no. 7, pp. 4733–4752, July 2012. [21] M. Gerami, M. Xiao, and M. Skoglund, “Optimal-cost repair in multi-hop distributed storage systems,” in Information Theory Proceedings (ISIT), 2011 IEEE International Symposium on, July 2011, pp. 1437–1441. [22] T. Ernvall, S. El Rouayheb, C. Hollanti, and H. Poor, “Capacity and security of heterogeneous distributed storage systems,” in Information Theory Proceedings (ISIT), 2013 IEEE International Symposium on, July 2013, pp. 1247–1251. [23] J. Pernas, C. Yuen, B. Gaston, and J. Pujol, “Nonhomogeneous two-rack model for distributed storage systems,” in Information Theory Proceedings (ISIT), 2013 IEEE International Symposium on, July 2013, pp. 1237– 1241. [24] K. G. Benerjee and M. K. Gupta, “On heterogeneous regenerating codes and capacity of distributed storage systems,” CoRR, vol. abs/1402.3801, 2014. [Online]. Available: http://arxiv.org/abs/1402.3801 [25] Q. Yu, K. W. Shum, and C. W. Sung, “Tradeoff between storage cost and repair cost in heterogeneous distributed storage systems,” Transactions on Emerging Telecommunications Technologies, pp. n/a–n/a, 2014. [Online]. Available: http://dx.doi.org/10.1002/ett.2887 [26] R. Ahlswede, N. Cai, S. yen Robert Li, and R. W. Yeung, “Network information flow,” IEEE TRANSACTIONS ON INFORMATION THEORY, vol. 46, no. 4, pp. 1204–1216, 2000. [27] P. Elias, A. Feinstein, and C. Shannon, “A note on the maximum flow through a network,” Information Theory, IEEE Transactions on, vol. 2, no. 4, pp. 117–119, Dec 1956. [28] L. R. Ford and D. R. Fulkerson, “Maximal Flow through a Network.” Canadian Journal of Mathematics, vol. 8, pp. 399–404. [Online]. Available: http://www.rand.org/ pubs/papers/P605/ [29] M. Berkelaar, P. Eikland, and P. Notebaert, “lp solve-a mixed integer linear programming (milp) solver,” Version 5.5.2.0,2011. Available from: http://lpsolve.sourceforge.net/5.5/.
