Distributed Node with Distributed Quota System (DNDQS).pdf ...

Distributed Node with Distributed Quota System (DNDQS) Author: Eric Yu ([email protected]) Date: 2016/02/23

Contents • • • • • • •

Scenario Assumption Architectures Roles of components Characteristics & Benefits Potential Problems Improvements

Scenario • A Distributed node system would like to apply API usage quota for some query service across all nodes. • We only allow nodes to call the query service for some certain amount of quota. No matter the call is come from which node, it just counts. • After use out the quota, all further query service would be blocked.

Assumptions • Distributed node system which allow dynamic node insert/delete. • Nodes are behind load balancing. Perfect load balance is not required. • Small and under-controlled number of false positive quota is acceptable. • Nodes are able to reach each others though network.

Architecture - Distributed node

Node 1

Node 2

Node 3 Node 4

With Consistent hashing

Node 1

Node 2

Node 3 Node 4

Consistent Hashing

With Consistent hashing – Delete node

Node 2

Node 1

Node 3

Consistent Hashing (Deleting Node 4)

With Consistent hashing – Insert node Node 5 Node 1

Node 2

Node 3 Node 4

Consistent Hashing (Inserting Node 5)

Quota Server – sync quota to nodes

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota Server – sync quota to nodes Quota: 250 Quota: 250

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 250

Quota: 250

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 150

Quota: 90

After some time ……

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Quota: 90

Node 4

Periodic Checking Quota:

I have 90

Quota: 150 Give you 30

(150+90)/2 = 120 150-30 = 90+30 = 120

Node 3 Quota Request next peer (Node 4) share quota

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 120

Quota: 120

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 50

Quota: 120

After some time ……

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000

I have 50 Give you 35

Node 3 Quota: 120

Node 4 Quota: 50

(120+50)/2 = 85 120-35 = 50+35 = 85

Node quota negotiation Quota: 180 Quota: 85

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 85

Quota: 120

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 10

Quota: 15

After some time ……

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000

I have 10 No quota for share

Node 3 Node 4

Quota: 15

Quota: 10

When self Quota < 10, don't share quota with preceeding peer.

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 10

Quota: 15

Remain no change

Node stored information Node X

Stored information: - node list (node 1, node 2, node 3, node 4, ...) - local quota

• DO know who is the next node along the node chain.

Roles of components 

Quota Server's role:   



Update quota configurations (e.g: quota=1000) to local nodes. Sync node list to local nodes (Each node keep a copy) upon node insert/delete. Collect & Redistribute quota at some time interval to keep distributed quota balance.

Node's role:   

Update local quota count. Run house keeping job to monitor local quota. Send quota share request to next node (mono-directional). Response to share quota to preceding node upon receiving quota share request.

Characteristic & Benefits 

Characteristic: 







Distributed quota on nodes, instead of centralized & synced quota count. Make use of consistent hashing to form a virtual ordered chain of nodes. Balance node quota by mono-directional flow of quota through the node chain.

Benefits: 





No need to sync quota with Quota Server to confirm quota usage very often. Avoiding heavy loading on Quota Server. Each internodes-sync would just happen between two nodes. Minimize sync operation overhead. Lower complexity of sync quota logic between nodes. 

Sync is    

Mono-directional In consistent order Circular Eventually balanced.

Potential problems – Dead/blocking node Potential issues: • Application issue in responding quota share request which broke the propagation of quota balance flow. (e.g: Dead Node) •

Solution: If the node cannot get response from next node, then it try to request quota from the next of next node instead.

Potential problems – Quota unbalance due to propagation delay •

•

Quota request loop is not quick enough to balance the local quotas. Solution: –

–

–

We can tune propagation speed by tuning Quota monitoring house keeping job interval. Total time for a complete loop is around N*T, where N is number of nodes, and T is house keeping job interval. Say, 10 nodes with 5 second, a complete loop would take around 50 seconds. 5 seconds interval by default. 2 or lower value seconds interval for high usage systems.

Potential problems – False negative quota blocking in threshold cases •

•

Some false negative quota blocking issues may happen when it is very close to the quota threshold. It is because of used out of local quota while actually there are still valid quota in other nodes. Solution: –

When it reached some threshold situation, all nodes would turn into threshold mode •

•

•

Temp quota is added to all nodes to make sure that every node has sufficient quota to work out for accepting remaining number of quota usage. The tradeoff is that, in worst case there may be (N-1)*Tq excess quota, where N is number of nodes and Tq is maximum potential temp quota quantity. With reasonable configuration tuning, and the assumption that small amount of excess quota usage is tolerable, this can solve the false negative quota blocking issue.

Improvements 

Improvements 



The distributed quota may not ideally balanced due to propagation delay, even we set a very short house keeping job interval. Proposed improvement 

 



By using two hash methods, we can have the 2nd virtual chain instead of a single virtual chain of nodes. Hence, each node would have When each time node sending quota share request, it can choose to request which one of the two『next-node』 randomly. It would make the distributed quota much more balanced.